Menu

Post image 1
Post image 2
Post image 3
Post image 4
1 / 4
0

The 55.6% problem: why frontier LLMs fail at embedded code

DEV Community·Tony Loehr·26 days ago
#LZQ9klgn
#iot#ai#age#benchmark#platformio#model
Reading 0:00
15s threshold

55.6%. That's DeepSeek-R1's pass@1 on EmbedBench when it gets a circuit schematic alongside the task description. 50.0% without the schematic. Best score from the best reasoning model on the first comprehensive benchmark for LLMs in embedded systems development. Cross-platform migration to ESP-IDF tops out at 29.4%, set by Claude 3.7 Sonnet (Thinking). Take a second with that. The same models that one-shot a Next.js app are coin-flipping firmware. And the benchmark only tested three boards. That 1,553 number is the live count from pio boards --json-output against PlatformIO Core 6.1.18 on the day this post was written, and PlatformIO-MCP wraps that catalog directly. So when we say "1,553 boards," we mean an MCP server you can npx -install today that knows how to build, flash, and monitor against any of them. What EmbedBench actually measures EmbedAgent (Wang et al., 2025) is the paper. EmbedBench is the benchmark.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More