ArticleAI CodingAtlassian Forge
Muse Spark Open benchmark: where the open-weight models we ran this month broke as coding agents
Meta's Muse Spark Open scored 0.091 and 0.0877 out of 1 on our Forge 2.0 coding benchmark and never got a Gauntlet verdict. The other open-weight models we ran locally, Google's Gemma 4 among them, did no better, and neither did our own Qwen3.8-27B fine-tunes. The best open result on our Macs came from the plain Qwen3.8-27B base. We read every build to find where each one broke, and a fair share of the breakage was ours.
Mihai Perdum20 min read