
Qwen4 Max benchmark: below Qwen3.8 Max Prime on both our boards, at a tenth of the cost
Key takeaways
- Qwen4 Max scored 0.7365 out of 1 on our Gauntlet 7.2 coding benchmark, 12th on the board, below Qwen3.8 Max Prime's 0.8364, although it earned slightly more credit before the score ceiling: 0.9513 against 0.9499.
- Three failed 3D checks capped it at 0.739, and one of them missed by a single tower top: the scorer read 63 readable tops where the rule wants 64.
- Its two Forge 2.0 builds scored 0.3473 and 0.4087. Its app's sprint screens went blank when our simulated Jira refused requests for quota and told it to retry in about 18 minutes.
- Its three runs cost $9.30 in total against $88.72 for Max Prime, and most of that gap is the price of re-reading cached context: $0.075 against $0.50 per million tokens.
Qwen4 Max scored 0.7365 out of 1 on Gauntlet 7.2, our agentic coding benchmark, on the day Alibaba released it. That's 12th on that board. Its older sibling, Qwen3.8 Max Prime, scored 0.8364 in October and sits 2nd. On Forge 2.0, where the agent upgrades a Jira app, Qwen4 Max's two builds scored 0.3473 and 0.4087 (20th of 46), against Max Prime's 0.5015 and 0.6515 (6th). The new flagship sits below Max Prime on both boards. Each model here has one Gauntlet run and two Forge builds, so I'd read every gap with that in mind.
My short verdict: strong on backend recovery, weak on screens, and very cheap. Its three runs cost $9.30. Max Prime's cost $88.72.
On Gauntlet, Qwen4 Max earned slightly more credit than Max Prime (0.9513 against 0.9499) and lost the difference to a score ceiling, partly over one tower top out of 64. On Forge, its app handled a quota refusal by waiting in silence, and a whole set of sprint screens came back blank.
We found no published agentic coding score for Qwen4 Max when we ran it, so we ran ours. We also ran Baidu's ERNIE Light DT 1.0, for which we found no published benchmark at all. It scored 0.4761 on Forge 2.0, above Qwen4 Max, and our scorer couldn't score its Gauntlet run.
Qwen4 Max benchmark results on Gauntlet 7.2 and Forge 2.0
Both benchmarks run in our agent harness, the desktop coding app we build. A call is one model request. Gauntlet 7.2 gives the agent 150 calls to build a payments console: two Python services and a 3D view of payment towers that has to stay live as events stream in. Forge 2.0 gives it 300 calls to upgrade a Jira app on Forge, Atlassian's app platform, and we run it twice per model because one build is noisy. Our scorers then boot what the agent built and test it like a user would, on simulated sites, and write one row per check. The rules are on the Gauntlet board and the Forge board.
All runs below went through OpenRouter, except Argon's, which went to Google directly. Qwen4 Max's requests went to Alibaba, its only host.
Model | Gauntlet 7.2 | Earned before the ceiling | Forge 2.0 builds | Recorded cost, all runs |
|---|---|---|---|---|
Qwen4 Max | 0.7365 | 0.9513 | 0.3473 / 0.4087 | $9.30 |
Qwen3.8 Max Prime | 0.8364 | 0.9499 | 0.5015 / 0.6515 | $88.72 |
Gemini 4 Carbon | 0.7981 | 0.9825 | 0.5425 / 0.7862 | at least $40.64 |
Gemini 4 Argon | 0.7078 | 0.9761 | 0.2943 / 0.7926 | not recorded |
Gemini 4 Hydrogen | 0.5926 | 0.8736 | 0.4389 / 0.5032 | $4.19 |
I read these scores from the run pages on 16 November 2026, and the boards may have moved since. With two Forge builds, the median the board shows is their average.
The Forge board asks you to compare models within the same comparison group, which tracks the harness version they ran on. Qwen4 Max shares its group only with ERNIE Light DT 1.0. Max Prime is in another group, and I still compare them because Qwen4 Max's better build (0.4087) is below Max Prime's worse one (0.5015).

What the bill looks like
Run | Qwen4 Max | Qwen3.8 Max Prime |
|---|---|---|
Gauntlet 7.2 | $2.04 (75 requests) | $16.68 (95 requests) |
Forge 2.0 build 1 | $3.67 (144 requests) | $35.98 (192 requests) |
Forge 2.0 build 2 | $3.59 (177 requests) | $36.06 (206 requests) |
Total | $9.30 | $88.72 |
Most of that gap is cached input. I'd check that price first. An agent re-reads its whole context on every call, and OpenRouter charges $0.50 per million cached tokens for Max Prime against $0.075 for Qwen4 Max. Cached reads came to $71.97 of Max Prime's $88.72 and $5.07 of Qwen4 Max's $9.30, rebuilt from the run pages' token counts at list prices. Max Prime also read about twice the tokens: about 63 million per Forge build against 29 to 30 million.
Qwen4 Max listed at $1.50 per million input tokens and $4.50 output when we ran it; by the evening of 16 November OpenRouter showed $1.425 and $4.275. Both bills were complete. Carbon's are lower bounds, because some of its calls came back without a billing id. Google's pricing page doubles Argon and Carbon from 1 December, from $2 to $4 per million input tokens and from $10 to $20 output. At Google's list price the same Forge pair for Carbon would cost about twice as much after that, though OpenRouter, where we ran Carbon, had posted no change when we checked.
Qwen4 Max vs Qwen3.8 Max Prime on Gauntlet: more credit, lower score
Gauntlet has admission bands, which work as a ceiling, and the board puts it this way: "The final score is the lower of earned credit and the ceiling that applies." Each band has a top: 0.899 for backend recovery, 0.799 for 3D interaction and overview. Every extra failed check inside a band lowers its top by 0.03, and the final score lands a hair under that.
The two Qwen models failed opposite bands. Max Prime failed three backend recovery checks: replaying a committed event, resuming its outbox after we killed it, and notifying exactly once. Three fails take the 0.899 band down to 0.839, and Max Prime scored 0.8364. Qwen4 Max passed all three and failed three 3D checks instead, which takes the 0.799 band down to 0.739. It scored 0.7365.
Had those three 3D checks passed, nothing would have capped it, and 0.9513 would sit second on the board, 0.006 behind GPT-6.1 Sol Pro. It didn't pass them. I agree with the ceiling: a payments view whose towers you can't read is broken for the person using it. So the whole 0.0999 gap comes from the 3D view. Max Prime lost its band on recovery, and Qwen4 Max passed recovery.
One tower top short of 64
The overview check counts how many tower tops a person can read at the default camera, and Qwen4 Max's view showed 63 readable tops, all 63 in the right colour. The rule is score = accuracy x min(1, sampled/64), so 63 gave 0.984 for that part and 0.9948 for the row. It's one of the three capping fails. One more readable tower top would have lifted the score from 0.7365 to 0.7665.

Max Prime showed all 96 of the tops the scorer samples, while Gemini 4 Carbon showed 86 of 86 and lost the same row on framing. We don't have Qwen4 Max's Gauntlet code to say why one top hid. My guess is smaller or more crowded towers at the fixed camera angle.
The live stream that never showed
I find the second miss stranger. Gauntlet streams new payment events into the running app and checks that the 3D view changes. In Qwen4 Max's app the scorer saw the graphics buffer change, inside its byte budget and with no reallocation, and then no pixel changed. Its stream rows read "no stream batch applied".
The four other models with a Gauntlet score applied the batch: Max Prime showed it in 63.6 ms, Argon in 141.1 ms, Carbon in 158.9 ms and Hydrogen in 206.2 ms. My guess is a missing redraw after the buffer write, and I hold it loosely: the code is gone, and our scorer's stream reading changed on 2 November, after Max Prime and Argon were scored.
The third fail is a 3D interaction check, t_brush_link, which scored 0.889 with one of its parts failing. I can't say more without the code.
Where Qwen4 Max did better
Its backend recovery beat Max Prime's. After we killed its outbox mid-send, it resumed with nothing lost and nothing sent twice; Max Prime's outbox didn't resume, and events were lost. Its notifier processed 2,970 events, all 2,970 unique, with full coverage where Max Prime's reached 0.60, and when the backend was down its screen told the person what to do: "The ledger backend is unreachable. Check the service, then r..." (the scorer keeps only the start). Max Prime's gave no next step. I'd take Qwen4 Max's backend.
Qwen4 Max on Forge 2.0: the quota wall
Forge 2.0's app lives inside Jira, and Jira rations how much an app may call it. Our test sites are simulated Jira sites that run on a virtual clock and give each app 2,400 points per virtual hour. Background work may use 1,680 of them. The task says, word for word: "No person-facing request is ever answered with a quota 429." A 429 is the HTTP code for "too many requests, come back later".
It broke that promise. Its first build had person-facing requests refused for quota on all three test sites, its second on two, and on one site it went badly. In that site's seventh virtual hour, build 1 used 2,401 of its 2,400 points: 1,600 on background work, inside its share, and 801 on requests for the person. Thirty requests were refused. Each was told to retry in 1,055 to 1,070 seconds. Its page code is the same in both builds. I'd look here first:
1// Resolvers hand a long Jira Retry-After back instead of sleeping past their 25 s limit; the page
2// waits it out and calls again.
3export async function call(key, payload) {
4 for (;;) {
5 const res = await invoke(key, payload);
6 if (res && res.rateLimited) {
7 await wait(Math.max(1, Number(res.retryAfter) || 1) * 1000 + 50);
8 continue;
9 }
10 return res;
11 }
12}So the page waits nearly 18 minutes. It draws nothing until the answer comes. Its sprint page waits for that data. All 13 sprint screens our scorer opened in that hour were blank and white, with no error on the page.

It had a better answer too. Its Rovo action (Rovo is Atlassian's AI assistant) handles the same refusal with a sentence:
1if (e instanceof RateLimited) return { error: `Jira is rate limiting this app; try again in ${e.retryAfterSeconds} seconds.` };It checked its quota during the build, but only the background share. In build 1 it wrote "Hour-0 run works: 1013/1680 points", and in build 2 "background dosing 1045 ≤ 1680 with no refusals". The 1,680 is the background allowance, and its checks never looked at what was left for the person. In the hour that broke, background work used 1,600 points, which left 800, and the person side used 801.
That one site cost build 1 a lot. Eight rows score 1.0 on the other two sites and 0 on this one. We re-ran our scorer's arithmetic without that site to measure it: build 1 goes from 0.3473 to 0.5471, build 2 only from 0.4087 to 0.4351. I wouldn't call it a fair re-rank, because other models would lose their worst site too. Even then the pair would average 0.491, still under Max Prime's 0.5765.
Max Prime's two builds had no person-facing request refused ("0 of 1084" and "0 of 951"). Its sprint screen loaded in both. Argon's and Carbon's better builds also had none refused. Their weaker builds had two and five, and Hydrogen had refusals in both builds.

A Forge score multiplies three numbers (tests earned, critical defects and reliability), and reliability shrinks the score for each group of checks whose worst check failed. Qwen4 Max earned 0.8581 and 0.8713 of the tests, close to Max Prime's 0.9345 and 0.9670, but reliability is where they part: 0.4048 and 0.4691 against 0.5366 and 0.6737. Split in log terms (an approximation), about 77% of the gap between the two models is reliability, led by the quota checks. Both models paid the same penalty for the sprint-numbers widget, so that isn't what separates them.
Build 2 had its own problem. After a sprint was re-estimated, its sprint numbers were right for 1 of 6 sprints, against 5 of 6 in build 1. Its Rovo action answered 0 of 5 questions correctly. Build 1 answered 5 of 5, and Max Prime's builds 2 and 0. The totals function is the same in both builds, so the difference is in the estimates that feed it. We didn't trace it further.
Done at half the budget, without looking at the screen
Qwen4 Max decided it was finished at 74 of 150 calls on Gauntlet, and at 143 and 176 of 300 on Forge. Our harness reminds a model to finish its work at 80% of its budget. That reminder never fired. Its last words in Forge build 1 were "All green." and "The app is complete and verified." Build 2 closed with "The Scope Ledger v2 app is complete and verified." (Scope Ledger v2 is the app the task asks for.)
It tested with scripts. In build 1 it opened two screenshots of its own app, and in build 2 it opened none, because it wrote Playwright scripts and read their JSON output. At 14:57 in build 1 it wrote "The sprint action renders perfectly.", which held on its own test data, while on our third site the same screen was blank.
I don't think stopping early explains the score by itself. Gemini 4 Carbon also stopped early (101 and 123 of 300 calls) and its better build scored 0.7862. Max Prime did use more requests, 192 and 206 billed on Forge against Qwen4 Max's 144 and 177, and on Gauntlet it used about twice the reasoning tokens per request, roughly 2,000 against 1,075. That's a difference in habit. I can't prove it caused anything.
You might suspect we ran it at a low reasoning setting. We didn't. We ask every model for medium, and the request logs we kept from its Forge builds show medium. Alibaba's page lists only low, high and xhigh for Qwen4 Max and says "medium → high", so by Alibaba's own mapping it most likely ran at high and still used fewer reasoning tokens per request than Max Prime.
Gemini 4 Carbon, Argon and Hydrogen benchmarks: how they compare
Google's three Gemini 4 models are on both boards too. Carbon scored highest of the three here, on one Gauntlet run and two Forge builds: 0.7981 on Gauntlet, 4th on the board, and 0.7862 and 0.5425 on Forge 2.0, an average of 0.6643. Its Gauntlet run lost one 3D row, the same overview check, on framing.
Argon earned 0.9761 on Gauntlet and kept 0.7078 after four 3D fails. Its Forge pair, 0.7926 then 0.2943, is the widest spread of the six models here; we covered it in our piece on how much one build can swing. Hydrogen is fast. Its Gauntlet build took 6 minutes 37 seconds against Carbon's 79 minutes 24 seconds, and it scored 0.5926, capped because its 3D surface wasn't visible enough.
Carbon beats Qwen4 Max on both boards. Argon scores lower on Gauntlet (0.7078 against 0.7365) and higher on the Forge average (0.5434 against 0.378). Hydrogen is below Qwen4 Max on Gauntlet and above it on Forge, for $4.19 in total. I like that price. Of the three, I'd pick Carbon.
Artificial Analysis, which runs its own index, had Carbon at 60 and Qwen4 Max at 43 on 16 November.
ERNIE Light DT 1.0 benchmark: 0.4761 on Forge 2.0, no Gauntlet score
ERNIE Light DT 1.0 is a new Baidu model that reached OpenRouter on 10 November, with Baidu as its only host. Baidu's Qianfan model list gives it a 1-million-token context and files it under its deep-thinking models, but doesn't spell out what DT stands for. OpenRouter's listing says it's "based on open-source GLM models", the family Z.ai publishes (our boards list one as GLM 5.3, with a model id starting z-ai/). Baidu's own pages don't say that, so that claim rests on the listing alone.
On Forge 2.0 its two builds scored 0.4675 and 0.4847, an average of 0.4761 and 13th of 46 on the board. That puts it above Qwen4 Max (0.378), the one other model in its comparison group. Across groups it sits a hair above Gemini 4 Hydrogen (0.4710) and below Max Prime (0.5765). Its two builds were the closest pair of the six models in this article, 0.017 apart. Two builds is a small sample, though, so I wouldn't lean on that.
It hit the quota wall too, but less hard. Build 1 had 5 of 502 person-facing requests refused for quota, and build 2 had 1 of 366, so both builds lost that row. Its sprint screen loaded in both. Its losses were spread across many checks, and the numbers were the biggest share. In both builds the widget never showed a moved sprint's new numbers live. Build 1's widget showed the right numbers in 2 of 8 views and its Rovo action answered 1 of 5 active sprints exactly. Build 2 got 5.5 of 8 views and 0 of 5 sprints, and also failed the check on which tools its Forge AI call may use.
It also used its budget, the opposite of Qwen4 Max: 299 and 282 of 300 calls, and our wrap-up reminder fired in both builds. It read a lot of context, 74 and 89 million input tokens with almost all of it cached. The pair cost $21.14. Qwen4 Max's cost $7.26. Cached reads at $0.113 per million came to about 86% of ERNIE's bill, the same pattern as Max Prime's.
Its Gauntlet run has no score. The build made 150 requests (one of them our harness's own helper call) and cost $6.37. Then our scorer refused to score it. It couldn't measure the live-stream check on the 3D view, and its rules don't yet say how to charge that case, so I count the missing score as our scorer's gap. The 3D capture also timed out, and we haven't worked out yet whether that came from its app or from our scorer. We'll re-score the saved build once the scorer handles it.
What Alibaba has published about Qwen4 Max
Alibaba's Model Studio page describes Qwen4 Max as "a new-generation flagship multimodal model", one "suitable for programming development, office work, front-end design, and complex agent tasks", with a 1,000,000-token context. Its model page, release notes and pricing page list no parameter count, no architecture and no benchmark table (read 16 November; Qwen's own blog didn't load for us, so we couldn't check it). For comparison, Alibaba calls Qwen3.8 Max "a 2.4-trillion-parameter MoE flagship", while Max Prime is described only by OpenRouter, as "a higher-throughput variant of Qwen3.8 Max". I'd like Alibaba to publish Qwen4 Max's size.
Where we got it wrong
Qwen4 Max's Gauntlet build tree, the folder with its code and transcript, was deleted right after the run was published, before we asked to keep it. We should have asked before the run started. So the tower and stream moments rest on the scorer's rows, without the code behind them. The Forge trees survived because we asked.
Our Gauntlet scorer changed on 2 and 4 November, in how it reads the stream and the screenshots. Max Prime was scored on 6 October and Argon on 16 October, on the older code, and their trees are gone, so they can't be re-scored. The board still ranks them side by side, and I don't know which way that cuts.
One of our harness rules broke on Forge: build 2 started scoring about ten minutes before build 1 finished building, which the harness should prevent. It doesn't show up in the scores we can check.
The harness also differs from model to model. Max Prime and Argon ran on an older build without the wrap-up reminder, one Forge build after the other. Qwen4 Max ran on the newest, both builds at once.
Would I use Qwen4 Max for agentic coding?
For backend services, it's worth trying at this price. On our one Gauntlet run its payments services came through all the failures we injected, more cleanly than Max Prime's. For screens a person uses, I'd wait. It checks its work through scripts and JSON, rarely opens a screenshot and calls the job done at half its budget, and on Forge its own server code handed the page an 18-minute wait instead of an answer.
If you use it, I'd do three things. Ask it to screenshot each screen before it says done. Don't accept "complete" at half the budget without checking. And test what your app shows a person when a quota runs out.
If you've run Qwen4 Max on your own agent work, I'd like to hear whether it stops early for you too.

Comments
0Add a comment