Nanbeige Beats Qwen on '18 of 20' Benchmarks. Read the Two Rows It Lost First.

Status: draft — autonomous loop, ready to render (not yet approved by the user)

For solo founders and small-team operators choosing an open-weight model to self-host or route for a coding or office agent — anyone staring at a vendor's "beats the incumbent on N of M benchmarks" table this week and wondering whether it is finally time to switch.

---

The one thing to take away

When a vendor says its open model "beats the incumbent on 18 of 20 benchmarks," that count is a property of the table the vendor chose, not of the model. Change which benchmarks are in the table, which harness each row uses, who judges it, or which thinking setting one side gets — hold the weights fixed — and the number moves.

The claim travelling around the internet this week is real and worth your attention: Nanbeige4.3-33B, a 33.119B-parameter dense open model under Apache 2.0 with a native 512K context, published by Nanbeige LLM Lab on Hugging Face. Its model card carries a 20-row table comparing it against Qwen3.8-27B, Muse-Glimmer-30B and Gemma4-31B — and on that table it beats Qwen3.8-27B on 18 of 20 rows (model card). That is technically what the card says.

It is also a sentence with a denominator, and the denominator is the interesting part. Four of those 18 wins are by under a single point. The harness and judge change row to row. One setting that helps multi-turn agent behavior — preserve_thinking=true — was applied to the model being promoted and not to its rivals. And the two rows the model loses are SWE-Bench Pro and GDPval: the professional, economically valuable output benchmarks a buyer for paid work would weight most (OrcaRouter audit).

That single sequence — 20 rows counted two ways, with the misses landing exactly where a buyer would look — is the whole piece in one artifact.

By the end you have one thing you can use: a six-question benchmark-table audit you run before you switch a model, plus the one cheap test the release itself invites — re-run the two load-bearing rows on your own harness. "Beats X on 18 of 20" is a reason to run your own test, not a reason to switch.

Here is where we are going:

• The claim as it is usually stated** — "18 of 20, Apache 2.0, 512K context" read as a reason to switch.
• What the evidence actually says** — the same table counted two ways, and where the margins really live.
• Where the claim holds and where it falls apart** — a real long-horizon lead, a tie, a small deficit, and no independent score.
• What actually changes** — the six-question audit, and the neutral re-run of the load-bearing rows.
• The trap** — how a win count sells you a deployment cost you did not price.

---

1. The claim as it is usually stated

The surface version writes itself. "A new 33B open model beat Qwen3.8-27B on 18 of 20 published benchmarks, and its Q4 is only 19.1GB." That framing spread fast after release, and it is accurate as far as it goes:

"Nanbeige just released Nanbeige4.3-33B, delivering surprisingly strong coding and agent performance for its size, beating similarly sized Qwen 3.8 27b and Gemma models on most benchmarks. The weights are openly available under Apache 2.0, with support for a 512K-token context window." (@kimmonismus)

The lab's own post is more careful and equally promotional:

"Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters... When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference." (Nanbeige official X post)

Under the tempting reading, the win count is a capability measurement: more wins means a better model, and 18–2 means switch. If you run a coding or office agent, the pull is strong — a same-size open model, free weights, a 512K context, and a table that says it wins.

That reading is wrong in a specific, load-bearing way. The correction is not fine print you can wave past. It is the thing that tells you whether the table justifies the one hard-to-undo decision you are about to make.

2. What the evidence actually says

Start with what is not in dispute. The artifact is real and easy to verify: config.json reads LlamaForCausalLM, 52 layers, hidden size 7,168, 56 query / 8 KV heads, vocabulary 166,912, and max_position_embeddings 524,288 — the 33.119B / 512K figures read straight off the file (config.json). Data Studios independently confirms the dense architecture and the dimension-by-dimension parameter accounting (Data Studios). The weights shipped under Apache 2.0 as a model-card declaration — note there is no standalone LICENSE file in the checkpoint repo, and the release discloses neither the base checkpoint, the training-token count, the corpus, the SFT recipe, the RL procedure, nor the training compute (nlp.com.ai). Open weights, not open method.

Then the table. Count it two ways. Nanbeige leads 17 of 20 rows, and beats Qwen3.8-27B on 18 of 20. Both counts are accurate, and the gap between them is the point.

• The 3 rows Nanbeige does not lead: SWE-Bench Pro (Qwen 61.7 vs Nanbeige 60.3), GDPval (Qwen 86.4 vs Nanbeige 84.8), and GPQA Diamond (Gemma4-31B 91.3 vs Nanbeige 89.5).
• Of the 18 wins over Qwen, four are by less than one point: EnterpriseOps-Gym (41.0 vs 40.2), Tau3-Bench (48.8 vs 48.5), GPQA Diamond (89.5 vs 89.2) and AIME 2026 (95.2 vs 94.5).
• 19 of the 20 rows are point-scored; one is a dollar figure** (YC-bench: $1,025,437 vs $641,290).

That is not a footnote problem. The card states its method plainly — "All models are evaluated at their highest thinking effort. For each benchmark, we use the same inference hyperparameters and harness across models" — but the card's own footnotes show the harness changing per benchmark family: Terminus 2 for Terminal-Bench 2.1; mini-swe-agent v2.4.6 for Terminal-Bench 4.0, DeepSWE and SWE-Atlas-QnA; Claude Code for NL2Repo and JobBench; and GPT-6 Sol as judge for SWE-Atlas-QnA's 1,304 rubrics (model card). "Same per row" is not "same across the suite." And the one setting that most helps multi-turn agent behavior — preserve_thinking=true — was enabled for Nanbeige's own model and is not shown applied to the others.

Say exactly which part is the card's claim and which is the footnote: the claim is same harness across models; the footnotes describe a different harness per row, disclosed. The disclosure is to Nanbeige's credit. It is also the reason the comparison is harder to read than a single win count — and the reason that count is a summary of the table's construction, not a measurement of the model.

3. Where the claim holds and where it falls apart

Where it roughly holds. Nanbeige's largest margins are concentrated in long-horizon, multi-step agent rows, and there the lead is big and consistent in direction:

• DeepSWE 65.5 vs 42.5** — a 23-point gap.
• Terminal-Bench 4.0 23.8 vs 6.3.**
• Automation Bench 54.2 vs 38.3.**
• DeepPlanning 63.3 vs 49.2.**
• YC-bench $1,025,437 vs $641,290.**

That pattern is consistent with a 512K native context and the preserve_thinking setting, and it is exactly the benchmark family where harness choice moves scores the most. Data Studios draws the narrow conclusion: the small margins on SWE-Bench Pro, GDPval, Toolathlon-Verified and Tau3-Bench "suggest that its advantage over Qwen3.8-27B is concentrated in specific long-horizon tasks rather than uniform" (Data Studios). The honest verdict is the one OrcaRouter reaches: Nanbeige "probably is the stronger long-horizon agent model of the two" — but it "is not yet a reason to switch."

Where it falls apart. Three specific places:

One — the comparison is asymmetric, because only one side has been measured independently. Nanbeige4.3-33B is days old and, as of late October 2026, has no independent evaluation at all — no Artificial Analysis index entry, no arena score, no reproduction. Its opponent does: Qwen3.8-27B carries an Artificial Analysis Intelligence of 33.7 and Coding of 68.1 (OrcaRouter Qwen3.8-27B page). So this is a vendor's self-measurement compared against an independent one — a comparison of the measurers, not the models. Note also that the card's AA-LCR figure of 84.0 is Nanbeige running Artificial Analysis's benchmark, not Artificial Analysis running Nanbeige. Put the asymmetry in one line: Qwen's 34 is a number someone else measured; Nanbeige's 80.9 on Terminal-Bench 2.1 is a number Nanbeige measured.

Two — at least three of the opponent's numbers may not have been re-run under Nanbeige's harness. OrcaRouter infers, from the values, that three Qwen figures "appear to be copied from Qwen's own published results rather than re-run by Nanbeige." Treat this as a flagged suspicion, not a fact — it is an inference from the numbers, not a documented admission. The complication that keeps it honest: Qwen3.8-27B's independent Terminal-Bench 2.1 score on Artificial Analysis is 79.8, higher than the 73.0 the card attributes to it. Either the harnesses differ or the attribution needs care. Do not treat either reading as settled.

Three — the headline rounds away the rows a buyer would weight. The two rows Qwen wins are SWE-Bench Pro (the harder software-engineering variant) and GDPval (output quality on real occupational tasks). Both are benchmarks of professional, economically valuable work, and Qwen wins both — by margins that are small in absolute terms but stable across two different task families. In a two-day-old model with no independent score, that is the signal worth pausing on, not the win count.

And two things the launch coverage treats as settled that are not:

• "State-of-the-art agentic performance among open models in the ~30B class" is the vendor's own framing.** No independent leaderboard had scored the model as of late October 2026. It is a claim, not a measured standing.
• The context-degradation caveat is unverified commentary.** A local-models commentator warns that "this is a conventional dense Llama architecture with a growing KV cache, and early community testing is already showing performance falling as context fills," and that "the 19.1GB Q4 weights can fit on a 24GB GPU but not at 512K context" (@TeksEdge). That is not in the model card and has not been independently reproduced. Flag it as a thing to test, not a finding.

4. What actually changes — the six-question audit, and the re-run

Here is the reusable object. Read every vendor model-comparison table — Nanbeige's, and the next wall of ~30B open models that ships one — through six questions, in this order:

1. Denominator and margin. How many wins are real, and how many are rounding under a single point? Here: 18 of 20 sounds decisive; four of those wins are by under a point, and two are ties dressed as wins.

2. Re-run or copied. Were the opponent's numbers re-measured under the promoting lab's harness, or lifted from the opponent's own published results? If you cannot find the re-run documented per row, treat that row's margin as provisional.

3. Harness and judge, per row. The card's headline says "the same harness across models"; the footnotes show Terminus 2, mini-swe-agent v2.4.6, Claude Code and a GPT-6 Sol judge on different rows. Agent scores are a function of model × scaffold — so name the scaffold before you trust the score.

4. The setting the vendor kept for itself. preserve_thinking=true was applied to Nanbeige's model and not to the rivals. Ask which flags each side ran with, and whether your stack will replicate them.

5. Who else has measured either side. Independent measurement is the scarce good, and at launch it is asymmetric. One side here has an external score; the other has none. Until someone runs the weights, you are comparing a self-measurement to an independent one.

6. Deployment cost — free weights are not free inference. Nanbeige4.3-33B has no per-token price and no inference provider deploys it. Running it means a multi-GPU deployment held open: the card's own server example splits the model across four GPUs and reuses Qwen3 reasoning/tool-call parsers (AiLookout), and the only real run reported on the weights — a developer loading the full bf16 checkpoint on an 80GB A100 — used ~63GB of VRAM and took roughly 11 seconds for a first 1,024-token reply (OrcaRouter). By contrast Qwen3.8-27B is served as an API at a current list price of about $0.40 / 1M input tokens, $4.21 / 1M output, with a 262K context (page here; the page pulls pricing live, so treat the figure as moves).

From those six questions the workflow follows:

• Pick your row first.** Before you read the table, write down the one or two benchmarks that actually match your workload — the closest thing to what your agent does all day. The vendor's win count is irrelevant to you if it is earned on rows you will never run.
• Then run the load-bearing rows yourself. Put both models in their own recommended configurations — not one model's scaffold — and re-run the rows carrying most of the margin. Here that is DeepSWE and Terminal-Bench 4.0**. This is the specific test the release asks for and does not perform. It is cheap; two rows, your harness.
• Price the deployment before the weights.** Add the multi-GPU hold-open, the VRAM, the KV-cache growth with context, and the operating time onto the decision — then compare it against an API at a per-token price. A win count can sell you a switching cost you never priced.

Where you will get stuck

• You will read the win count as a capability score.** It is a summary of the vendor's table construction. Twenty rows picked four minutes after a base checkpoint, with per-row harnesses, is a measurement of choices, not of the model.
• You will miss the sub-point wins.** Four of the 18 wins are rounding. If your margin threshold is "wins by a meaningful amount," the real score is closer to a tie on several rows.
• You will trust "same harness across models" literally.** The footnotes say otherwise: Terminus 2, mini-swe-agent, Claude Code, a GPT-6 Sol judge, one row at a time. Name the scaffold.
• You will compare a self-measurement to an independent one and think you compared models.** One side has an external score, the other has none. That is a comparison of measurers.
• You will treat the copied-vs-re-run point as proven.** It is an inference; the Terminal-Bench 2.1 discrepancy (card 73.0 vs AA 79.8) says the attribution needs care, not a verdict.
• You will forget that free weights are not free inference.** No provider serves it; the reference deployment is four GPUs; the one real run used ~63GB for a first reply.
• You will act on the margin rows the headline showcases and ignore the two rows a buyer would buy on.** SWE-Bench Pro and GDPval are where Qwen leads — and both are professional-output benchmarks.

What to do first

Do not switch. Do one thing before your next model decision:

Write your row down, then re-run it. Open the table and name the single benchmark closest to your actual workload. Then take the two models in their own recommended configurations and run that row — plus one more that carries margin — on your own harness. For this pair, the two rows that matter are DeepSWE and Terminal-Bench 4.0.

If Nanbeige wins the row you run, under your scaffold, at a price you have actually costed — switch. If it does not, you have saved yourself a four-GPU migration you cannot easily undo. Either way, the win count stays where it belongs: on the table.

The compressed ending

Remember it in four beats:

• A benchmark win-count measures the table, not the model.** Nanbeige leads 17 of 20 rows and beats Qwen on 18 — both true, and the gap between them is the point. Change the rows, the harness, the judge, or the thinking setting, and the count moves at the same weights.
• The rows that carry the margin are the ones a lab can tune most* — long-horizon agent tasks, where the score is model × scaffold, and where the pitch is literally "when integrated into mainstream agent harnesses." The rows it loses* are the professional-output ones (SWE-Bench Pro, GDPval) a buyer would weight.
• Independent measurement is asymmetric at launch.** Qwen's score is a number someone else measured; Nanbeige's 80.9 is a number Nanbeige measured. Until the weights are independently run, you are comparing measurers.
• Free weights are not free inference.** No provider serves it; the reference deployment is four GPUs; the one real run used ~63GB. Price the switch before the table convinces you.

When every open ~30B model this month ships with a vendor table claiming to beat the others on N of M, the scarce, defensible skill is not reading the headline. It is reading the table before the headline — the denominator, the margins, the per-row harness, the setting kept for one side, the independent score, the deployment cost — and then running the one row that matches your workload. "Beats X on 18 of 20" is a reason to run your own test, not a reason to switch.

---

Sources

• Nanbeige — Nanbeige4.3-33B model card (primary; repo public 2026-10-24 02:49 UTC, card finalized Oct 25; Apache 2.0 per the card declaration) — the strategy ("build a high-capacity teacher agent first"), the SOTA claim, the 20-row table (Terminal-Bench 2.1 80.9/73.0; Terminal-Bench 4.0 23.8/6.3; DeepSWE 65.5/42.5; SWE-Bench Pro 60.3/61.7; SWE-Atlas-QnA 50.8/41.9; NL2Repo 53.0/48.3; Scicode 52.6/46.7; SkillsBench 61.1/56.4; Automation Bench 54.2/38.3; JobBench 44.8/34.5; GDPval 84.8/86.4; Claw-Eval 78.3/75.2; Toolathlon-Verified 70.4/68.5; DeepPlanning 63.3/49.2; EnterpriseOps-Gym 41.0/40.2; YC-bench $1,025,437/$641,290; Tau3-Bench 48.8/48.5; GPQA Diamond 89.5/89.2/91.3 Gemma; AIME 2026 95.2/94.5; AA-LCR 84.0/82.0), the "same inference hyperparameters and harness across models" line, the preserve_thinking=true footnote, and the per-row harness footnotes (Terminus 2; mini-swe-agent v2.4.6; Claude Code; GPT-6 Sol judge).
• Nanbeige — config.json (primary artifact) — LlamaForCausalLM; hidden size 7,168; 52 layers; 56 query / 8 KV heads; vocab 166,912; max_position_embeddings / seq_length 524,288; rope_theta 120,000,000; bfloat16.
• OrcaRouter — Nanbeige4.3-33B vs Qwen3.8 27B: Auditing the 18-of-20 Claim (independent, Oct 25 2026) — "a sentence with a denominator, and the denominator is the interesting part"; the sub-point wins; the copied-vs-re-run suspicion; the per-row harness disclosure; the epistemically-unequal comparison; "'18 of 20' is a real result dressed as a bigger one"; the verdict ("probably is the stronger long-horizon agent model of the two... not yet a reason to switch").
• OrcaRouter — Chinese edition, bylined Alistair Wren (Oct 25 2026) — the repo appeared 2026-10-24 02:49 UTC, four minutes after the base checkpoint, with no launch post/blog/press release; the 17-of-20 vs 18-of-20 dual count; the only real run reported on the weights (80GB A100, ~63GB VRAM, ~11s first 1,024-token reply).
• OrcaRouter — Qwen3.8-27B model page (independent counter-side) — Artificial Analysis Intelligence 33.7, Coding 68.1; context 262K; current list price ~$0.40 / 1M input, $4.21 / 1M output (live-pulled, moves).
• Data Studios — Nanbeige4.3-33B, open-weight dense agentic model, 512K context (independent) — "What can be verified directly is the artifact itself... The performance claims are a different category"; the narrow-margin-concentration reading.
• AiLookout — Nanbeige releases Nanbeige4.3-33B (independent) — the "tops Qwen3.8-27B on its own tests" framing; the four-GPU server example reusing Qwen3 parsers.
• nlp.com.ai — Nanbeige4.3-33B model record — 33.119B dense; 524,288-token context; "Apache 2.0 (model-card declaration; no standalone LICENSE file)"; the non-disclosure of base checkpoint, training tokens, corpus, SFT recipe, RL procedure, and training compute.
• Nanbeige — official X post (primary, Oct 25 2026) — the first-party SOTA claim and the "teacher for upcoming smaller Nanbeige models" plan.
• @TeksEdge — local-models commentary (Oct 25 2026) — the 18-of-20 relay plus the context-degradation and Q4-at-512K caveats (unverified; not in the card, not independently reproduced).
• @kimmonismus — the claim as it spread (Oct 25 2026) — the "beats most benchmarks" version.

Precision notes. (a) Date the piece "as of late October 2026," not a permanent verdict: independent evaluations, a technical report and quantized builds are all announced as coming. (b) The copied-vs-re-run point is OrcaRouter's inference from the numbers, not a documented admission; the Qwen Terminal-Bench 2.1 discrepancy (card 73.0 vs Artificial Analysis 79.8) is flagged, not resolved — do not present either reading as settled. (c) Pricing is pulled live by the model page and moves; cite the page and the date, not a fixed number. (d) The context-degradation claim is a single commentator's, unverified and unreproduced — treat as "test this," not a finding. (e) "State-of-the-art agentic performance among open models in the ~30B class" is Nanbeige's own framing; no independent leaderboard had scored the model as of late October 2026. (f) "Apache 2.0" is a model-card declaration with no standalone LICENSE in the checkpoint repo, and the release is open weights, not open method. (g) The six-question audit, the "win count measures the table" reframe, and the pick-your-row-then-re-run workflow are this piece's inference from the primary artifacts and the independent audit — an argument, not a vendor guarantee. (h) Nothing here claims Nanbeige4.3-33B is a weak model or not the strongest open long-horizon agent model at ~30B; the claim under pressure is narrower — that the headline is unverified and less decisive than it sounds.

---

X entryway list (per the approved angle brief — entryways named, not written yet)

Each hooks independently; the article link is a bonus, never the whole point.

• Hook type: surprising proof.** The same 20-row table, counted two ways: Nanbeige leads 17 of 20 rows and beats Qwen3.8-27B on 18 — both true, and the gap is the point. The three rows it does not lead are SWE-Bench Pro and GDPval (Qwen wins both — the professional-output benchmarks) and GPQA Diamond (Gemma wins). Meanwhile one side has an independent score (Qwen, Artificial Analysis) and the other has none. → Links to article (the two-count reveal is self-contained; the article supplies the audit that acts on it).

• Hook type: mechanism.** A benchmark win-count is a property of the table, not the model. Change the benchmark family, the harness per row, the judge, or the thinking setting one side gets — and the count moves, at the same weights. That is why "beats X on 18 of 20" is a measurement of the vendor's choices wearing the costume of a capability measurement. → Links to article (the mechanism stands alone; the article carries the per-row receipts).

• Hook type: workflow.** The six-question benchmark-table audit before you switch: denominator and margin; re-run or copied; harness and judge per row; the setting the vendor kept for itself; who else has measured either side; and deployment cost (free weights are not free inference). Then re-run the two load-bearing rows on your own harness. → Links to article (the workflow is the actionable bit; the article is the why behind each question).

• Hook type: risk.** The headline rounds away the two rows you would buy for — and free weights are not free inference. Nanbeige's biggest wins are on long-horizon agent rows (where a lab can tune its own scaffold to its own model); the two it loses are professional-output benchmarks; and running it means a multi-GPU deployment held open (~63GB for one real run), not a download. A win count can sell you a switching cost you did not price. → Links to article (names the trap; the article generalizes it into the audit).

• Hook type: result.** The cheap falsifying test the release invites: put both models in their own recommended configurations and re-run DeepSWE and Terminal-Bench 4.0 — the rows carrying most of the margin. "18 of 20" is a reason to run your own test on the two rows that match your workload, not a reason to switch on the count. → Links to article (concrete test; the article ties it to the pick-your-row rule).

Hook-type map: 1 = proof; 2 = mechanism; 3 = workflow; 4 = risk; 5 = result — the five entryways the angle brief named, none added, none repeating another's hook. The actual posts are not written here; that happens after the draft and the entryways are approved.