The Moat Was Never the Model: One Model Swings 18 Points on the Same Benchmark, and the Only Variable Is the Harness

For solo founders and small-team operators running AI coding and agent products — anyone about to pick a model, reprice a stack, or re-architect around an "X beats Y" agentic benchmark.

---

The first screen: the number was produced in someone else's harness

On October 17, 2026, DeepSeek shipped DeepSeek-V4.1-Flash-1017 as a silent update. It went live on the website, the app, and the API under the unchanged model name deepseek-flash — the legacy names deepseek-v4-flash and deepseek-v4-flash-vision-exp are still accepted and now routed to it — with the weights published under the MIT license and no launch event. The model card calls it, plainly, "a minor update." (model card; API changelog.)

The headline everywhere was the benchmark: 77.0 on DeepSWE v1.1, ahead of Claude Opus 5 (74.0) and GPT-5.6 Sol (73.0). "An open model beat Opus." (model card.)

Here is the sentence almost every write-up left out, and it is in DeepSeek's own footnote:

"We use the Claude Code harness for Terminal-Bench 4.0 and the Mini-SWE-V2 harness for DeepSWE v1.1."

That 77.0 is not a property of the model. It is a property of the model running inside a specific scaffold — and DeepSeek says so itself, describing the release as "advancing the performance-cost Pareto frontier across multiple agent harnesses." (model card.)

Now look at a public leaderboard that keeps the two things apart. On Terminal-Bench 2.0, MODEL and AGENT are separate columns — and one unchanged model spans a huge range depending only on which agent runs it:

  • Claude Opus 4.6 — Claude Code: 58.0%**
  • Claude Opus 4.6 — Terminus 2: 62.9%
  • Claude Opus 4.6 — Mux: 66.5%
  • Claude Opus 4.6 — Droid: 69.9%
  • Claude Opus 4.6 — MAYA-V2: 72.1%
  • Claude Opus 4.6 — Meta-Harness: 76.4%**

(Terminal-Bench 2.0 leaderboard.)

Same weights. An 18.4-point swing. The only variable that moved was the harness.

By the end of this piece you have two things you can use this month:

  • The reframe* — an agentic score is a model-times-harness* number, not a model ranking, and the harness is a first-class column on the benchmark itself.
  • The harness-portability check** — five rules for comparing and shipping models that keep the number honest and keep you from betting your architecture on someone else's scaffolding.

Here is where we are going:

  • The surface version** — why "the leaderboard says which model wins" reads as settled.
  • The part nobody explains** — the scaffold is half the number, and it is a separate column for a reason.
  • Why it matters** — the 18-point swing, and the gain the whole model cycle is arguing about came from the scaffold, not a smarter base model.
  • How to use it** — the five-rule harness-portability check.
  • The misunderstanding that comes from skipping this** — why a cross-harness leaderboard read is a category error, and what to do first.

---

1. The surface version: "the leaderboard says which model wins"

The story as it is told is a horse race. A lab ships a model, the model posts a number, the number beats a rival's number, and everyone reprices their stack and re-plans their architecture around the winner. This cycle has been nothing but that: DeepSeek against Opus 5, Reflection Beam, Meta Muse Spark Open, Gemini 4 Argon — a new "X beats Y" every week.

The compression is understandable. A leaderboard looks like a pure model ranking. One row, one model, one score. The implicit assumption — the one nobody states because it feels obvious — is that the harness is a constant, a neutral pipe that just delivers the model's intelligence to the tasks.

If that assumption were true, a cross-model leaderboard read would be exactly what it looks like: a ranking. It is not.

2. The part nobody explains: an agentic score is a model-times-harness number

An agentic benchmark does not run a model. It runs a scaffold around a model, and reports how the two performed together. That scaffold is everything the model does not supply itself:

  • the agent loop — how it plans, acts, and decides it is done;
  • the tools — which ones exist, and how they are described;
  • the system prompt and prompt encoding — how the conversation is turned into tokens;
  • context management** — what gets kept, summarized, or dropped when the window fills;
  • the edit/retry policy — how failures are detected and re-attempted;
  • the thinking budget — how long the model is allowed to reason.

Change any of those and the same weights score differently. That is not a bug in the benchmark; it is the benchmark being honest. Terminal-Bench literally lists MODEL and AGENT as two columns on the same row, because the score belongs to the pair. (Terminal-Bench 2.0 leaderboard.)

Third-party leaderboards make the conditioning explicit by freezing one harness and saying so. Artificial Analysis states it runs Terminal-Bench 2.1 "with the Terminus 2 agent harness in an e2b sandbox" and reports pass@1 averaged over 3 repeats — the published number is harness-conditioned by construction, not by accident. (Artificial Analysis.)

So when you read "Model A beats Model B," you are comparing two different scaffolds with different models in them — not two intelligences. That is the part nobody explains, and it makes the surface version useless the moment you try to act on it.

3. Why it matters: the 18-point swing, and where the gain actually came from

The number is not a rounding error, and it is not only a leaderboard curiosity. It is the size of the decision you are about to make.

One row, two columns, same model. Claude Opus 4.6 scores 58.0% on Terminal-Bench 2.0 under the Claude Code harness and 76.4% under Meta-Harness — an 18.4-point spread for an unchanged model. (Terminal-Bench 2.0 leaderboard.) If you pick a model off a cross-harness leaderboard, that spread is the error bar on your read. It is larger than most of the "beats" margins the cycle is arguing about.

The headline numbers sit inside that error bar — and the harness is not even held constant inside one lab's own reporting. DeepSeek's Oct 17 footnote names the Claude Code harness for Terminal-Bench 4.0 and Mini-SWE-V2 for DeepSWE v1.1. Earlier DeepSeek benchmark notes used "the DeepSeek Harness minimal mode as the framework" for the same family. (model card; API changelog.) Cross-version comparisons therefore mix harnesses too, not just cross-vendor ones.

And here is the turn that makes the whole cycle ironic. The flagship gain did not come from a smarter base model. The Decoder reports DeepSeek's success as reinforcement learning with self-feedback (RLSF) post-training plus shorter outputs — the same architecture, re-trained to behave well inside the loop. (The Decoder.) That is the same shape a third party put bluntly on X: the improvement came "with 'no algorithmic innovation' but instead by RL training it on multiple coding agent harnesses" — a claim that is unverified and should be treated as such. (Cline on X, flagged: not the primary announcement.)

Why does post-training the scaffold cut cost? Because agentic spend is dominated by output tokens and retries. Per The Decoder, DeepSeek hit 77.0 on DeepSWE at just under $0.20 per task (its predecessor scored 74.2 at roughly $0.50), with rivals landing "between $2 and over" $10 per task on the same chart. That is a company figure, not an independent reproduction — but the mechanism is legible: train the model to use fewer tokens and fewer retries per task, and the score rises while the bill falls. (The Decoder.)

That is also why the harness is yours to own. This release ships no Jinja chat template — adopters must encode multi-turn, tool-call, and thinking-mode prompts themselves, using the provided encoding.py reference or the deepseek-recipe toolkit. (model card.) The prompt-encoding layer is part of the harness, and the vendor has deliberately handed it to you.

4. How to actually use it: the harness-portability check

You do not need to stop using benchmarks, and you do not need to stop comparing models. You need to stop treating a cross-harness number as a model ranking. Five rules do that.

1. Pin the harness next to every score. Never put a number beside a model without the agent/harness name beside it. Write "Opus 4.6 in Meta-Harness, 76.4%" — not "Opus 4.6, 76.4%." If a source gives you the model without the harness, the number is not comparable to anything, including itself. (Terminal-Bench keeps these as separate columns for exactly this reason.)

2. Own your eval. A public benchmark is not your workload. Build a versioned task set and a scoring function from your real work — the tickets, prompts, and failure modes you actually see — and treat that set as a product you maintain. This is the asset no leaderboard can hand you.

3. Measure cost per completed task, not per token. Agentic spend is dominated by output length and retries, and those move independently of headline token price. The DeepSeek cost chart is the illustration: same benchmark, same model family, and the win was shorter outputs and fewer retries — a completed-task metric, not a token-price one. (The Decoder.)

4. Keep the model behind an adapter. Put the model behind your own prompt-encoding, tool-schema, and thinking-budget layer, so a swap is a config change and not a rewrite. This release makes the case for you: with no shipped chat template, the encoding is yours to implement anyway. Own it, and every future "this model beats that one" becomes a test you can run rather than a bet you have to take.

5. Re-run the eval on every model swap. A "minor update" that shortens outputs or changes tool behaviour is a production change, not a footnote. -1017 shipped under the same API name with no price change; if you were pinned to deepseek-flash, your production behaviour moved without a version bump on your side. Re-run the eval on every swap and treat a silent checkpoint as a change you must measure.

5. The misunderstanding that comes from skipping this

The misunderstanding is that a leaderboard number is a fact about a model. It is a fact about a pair — and when two rows on that leaderboard were produced by different scaffolds, comparing them is measuring the scaffolding as much as the intelligence. An 18.4-point spread for one unchanged model is the whole argument: your cross-harness read can be wrong by more than the margin you are repricing around.

Two honest caveats, so you do not overcorrect:

  • This does not prove the harness explains the whole gap, or that models do not matter.** It proves agentic scores are harness-conditioned (a large, public spread) and that DeepSeek framed its own gains as harness-general. Capability still matters; it just is not the only variable, and you cannot read it off a cross-harness table.
  • Do not treat the vendor numbers as independent.* All the 1017 benchmark figures are DeepSeek's own, and independent reproductions were not published at the time of writing. The license is also muddier than the headline: the Hugging Face repository states the repo and weights are MIT, while at least one outlet reports the 1017* checkpoint weights as not disclosed (only the September weights being MIT). Rely on the HF page, and flag the discrepancy rather than repeating a press claim. (model card; MarkTechPost.)

The operator move is smaller than it sounds: stop trusting the cross-harness read, and start owning the two things that survive a model swap — your harness and your eval.

Where you will get stuck

  • You will read a leaderboard as a model ranking.** It is a model-times-harness table; the harness is a column for a reason.
  • You will compare two models that ran in two different scaffolds** and call the difference "intelligence."
  • You will optimise token price** while your agentic bill is set by output length and retries, which move independently of it.
  • You will hard-wire one model's prompt format into your product**, then treat the next swap as a rewrite instead of a config change.
  • You will let a "minor update" change production behaviour silently** because it shipped under the same model name.

What to do first

Do not rewrite your stack today. Do one thing: put the harness name next to the next score you look at, and start your own eval.

  • Re-read the last model comparison you acted on.** If it did not name the harness for each number, treat the comparison as unusable and redo it on one scaffold.
  • Write down your harness.** One line each for the loop, the tools, the encoding, the context policy, and the thinking budget. If you cannot write the encoding line, you do not own it — and this release says it is yours anyway.
  • Start a versioned eval from your own tasks.** Twenty real tasks with a scoring function beats any public benchmark you cannot run on your own work.
  • Switch your metric to cost per completed task.** Track output tokens and retries, not just price per token.
  • Put the model behind an adapter.** Make a swap a config change, and re-run the eval every time — including on silent checkpoints.

The compressed ending

Remember it in four beats:

  • An agentic score is a model-times-harness number.** The benchmark runs the scaffold — loop, tools, encoding, context policy, thinking budget — not just the weights.
  • The harness is a first-class column.** Terminal-Bench lists MODEL and AGENT separately, and Claude Opus 4.6 swings from 58.0% to 76.4% — 18.4 points — with the model unchanged.
  • The gain the cycle is arguing about came from the scaffold.** DeepSeek's advance was post-training the same architecture to behave well in the loop (shorter outputs, fewer retries), which raised the score and cut the cost together.
  • The moat is the harness and the eval you own.** Hold the harness fixed, keep the model behind an adapter, measure cost per completed task, and every future "this model beats that one" becomes a config change you can test.

As agentic capability commoditises — open weights, MIT licences, near-frontier scores at a fraction of the cost — the model stops being the differentiator. The model's leaderboard number was produced in someone else's harness. The harness you own, and the eval you version, are the parts nobody can hand you.

---

Sources

  • DeepSeek — DeepSeek-V4.1-Flash-1017 model card — "minor update"; the benchmark table; the footnote naming the Claude Code harness (Terminal-Bench 4.0) and the Mini-SWE-V2 harness (DeepSWE v1.1); MIT license; no Jinja chat template; encoding.py and deepseek-recipe.
  • DeepSeek — API changelog / updates — deepseek-flash now served by DeepSeek-V4.1-Flash-1017; legacy names retired and routed to it; DeepSeek Harness developer preview; earlier notes' "DeepSeek Harness minimal mode as the framework."
  • DeepSeek — Models & Pricing — deepseek-flash peak/off-peak rates; off-peak = half of peak.
  • DeepSeek — announcement on X — live on website/app/API; "Pareto improvement in performance and cost."
  • Terminal-Bench 2.0 leaderboard — separate MODEL and AGENT columns; Claude Opus 4.6 spanning 58.0% (Claude Code) to 76.4% (Meta-Harness).
  • Artificial Analysis — Terminal-Bench 2.1 evaluation — "We run Terminal-Bench 2.1 with the Terminus 2 agent harness in an e2b sandbox and report pass@1 averaged over 3 repeats per task."
  • The Decoder — DeepSeek V4.1 Flash 1017 beats larger rivals in agentic tasks… — RLSF (RL with self-feedback) post-training; 77.0 at just under $0.20/task vs ~$0.50 for the predecessor (74.2); rivals $2–$10+/task; company figures.
  • MarkTechPost — DeepSeek Just Released V4.1-Flash-1017… — vendor comparison table; the "1017 weights: Not disclosed" vs MIT discrepancy; Search/Fetch APIs.
  • SWE-agent — mini-swe-agent — the minimal ~100-line SWE harness (context for what "Mini-SWE-V2" derives from); its own note that it "beats Claude Code and Codex on DeepSWE."
  • Cline on X — post (flagged: unverified third-party claim) — "no algorithmic innovation … RL training it on multiple coding agent harnesses."

Precision notes: (a) All 1017 benchmark numbers are DeepSeek's own, self-reported, with no independent reproduction published at the time of writing — treat them as vendor figures, not independent results. (b) The central reframe — "an agentic score is a model-times-harness number" — is this piece's inference from the model-card footnote, the Terminal-Bench leaderboard's two columns, and Artificial Analysis's methodology statement; it is an argument, not a quote. (c) The harness is not constant even across DeepSeek's own reporting over time: earlier notes cited "the DeepSeek Harness minimal mode," while the 1017 card cites Claude Code / Mini-SWE-V2; cross-version comparisons mix harnesses. (d) 'Mini-SWE-V2' as a named harness is not independently documented; mini-swe-agent v2 is the probable ancestor and is linked as such. (e) The $/task figures are a company chart on one benchmark (DeepSWE) and are not proved to survive on any given operator's tasks. (f) Cline's "no algorithmic innovation" and the BugBunny VulnPR-100 figures circulating on X are third-party, unverified, and are not published here as fact. (g) The "-1017 weights: Not disclosed vs MIT" discrepancy is inherited from press reporting against the HF page; the HF repository page states MIT. (h) Not proven by any source here: that the harness explains the whole capability gap, or that MIT-licensed ~763B weights are economic to self-host — the realistic path for a solo operator is the API.

---

X entryway list (from the approved angle)

Distribution entryways named in the angle brief. Hooks are not yet written as posts — this names each post's type, the one claim or mechanism it carries, and whether it links.

  • Hook type: surprising proof.** Same model, same benchmark, different harness: Claude Opus 4.6 scores 58.0% on Terminal-Bench 2.0 with the Claude Code harness and 76.4% with Meta-Harness. An 18-point swing, and the model never changed. → Links to article (the number earns the read; the article supplies the model-times-harness reframe and the check that acts on it — the claim stands alone, the link is the deepening, not the reason it holds).
  • Hook type: mechanism.** An agentic score is a model-times-harness number. The benchmark runs the scaffold — loop, tools, prompt encoding, context management, thinking budget — not just the weights, which is why the leaderboard lists MODEL and AGENT as separate columns, and why DeepSeek's advance came from post-training the same architecture to behave well inside the loop. → Links to article (the mechanism is self-contained; the article carries the worked spread and the "where the gain actually came from" evidence).
  • Hook type: build sequence.** The harness-portability check: pin the harness next to every score; own a versioned eval from your own tasks; measure cost per completed task; keep the model behind an adapter; re-run the eval on every swap. → Links to article (the check is the actionable bit; the article is the why behind each of the five rules).
  • Hook type: risk.** You can pick a model off a leaderboard number that was produced in someone else's harness, reprice and re-architect around it, and watch the delta shrink or reverse the moment you run it in yours — while a "minor update" that silently shortens outputs changes your production behaviour without a version bump on your side. → Links to article (names the trap; the article generalizes it into the pinned-harness and adapter rules).
  • Hook type: result.** The defensible asset is not the model you pick this month; it is the harness plus the versioned eval you own. Hold the harness fixed, swap models behind an adapter, and every future "this model beats that one" becomes a config change you can test, not a rewrite you have to bet on. → Links to article (concrete before/after; the article ties it to the full check and the post-training mechanism).

Hook-type map: 1 = surprising proof; 2 = mechanism; 3 = build sequence; 4 = risk; 5 = result. All five link, and each link is a bonus — every post carries a claim that stands on its own. These are the five entryways the angle brief named; no additional entryway was invented, and no entryway repeats another's hook.

Source-fetch note: during drafting I re-fetched and confirmed — the DeepSeek-V4.1-Flash-1017 model card (introduction text, benchmark table, the Claude Code / Mini-SWE-V2 footnote, MIT license line, and the "no Jinja-format chat template" prompt-encoding section); the DeepSeek API changelog (deepseek-flash, retired legacy names, DeepSeek Harness developer preview); the DeepSeek pricing page (peak/off-peak); the Terminal-Bench 2.0 leaderboard, whose live table returned the exact Opus 4.6 rows — Claude Code 58.0%, Terminus 2 62.9%, Mux 66.5%, Droid 69.9%, TongAgents 71.9%, MAYA-V2 72.1%, Meta-Harness 76.4% — with MODEL and AGENT as separate columns; the Artificial Analysis Terminal-Bench 2.1 page (the Terminus 2 / e2b / pass@1-over-3-repeats methodology line); The Decoder piece (RLSF, the ~$0.20/task at 77.0 vs ~$0.50 predecessor, rivals $2–$10+); the MarkTechPost piece (vendor table and the weights-disclosure discrepancy); and the mini-swe-agent repository. All resolved to the content the research packet describes.