Back to Insights
AI Software

DeepSeek Open-Sourced the Harness That Graded Its Own Benchmarks

DeepSeek Harness (dsh) is MIT-licensed and model-agnostic. It is also the software that produced V4-Pro's Terminal-Bench and DeepSWE scores.

S5 Labs Team August 13, 2026

DeepSeek published the source of DeepSeek Harness, an MIT-licensed agent runtime installed with npx @deepseek-ai/dsh web, on August 13, the same day it took DeepSeek-V4-Pro out of preview. The repository went public around midday UTC and had, in The New Stack’s count, passed 33,000 GitHub stars within a few hours. A footnote on the model card ties the two releases together: V4-Pro-0813’s code-agent benchmark scores were “evaluated with the minimal mode of DeepSeek Harness as the agent framework,” so the harness that graded the model is now the same code anyone can clone.

That footnote is what changed since July. When DeepSeek posted the V4-Flash-0731 numbers on July 31, the equivalent line read “DeepSeek Harness (to be released),” which meant the Terminal-Bench and DeepSWE figures came from a tool nobody outside the company could run. As of today the tool exists in public. The numbers are reproducible in principle, and until someone outside DeepSeek reproduces them, that is all they are.

What dsh is

The README describes dsh as “an open-source agent harness” built on an architecture where “everything is a plugin,” powered by Cordis, a plugin framework whose design paper the README links. The architecture doc is blunter about what that means in practice: the model adapter, the tool registry, the session log, and the agent loop itself are all plugins, “so every part is replaceable from configuration,” and there is “no privileged core to patch.” A running dsh is a plugin tree composed at boot from ordered layers, and dsh --profile web --dump-config prints the tree your machine actually boots. Any row it prints can be replaced by a patch of your own. For a team that has been assembling its own agent loop from parts, that is a different proposition from a closed binary: the seams are documented, and the pieces you would otherwise fork are configuration rows.

The status warning is just as direct. The README says the project “is currently in developer preview and is iterating rapidly. THERE WILL BE COMPATIBILITY-BREAKING CHANGES.” (The capitals are DeepSeek’s.) The npm package sits at version 0.1.0 release candidates. CONTRIBUTING.md adds that DeepSeek “cannot accept external pull requests at the moment” and points would-be contributors at GitHub Discussions and at writing plugins under the dsh-plugin topic. The engineering notes checked into the repository carry dates going back to June, so the harness has been in internal use for at least two months; the public got it at the point DeepSeek was willing to break it in front of them.

Running npx @deepseek-ai/dsh web starts a local server at http://127.0.0.1:3080; the alternative is cloning the repo and running pnpm install, pnpm run build, pnpm dsh web. There is also a headless one-shot profile and a Python SDK that bundles its own runtime and needs no system Node.js.

Where it sits next to Claude Code and Codex

As a product, dsh belongs to the same category as Claude Code, Codex, and Meta’s Muse Code from last week: a local agent loop that reads and edits files, runs shell commands, delegates to sub-agents, and keeps an append-only log of everything the model saw. The differences are in what is fixed and what is swappable.

The default local sandbox wraps subprocesses in Linux Landlock (or bubblewrap where available), macOS Seatbelt, or a Windows restricted-token runner, with a file-effect policy that defaults to read-only and steps up to workspace-write or danger-full-access. Filesystem and subprocess providers “share one execution world,” so pointing them at a remote sandbox (an E2B provider ships in the tree) moves Bash, terminal, and language-server access with them.

On models, nothing ties dsh to DeepSeek. The provider catalog covers Anthropic, OpenAI, Bedrock, Vertex, and Azure alongside DeepSeek’s own endpoint, plus custom OpenAI-compatible gateways for anything else. The less expected feature is that dsh ships sub-agent providers that delegate work directly to Claude Code and Codex, resolving each product’s binary from the host PATH so the user supplies the install and the login; both are off by default. It reads AGENTS.md and CLAUDE.md, speaks MCP, supports the Agent Client Protocol, and ships bridges that run an existing Claude Code or Codex hooks.json against its own interception points.

The four shipped presets are where the benchmark story starts. Standard mode is the full coding agent. Code mode exposes the tools through a TypeScript SDK so the model writes one program instead of five round trips. Creator mode adds runtime inspection for building new presets. Minimal mode, in DeepSeek’s words, “keeps only a shell tool and a file editor for benchmarking models in a minimal environment.”

Inside minimal mode

The repository’s BENCHMARK.md is one paragraph long: follow the Python SDK guide, run the jsonrpc-agent minimal variant, use separate workspaces and session IDs per task. The composition it points at, examples/jsonrpc-agent/minimal.cordis.yml, is short enough to read in a minute.

The model gets one system prompt (default: “You are a helpful software engineer assistant.”) and exactly two tools: a persistent bash shell with a 300-second timeout and a str_replace_editor with a 16,000-character output cap. Context compaction is disabled. Harness identity, workspace prompt text, skills, one-shot Bash, task tools, and every other model-facing plugin are omitted. The sandbox policy is danger-full-access, so the docs tell you to run it only inside a disposable checkout or container. The default model in the example is deepseek-v4-flash with a one-million-token context window, and the SDK sample caps output at 49,152 tokens.

That is the environment behind the headline scores. The V4-Pro-0813 card says the code-agent tasks were run in this mode at the max reasoning effort level with temperature = 1.0, top_p = 0.95. Under those settings DeepSeek reports Terminal-Bench 2.1 at 87.9 (up from 72.1 for the preview) and DeepSWE at 62.7 (up from 12.8), with the model unchanged in structure from the preview: 61 layers, hidden size 7,168, 384 routed experts plus one shared, six active per token, a 1,048,576-token context, and DeepSeek’s recommended 384K maximum output at high and max effort. DeepSeek’s summary is that the release is “broadly competitive with the strongest proprietary models available.” The table on the same card does not put V4-Pro-0813 first on either agent benchmark: it lists Kimi K3 at 88.3 and Fable 5 (marked “w/ fallback”) at 88.0 on Terminal-Bench 2.1, and Fable 5 at 70.0 and Kimi K3 at 67.5 on DeepSWE. How those comparison columns were produced is not stated.

The bare two-tool setup is a defensible way to isolate a model from its scaffolding, and it is plausibly close to the interface the model was post-trained against, though DeepSeek does not say so. A model that scores 87.9 with one fixed prompt and two tool schemas may not score 87.9 inside Claude Code, Codex, or dsh’s own Standard mode with skills, planning, and sub-agents switched on. The card does not claim otherwise, but the number will travel without the footnote.

Where the outside scoreboards stand

The verified Terminal-Bench 2.1 leaderboard, where a Terminal-Bench team member reruns each submission, currently holds 17 entries. The top is Claude Code running Fable 5 at 83.8% ± 1.2%, dated June 7, then Codex with GPT-5.5 at 83.1%. There is no DeepSeek entry at all, and no entry from any vendor above 84. A verified 87.9 would sit four points clear of the board. The precedent for the gap between vendor charts and verified runs is recent: Meta published 80.0 for Muse Spark 1.1 and the verified entry landed at 76.2.

DeepSWE is more interesting. Datacurve, which runs the benchmark, publishes a leaderboard where “all models run on mini-swe-agent for consistency,” and its changelog says it added “DeepSeek v4 Pro results” on August 12. The entry, deepseek-v4-pro at max effort, shows 63% ± 6% at an average cost of $0.24 per task over 155 steps; deepseek-v4-flash, added August 6, shows 53% ± 4%. Both sit within rounding of DeepSeek’s own 62.7 and 54.4, on a scaffold DeepSeek did not write. The complication is the date. Datacurve’s run predates today’s GA by a day, and DeepSeek’s API keeps model names unchanged across checkpoints, so the leaderboard does not say which weights answered. If it was the 0813 checkpoint, DeepSWE has an outside corroboration on a different harness. If it was the preview, DeepSeek’s own 12.8 for that checkpoint is the number that needs explaining. Either way, Terminal-Bench, the score with the bigger claim attached, has no outside check at all.

What a reproduction would take

Everything needed to check the Terminal-Bench figure is now public except the compute bill and someone’s willingness to spend it. The composition file is in the repo. The model is on the API under its usual name and, for anyone with the hardware to serve 1.6 trillion parameters, on Hugging Face under MIT. The sampling parameters are on the model card. The Terminal-Bench 2.1 leaderboard accepts custom agents through its harbor runner with five trials per task, on the condition that submissions “may not modify timeouts or resources,” and a team member reruns the result before it is posted. A dsh minimal-mode agent submitted through that path, scoring anywhere near 87.9, would be the first independently verified DeepSeek entry on the board and would settle the question. A score in the high seventies would settle it differently, and would be worth knowing before choosing this model over the ones already verified there.

Temperature 1.0 across five trials leaves room for variance, so a single cheap run proves little either way, and reproducing the score with the same two-tool interface would still leave open how the model behaves inside the harness most people would use it in. Those are the terms on which the number can now be argued about at all, which is more than could be said on July 31, when the tool that produced it did not exist outside DeepSeek. Open-sourcing the harness converts a vendor claim into a testable one, and the test is now someone else’s to run.

Sources

Want to discuss this topic?

We'd love to hear about your specific challenges and how we might help.

Start a conversation