Back to Insights
AI Automation

Nvidia's Nemotron Pitch Is a Router That Stops Paying Frontier Prices for Easy Subtasks

Nemotron 3.5 Lightning is a 30B MoE with 3B active, but the open NeMo Switchyard router beside it is Nvidia's real product. How it decides, where it fails.

S5 Labs Team August 11, 2026

Nvidia released Nemotron 3.5 Lightning on August 11, a 30-billion-parameter mixture-of-experts model that activates about 3 billion parameters per token, and alongside it NeMo Switchyard, an open-source router that decides, turn by turn, whether an agent’s next call goes to a cheap model or an expensive one. The launch blog leads with the model, but the number that matters sits in the second half: LangChain ran 145 multi-turn agent tasks through Switchyard and reports 74% lower cost by sending only 7% of calls to a frontier model.

Read the two releases together and Nvidia is saying that most of the tokens an agent burns do not need a frontier model, and it is shipping the plumbing to act on that. The conclusion is correct; the evidence attached to it is thinner than the headline figures suggest, and the router’s own documentation is candid about where it will send the wrong request to the wrong model.

What the model is

The Hugging Face card describes Lightning as a hybrid: interleaved Mamba-2 and MoE layers with a few attention layers, 30B total and 3B active, pretrained on more than 20 trillion tokens with an NVFP4 recipe. Context runs to 1 million tokens, though Nvidia’s single-H100 guidance uses 256K. Post-training data stops at May 2026. The license is OpenMDW-1.1, the Linux Foundation’s permissive model license, with no user threshold or acceptable-use rider.

Two checkpoints ship. The BF16 build is the reference weights, meant for customization and post-training. The NVFP4 build is the one meant to run: Nvidia lists a single DGX Spark (GB10), a single H100, an RTX 5090, or Ampere cards via W4A16, and names the intended use as long-running agents, sub-agent workhorses and local inference. Three speculative-decoding drafters come with it (DSpark, DFlash and multi-token prediction), and that is where the “up to 4x output speed” claim lives; Nvidia does not say which model or hardware the 4x is measured against. The PinchBench claim is more specific: 86% accuracy while completing 10,000 tasks 30% faster than Qwen3.6 35B A3B at similar accuracy.

Read the rest of the benchmark table before you plan around “frontier-level accuracy,” a phrase Nvidia uses freely. Against Qwen3.6 35B A3B, the closest peer, Lightning trails on 12 of 14 rows on Nvidia’s own card: 51.56 vs 70.12 on SWE-bench Verified, 24.58 vs 44.38 on Terminal-Bench 2.1, 85.37 vs 88.07 on PinchBench in the card’s table (the blog rounds Lightning’s score to 86%). Artificial Analysis clocks it at a median 299.7 tokens per second across hosted providers, second of 135 open-weight models it tracks, at $0.07 per million input tokens and $0.22 per million output. That is the profile of a fast, competent execution model, which is the job Nvidia is assigning it, and not one you would put in charge of planning.

Locally, one blogger’s RTX 5090 test with 4-bit builds had Lightning at roughly 123 tokens per second on short tasks against 76 for Meta’s Muse Glimmer 30B, and 138 versus 20 with a 40,000-token document in context. His Lightning GGUF was 25 GB, past the 24 GB card that Glimmer’s 17 GB build fits, so the local floor is a 32 GB card or a Spark. That is one tester and one run per task.

What Switchyard actually does

Switchyard is a Rust proxy and library on GitHub under Apache 2.0, and the README calls it pre-alpha software that is not for production use. It accepts OpenAI Chat Completions, OpenAI Responses and Anthropic Messages, translates between them, and forwards to vLLM, NVIDIA NIM, Ollama or any OpenAI-compatible endpoint, so the agent keeps its native API whichever model serves the request. Configuration is a TOML file with three layers: LLM clients, targets (named models on those clients) and routes (the algorithm that picks a target). You name every model in the pool yourself.

The routing algorithms are the substance. The LLM classifier sends each request to a judge model first, which returns p_solve, its estimated probability that the weak model completes the task, plus a capability boundary of supported, uncertain, unsupported or unmatched. Switchyard routes to the weak target when p_solve clears a threshold you set (base_threshold, stepped up for each riskier boundary class), and defaults to the strong model when the verdict does not parse. Nvidia’s recommended confidence threshold of 0.5 was calibrated on a 75-task Python slice of SWE-Bench Pro. The stage router skips the judge and reads evidence already in the conversation: error severity, “spinning” without reads or writes, and prolonged exploring push a turn to the capable model; recent writes and edits, especially after tests pass, keep it on the efficient one. The escalation router runs every turn on the weak model first, has a judge read the completed turn, and after two consecutive escalate verdicts (the default) latches the session to the strong model and discards the weak reply. A random splitter, an advisor-gate mode and a tunable “prefill router” trained on a model’s residual stream round out the list.

What you have to configure, then, is everything that matters: the model pool, which model is weak and which is strong, which judge runs the classifier, and the threshold that sets how much risk of a bad cheap answer you will tolerate. Switchyard writes each decision and its rationale into response headers, so the audit trail exists. It fits the pattern in our agentic architecture piece: a frontier model for planning, cheaper models for execution, and a routing layer between them that someone has to own.

Where the case-study numbers come from

Every cost figure in Nvidia’s blog is either Nvidia’s own internal benchmark or supplied by an adopter, and none had been independently reproduced at launch. Nvidia’s headline claim, that Switchyard “reduces task completion cost to nearly one-third of Opus 4.8 alone” while holding frontier accuracy, does not name the workload. Ramp’s 58% lower cost and 33% shorter runtime are on an internal “Ramp SWE-Bench,” and Nvidia says Ramp matched frontier performance without publishing the accuracy figures. Boomi’s 100% domain-routing accuracy describes a classifier correctly sorting traffic across five domains, with 59% of it going to a 5x faster fine-tuned model and later-turn latency down 21%; that is a domain-dispatch result, not a difficulty-routing one, and the coverage conflates the two.

LangChain published its methodology, and it is the most useful document in the launch because it shows what the headline hides. The suite is 145 Deep Agents tasks averaging 6.3 model calls each, run five times. Opus 4.8 alone scored 86.0% at $11.45 per run. Routed between Opus and Lightning with the escalation router and Gemini 3.1 Flash Lite as judge, the system scored 80.0% at $3.00. Lightning alone scored 77.7% at $0.72. So the 74% saving came with a six-point accuracy loss, and the router bought 2.3 points over the small model on its own at four times the cost. The 7% of calls that went to Opus consumed 68.4% of the spend and the judge another 21.2%. Cost swung 67% between runs on an identical configuration. LangChain says outright that this is one workload, not a forecast.

None of that makes the numbers wrong, but it does mean a vendor and its launch partners chose the benchmarks and the framing, and published the ratio without, in most cases, the accuracy delta. It is the pattern we flagged in the production cost breakdown: list price per token is the small part of the bill, and routing, judging and retry overhead is where budgets move.

Where routing breaks, and how to check it on your own traffic

The failure modes are visible in the router designs themselves. A classifier route guesses difficulty before the work starts, and p_solve is a judge model’s estimate, not a measurement; a hard subtask that reads as easy gets a cheap answer that may be confidently wrong, and nothing in that path catches it unless the next tool call errors out. The escalation route catches problems only after the weak model has visibly failed, so a misroute costs a wasted turn, a judge call and the strong-model retry, and by default it takes two bad turns to latch. The stage router needs tool-result history to work at all; Nvidia’s docs say it has nothing to read on a pure chat workload. Boomi’s 100% is a reminder that classifying domain is far easier than classifying difficulty, and most of the savings depend on the second.

Then there is drift. The recommended 0.5 threshold was calibrated on Python SWE-Bench tasks, and your traffic is not that. Swap in a fine-tuned Lightning, add a model to the pool, or let the frontier model shift under a version bump, and every p_solve estimate and calibrated threshold is stale without anyone touching the config. The threshold itself is a business decision that happens to live in a config key: whoever sets base_threshold is deciding how many wrong cheap answers a customer sees per thousand tasks, and in most shops that is an engineer asked to cut the bill.

The way to test Nvidia’s figures is to run LangChain’s design on your own tasks. Take a sample of real agent traces, at least the 40 to 75 hard tasks and 20 easy ones Nvidia’s calibration guide asks for, and run three arms: frontier only, Lightning only, and routed. Record accuracy and cost per run, repeat it, and log the routing rationale from the response headers so you can see which decision source produced each misroute. Sort outcomes into the two cells that matter (tasks the frontier failed and the small model passed, and tasks the frontier passed and the small model failed) and set the threshold where it protects the second cell without giving up the first. If Lightning alone lands within a couple of points of the routed arm, as it did for LangChain, the question is whether the router pays for its own judge cost or whether you should run the small model and escalate by hand on failure.

That last possibility is the real concession in this launch. Nvidia built a 3B-active model that runs at 300 tokens per second and is plainly not frontier, then built the router because it knew customers would not switch wholesale. Both moves say the same thing: the company that sells the GPUs frontier models run on has concluded that most agent tokens should not touch one. The case studies were chosen to make that look easy; the conclusion holds even on traffic nobody picked to flatter it.

Sources

Want to discuss this topic?

We'd love to hear about your specific challenges and how we might help.

Start a conversation