Two open-source inference runtimes shipped breaking releases three days apart. llama.cpp published v1.0.0 on November 13, together with ggml v1.0.0, and committed both libraries to semantic versioning. The release removes 36 functions from llama.h, tightens several llama-server defaults, and adds an FP8 KV cache and a GPU cache for mixture-of-experts weights held in host memory. vLLM published v0.34.0 on November 16, with 1,061 commits from 412 contributors. It removes Model Runner V1, raises the default block size from 16 to 64, and by default sizes the KV cache from free GPU memory, with gpu_memory_utilization at 1.0. llama.cpp v1.0.0 · ggml v1.0.0 · vLLM v0.34.0
What follows is read from the release notes, the pull requests they cite, the source at each tag and GitHub issues. S5 Labs has not built, run or benchmarked either release. Every performance figure is a project’s or a PR author’s own, and we found no third-party benchmark of vLLM 0.34.0 against 0.33.x.
llama.cpp v1.0.0
What the version number now promises
The Hugging Face post announcing the release says: “Moving forward, both llama.cpp and ggml will reflect API and ABI incompatibilities with major-version bumps.” Its rules apply to “all ggml-org projects using semantic versioning”:
| Bump | Meaning in the post |
|---|---|
| MAJOR | “Breaking change to the public C API in ./include” |
| MINOR | “Backward-compatible feature, backend updates, model support, or API addition” |
| PATCH | “Bug fix with no API change” |
The promise covers the headers in ./include for two libraries, libllama and libggml. Backend updates and model support are minor-level changes, so output and performance can still move between 1.x releases. The table does not mention command-line flags or server routes, which we read as leaving them outside the rule.
For packagers the practical effect is a shared-library version. At the v1.0.0 tags, CMake sets SOVERSION to the major version for both libraries. In the project’s versioning thread, Debian maintainer ckastner reported on November 15 that “the major version update worked well” for standalone ggml, that Debian “will now start shipping libggml1”, and that it can be installed alongside libggml0. That update did not yet cover llama.cpp itself, and this is a packager’s account, not the project’s. ggml discussion #1579
Numbered b builds continue, with more than 20 tagged on November 17 alone. The v1.0.0 tag is not the same commit as b12359, the nightly its release page names: it is that build plus one CUDA commit (#32179).
The removals that bindings have to absorb
The release notes give the API break one line: “remove llama_set_warmup, 35 deprecated functions, defrag_thold and --defrag-thold”. They do not name the functions. PR #30565, merged November 12, does. Counting the removed lines in its llama.h diff gives 36 functions, the 35 plus llama_set_warmup.
| Removed | Use instead |
|---|---|
llama_load_model_from_file, llama_free_model | llama_model_load_from_file, llama_model_free |
llama_new_context_with_model | llama_init_from_model |
llama_n_ctx_train, llama_n_embd, llama_n_layer, llama_n_head | llama_model_n_ctx_train and the matching llama_model_n_* names |
llama_n_vocab | llama_vocab_n_tokens |
llama_get_state_size, llama_copy_state_data, llama_set_state_data | llama_state_get_size, llama_state_get_data, llama_state_set_data |
llama_load_session_file, llama_save_session_file | llama_state_load_file, llama_state_save_file |
llama_token_get_text, _get_score, _get_attr, _is_eog, _is_control | the matching llama_vocab_* names |
llama_token_bos, _eos, _eot, _cls, _sep, _nl, _pad | the matching llama_vocab_* names, except _cls: llama_vocab_cls is removed too (see below) |
llama_add_bos_token, llama_add_eos_token | llama_vocab_get_add_bos, llama_vocab_get_add_eos |
Six llama_token_fim_* functions | llama_vocab_fim_* |
llama_vocab_cls (the deprecated duplicate) | header comment: “use llama_vocab_bos instead” |
llama_sampler_init_grammar_lazy | llama_sampler_init_grammar_lazy_patterns |
llama_set_warmup | none; “user code should do warmup runs manually” |
The same PR removes the defrag_thold field from llama_context_params and the -dt, --defrag-thold option, renames llama_set_embd_state_src to llama_set_ctx_state_src, and moves the NextN and speculative-decoding functions from the internal staging header into llama.h. The PR leaves one item unchecked: “Deprecate llama_batch ?”.
Other breaks sit outside llama.h. The multimodal library drops clip.h (#31865). ggml pruned its own deprecated API and the legacy DIAG_MASK_INF and DIAG_MASK_ZERO ops, “bumping the RPC protocol to reject incompatible peers”. At the v1.0.0 tag, ggml-rpc.h sets the protocol’s major version to 10. We infer that RPC hosts should be upgraded together; the notes do not describe mixed-version behavior.
Downstream, llama-cpp-python v0.3.37 keeps the removed names as Python wrappers that warn and forward to the replacements, with llama_set_warmup as a no-op, so its users are shielded from the renames. It vendors a llama.cpp snapshot 27 commits past v1.0.0 and binds one function, llama_model_dflash_variant, that is not in the v1.0.0 header. Going by the loader code, its Python layer would likely fail at import against a separately installed stock v1.0.0 libllama. We did not run that, and the wheel’s bundled library is unaffected. llama_cpp.py at v0.3.37
Ollama’s latest releases, including the v0.40.16 that added Ripple-65B Preview, predate v1.0.0, and as of November 17 we found no public statement about it from Ollama or LM Studio. Ollama releases A model that needs a vendor fork of llama.cpp, as Bonsai 2 does, gets these changes only when the fork catches up. Upstream, v1.0.0 adds text support for Kimi-K3.1-Pro and Ripple-65B-Preview. For the week’s other large open-weight releases and what they need to run, see our piece on Inkling 2 Small, Sage S2.1 Flash and MiniMax-H3-Image.
Server defaults got stricter
The release notes file a group of server changes under “security:”. Their author, maintainer ngxson, describes the two changes in PR #32113 differently: “None of those are actually vulnerability, but since people nitpick this too much, we now forced to hurt the UX a bit”. We found no CVE or GHSA identifier for any of them. They can still break a working setup:
| Situation | v1.0.0 behavior | Opt-out |
|---|---|---|
| Tools, MCP servers or the MCP proxy on a non-loopback address without an API key | Refused | --allow-unauthenticated-tools, “only for trusted networks” |
Media URLs such as image_url that point to local or private addresses | Refused | --media-allow-private-urls |
The web UI’s CORS proxy (--ui-mcp-proxy) reaching localhost or LAN MCP servers | Refused | None; the PR says “no escape hatch” |
--cors-origins with tools, agent mode, MCP server definitions or the proxy enabled | Defaults to localhost instead of * | Pass --cors-origins explicitly |
Server README at v1.0.0 · PR #32115
For any --cors-origins value other than *, PR #32098 lists what the server now rejects: “foreign or null origins, and cross-site or same-site requests without an Origin header”. The notes also say HF_TOKEN is now restricted to huggingface.co hosts.
Being strict about what counts as local is the right default. GPUStack’s exploited auth bypass, which we covered last week, turned on a remote request being treated as local. The cost falls on setups that reached localhost or LAN MCP servers through the web UI’s proxy, which now need another route.
Memory features arrive with few numbers
FP8 KV cache. -ctk and -ctv now accept f8_e4m3, with f16 still the default. PR #28392 describes support for “OCP-FP8-based KV-caches” with optional static per-tensor scales, and the ggml notes list OCP FP8 support on CUDA. Neither publishes a memory or quality measurement. An 8-bit type stores half the bytes of f16 per element, which is arithmetic from the type names and not a project claim about total savings.
MoE expert cache. --moe-cache-mib sets a “GPU cache size in MiB for the MoE experts kept in the CPU”. It defaults to 0, so it is off until you size it. The only numbers are in PR #30966: a tuning-script demo on one machine with six calibration runs, one of which measured 29.6 tokens per second at a 63.4% hit rate with a 6,464 MiB cache. The PR’s suggested setting, 24,000 MiB for a predicted 83.0 tokens per second, lies beyond its largest measured cache of 19,392 MiB, so treat it as an extrapolation.
vLLM 0.34.0
The notes open with a “Before you upgrade” table. Condensed, with details from the full changelog and the source:
| Change | What the notes say to do | Detail |
|---|---|---|
| Model Runner V1 removed | Remove VLLM_USE_V2_MODEL_RUNNER=0; port V1 logits processors to vllm.v1.worker.gpu.sample.logits_processor | Any value but 1 fails at startup |
Default --block-size 64, was 16 | Pass --block-size 16 | ROCm paged attention still uses 16; an explicit value wins |
Extensible KV cache on; gpu_memory_utilization defaults to 1.0 | --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache | CUDA and ROCm only |
ROCm 7.2 wheels and -rocm72 images dropped | Use ROCm 10.0 wheels and images | |
| Dependency floors | huggingface_hub >= 2.0.0, tokenizers >= 0.23.3, timm >= 1.0.30 | Also transformers >= 5.16.1, < 5.23.0; PyTorch stays at 2.14.0 |
fp_quant and Marlin W4A8 removed | fp_quant checkpoints no longer load; use --quantization humming | Two open issues about humming, #68194 and #68195, were filed November 17 |
The two memory opt-outs are not equivalent
With the extensible cache, vLLM sizes the KV cache “from memory actually free after CUDA graph capture and warmup, not from an estimate”. The notes offer the two flags above as alternative ways “to keep the old sizing”. The source shows they do different things. vllm/config/cache.py
The old default was 0.92, not 0.9: that is the value in cache.py at both v0.32.0 and v0.33.0. At v0.34.0 an explicit value wins. With none set, the default is 1.0 when the extensible cache is on and 0.92 when it is off.
So --no-enable-extensible-kv-cache alone restores both the estimate-based sizing and the 0.92 default. --gpu-memory-utilization 0.9 keeps the extensible cache on and caps it, slightly below the old default. The notes add that a value between 0.8 and 1.0 that withholds 1 GiB or more logs a startup warning. A deployment that relied on the old default to leave about 8% of the card free loses that margin unless it sets a value.
vLLM’s own figures show a trade-off. It reports the MiniMax-M3 cache on H200 growing from 433k to 672k tokens, which it says raises throughput and lowers time to first token at high concurrency. It also reports time per output token rising 6% to 18% “on some H200 models at concurrency 128”.
Early reports from a 7.62 GiB laptop GPU
Two issues filed on November 17 by one reporter describe startup failures on an RTX 4070 Laptop GPU with 7.62 GiB. Both were run on main commits and not the v0.34.0 tag, both were prepared with AI assistance, and neither is confirmed by a maintainer.
In issue #68184, a 4B NVFP4 model at a 4,096-token maximum length fails on defaults because “0.89 GiB KV cache is needed, which is larger than the available KV cache memory (0.27 GiB)”. With the extensible cache turned off at 0.92 utilization, the same setup starts with a 2.3 GiB cache. The error message does not mention that flag.
In issue #68209, the opt-out path fails instead. The reporter says CUDA graph memory profiling builds a throwaway KV cache counted in blocks, and with 64-token blocks a Qwen3-1.7B model plus a Qwen3-0.6B speculative draft asked for 3.50 GiB and ran out of memory. A commenter has volunteered to take it, and there is no fix yet.
Smaller breaks and loose ends
llama3_jsontool arguments no longer stream incrementally, and corrupt or oversized images now return 400 instead of 500.- The NIXL connector now rejects mismatched push handshakes. Pinning
--block-sizeexplicitly in a mixed-version prefill/decode fleet is our inference, not the notes’ advice. - The notes say Kimi K3.1 groundwork landed but “K3.1 is not servable in this release”. The vLLM recipe cited in our Kimi K3.1 Pro article names 0.34.0+ as its baseline, so check the recipe again before planning around the tag.
- The release body still contains reviewer TODO comments, one asking whether the XPU install line resolves. Verify that line before relying on it.
- The security section lists four items and no CVE or GHSA identifiers. As of November 17, no vLLM advisory is dated on or after November 10. vLLM advisories
- The headline speedups, such as up to 1.95x decode for Step5 on 4×GB200, are vLLM’s own, and we found no independent measurement of them.
What to pin and test
Anything linking libllama. Build against the v1.0.0 headers, fix the errors using the table above, and pin to SOVERSION 1. With llama-cpp-python, use the library the wheel bundles, and test the import before pointing v0.3.37 at a system v1.0.0 libllama.
llama-server integrations. Retest browser clients on another origin, tools exposed beyond loopback, media URLs on private addresses and MCP servers reached through the UI proxy. Add the opt-out flags only on networks you trust.
vLLM memory settings. Write --gpu-memory-utilization, --block-size and the extensible-cache flag into your deployment config so a default cannot move them again. Compare KV cache capacity at startup before and after, and measure time per output token at your real concurrency. On small or shared GPUs, test startup with the extensible cache on and off.
Environment. Move ROCm 7.2 hosts to ROCm 10.0, check the dependency floors against your lockfile, and remove VLLM_USE_V2_MODEL_RUNNER=0 from every launch script.
The llama.cpp API break is the easy part, because a removed function fails at compile or load time. vLLM’s memory defaults and llama-server’s new refusals appear only at runtime, as different capacity, a refused request or an out-of-memory error at startup. Those need a staging run first, and vLLM 0.34.0 is the release we would not take unattended.
Sources
llama.cpp and ggml
- llama.cpp v1.0.0 release notes, ggml v1.0.0 release notes — ggml-org on GitHub
- llama.cpp v1 announcement — Hugging Face blog
- Versioning discussion #1579 — ggml-org on GitHub
- PR #30565, API prune, PR #28392, FP8 KV cache, PR #30966, MoE cache, PR #32098, PR #32113, PR #32115 — ggml-org on GitHub
- Server README at v1.0.0 — ggml-org on GitHub
llama_cpp.pyat v0.3.37 — llama-cpp-python on GitHub- Ollama releases — Ollama on GitHub
vLLM
- vLLM v0.34.0 release notes,
vllm/config/cache.pyat v0.34.0 — vLLM on GitHub - Issue #68184, Issue #68209, Security advisories — vLLM on GitHub
