Claude Code + CodeRouter + Local LLM on a 16GB Laptop, The Next Move — Speeding Up Generation (MTP) and Offloading Heavy Tasks to Claude (External Agents)

Recap of the previous article (3 lines)

In the previous article (Claude Code + local LLM is genuinely usable even on a 16GB laptop), we squashed the issue where small models break Tool Calls using CodeRouter's repair layer, and built a setup where it "runs stably" using Ollama + CodeRouter. Even if you haven't read it, you can read this article on its own, but the premise is that you are in a state where "Claude Code is pointed at a local LLM via CodeRouter." From here on, it's about adding "speed" and "intelligence" on top of that foundation.

What we are doing this time — An overview

Pillar 0 and the two pillars of the foundation have different purposes and workloads for you. First, let's grasp the big picture with a table.

Hereinafter, the CodeRouter used in this article assumes v2.7.7 or later. If you haven't installed it yet, or if you are using an older version, please update it first.

uv tool install coderouter-cli          # 恒久インストール
uv tool update-shell
exec $SHELL -l
coderouter --version                    # → 2.7.7 以降ならOK

If you are a `uvx` user, you can pull the latest version at launch with `uvx coderouter-cli serve --port 8088`.

Now, let's start with Pillar 0.

Pillar 0: First, choose a model — Just the conclusion (details in another article)

As a foundation to maximize the effects of Pillar 1 (MTP) and Pillar 2 (external agents) that follow, let's decide on one model to run. If you are still using the "standard" models from last time (like Qwen2.5-Coder 7B), simply switching to a distilled model will improve accuracy by one level. Distilled models are models where the output or reasoning process of a large model (teacher) is learned by a small model (student). Models specialized for "agents," learned through traces of Claude-like models, tend to perform better in coding and Tool Calls than general-purpose models of the same size.

I will provide just the conclusion table for 16GB machines based on my actual benchmarks (llmbench, 40 coding tasks).

The premise is only two lines. Since the scores are actual measurements in an RTX5090 environment, please use them for judging whether it fits in 16GB and for relative comparison between models. Since the effective memory of a 16GB machine is about 6-8GB, GGUFs in the 6-7GB range with modest context are in the safe zone.

A detailed, serious comparison by category (coding / Tool Call / Japanese / ultra-lightweight / reasoning math), how to choose quantization, and the reasons for excluding others are all summarized in a separate article → Which is the best LLM by category for 16GB laptops — Distilled/9B class model serious comparison (July 2026 version)

What is important for this article is the connection from here on. GGUF variants with nextn (MTP) layers are distributed for Ornith-1.0-9B and Qwythos-9B, and the automatic detection in Pillar 1 below works as is(the one I confirmed MTP startup on my actual machine was also an Ornith-based 9B variant). In other words ── ① Choose a distilled model → ② If an MTP variant exists, take that one → ③ Place it in the launcher and it will automatically become faster. ① for accuracy, ② and ③ for speed. The mechanism for ② and ③ is Pillar 1.

Pillar 1: MTP / Speculative Decoding — Speeding up local generation

First, the theory — Why does it work "especially" on 16GB machines?

Speculative decoding is a speed-up method that reads ahead a few tokens with a small, fast "draft" and verifies them in bulk with the main model, confirming the ones that match all at once. Among these, MTP (Multi-Token Prediction) is a method that uses the nextn layer built into the main model's GGUF for the draft, and it is characterized by not requiring a separate draft model file.

llama.cpp switches this with the `--spec-type` flag.

The number of look-ahead tokens is determined by `--spec-draft-n-max` (default 3), and 2-3 is recommended for MTP. In llama.cpp PR #22673, there is a report that with 3 draft tokens, the acceptance rate is about 75% and decoding became more than twice as fast (measurements are mainly CUDA-based, and maturity varies by backend).

Here, people often think "MTP is for machines with plenty of VRAM," but there are benefits even for the 16GB class. In fact, structurally, the narrower the machine's bandwidth, the greater the relative room for improvement.

The reason is that local LLM decoding is bandwidth-limited. Every time a token is generated, the model weights are re-read from memory. If you verify 2–3 tokens at once using a draft, you can amortize that single weight read across multiple tokens. This is a structure where you gain an advantage from how memory bandwidth is used rather than from the computation itself, and this is exactly what applies to 16GB Macs and mini PCs.

Procedure — There are only 3 things you need to do

The CodeRouter v2.7.6 launcher handles MTP fully automatically. The 3-stage check for `mtp_mode="auto"` (default) is as follows.

1. Detect nextn in the main GGUF → If present, use `--spec-type draft-mtp`. No separate file required.

2. If not, automatically detect a companion GGUF from the same folder(name contains `mtp` / `draft`, size is less than 50% of the main model, matching arch, etc.) → If found, internally assemble `draft-simple` + `--model-draft`.

3. If neither, start normally.

Furthermore, if a spec launch initiated by auto fails abnormally, it will automatically restart once without the spec (a safety net for the "testing phase" mentioned later). In other words, it won't break even if you pick a model with immature support.

Because of this, the number of things the user needs to do is reduced to 3.

(a) Update llama.cpp ── This is the biggest trap

As a prerequisite for trying MTP, having a new version of llama.cpp is an absolute requirement. It goes without saying that you need a recent build that supports `--spec-type` flags, but there is a more obscure trap ── with older builds, nextn-embedded GGUFs don't just fail to use MTP, they fail to load the model entirely.

If you load a nextn-embedded GGUF with an old build, it will die with an error like this.

llama_model_load: error loading model: missing tensor 'blk.32.ssm_conv1d.weight'

In a 32-layer model with standard layers 0–31, `blk.32` is an additional block for MTP (nextn). An old build that doesn't know about nextn thinks this is a standard layer, goes looking for the tensor, and dies because it's "missing." The mean part is that you can't even see the letter M for MTP in the error message.

First, check for support.

llama-server --version                  # b99xx 台の新しめか
llama-server --help | grep spec-type    # draft-mtp が列挙されるか

If it's old, rebuild it. On Mac (Apple Silicon), Metal is enabled by default, no flags are needed.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build
cmake --build build --config Release -j $(sysctl -n hw.ncpu)

Always run `rm -rf build` before updating. The trap of rebuilding with old flags remaining in `CMakeCache.txt` is something you can fall into regardless of the OS.

(b) Choose a model where MTP is effective

There are two ways.

• Use a nextn-embedded GGUF variant. They often have `-mtp` in the name. The MTP variants of Ornith-1.0-9B or Qwythos-9B mentioned in column 0 are exactly this.

• Use a model with built-in nextn. GLM models (like GLM-4 MoE) have `{arch}.nextn_predict_layers` in their metadata, and if this is greater than 0, it is a model that "has its own draft head."

If you are using 16GB, it is safer to choose the built-in nextn type, which has smaller memory overhead than the separate draft model method (reasons for this are noted later).

(c) Launch normally with the launcher

All you have to do is select a model in the CodeRouter launcher and start it.You can leave the MTP field blank or set to 'auto'. The three-stage judgment above will run automatically. If the log shows this, the first stage has been hit.

[launcher] MTP: nextn layers (1) detected in main gguf → --spec-type draft-mtp

Here is one real-world example. When I used the **Ornith-based 9B MTP variant (nextn=1)** mentioned in Pillar 0, the detection hit at the first stage, `--spec-type draft-mtp` was added without needing a separate file, and it finished loading and started working normally in 12 seconds.

A note on 16GB — It's counterproductive if it hits swap

MTP is not magic that makes everything faster unconditionally.Both the GGUF and context memory usage increase by the size of the nextn layers (depending on the model size, this is anywhere from a few hundred MB to just under 1GB). If that causes it to hit swap, it defeats the purpose — what you added to gain bandwidth ends up calling for slower storage I/O, making it counterproductive.

The guidelines for using it on 16GB are simple.

• Choose built-in nextn types (the memory overhead is smaller than the separate draft model method).

• Keep ctx-size modest.

• Leave `--spec-draft-n-max` at the default of 2–3 and don't touch it.

The principle of "make it work in environments where it's effective, and don't break it in environments where it isn't" applies to performance as well. Start with settings that leave plenty of memory headroom.

A frank warning — Please treat MTP support as being in the "testing phase"

I want to emphasize this.Please treat MTP support as being in the "testing phase".

What is immature is not the CodeRouter launcher, but the maturity level of different architectures on the llama.cpp side. GLM-based models and some built-in nextn models work starting today. On the other hand, even if detection works 100% correctly, there are architectures that crash on the llama.cpp side at startup. In fact, for one community-made 12B (nextn=4) model, detection was perfect, but `llama-server` would SIGSEGV. Even after changing builds or flags, it would crash even with a clean startup without MTP — this was an area that the launcher couldn't save, as it was still under construction on the upstream side.

That is why CodeRouter has a safety net that if a spec-enabled startup triggered by 'auto' terminates abnormally, it automatically restarts once without the spec. Even if it crashes, it quietly returns to a normal startup.

[launcher] MTP startup failure detected (exit code -11); retrying without speculative decoding

In summary — it gets faster in environments where it works, and it doesn't break in environments where it fails (due to automatic fallback). So, please use it with the mindset that "there's no harm in having it enabled, but don't expect it to make every model faster." I have written about the background of this (the story of how detection hit but it couldn't start, build separation, and the automatic fallback design) in detail in my development diary → Detection hit but it couldn't start — Why you should treat MTP support in CodeRouter v2.7.6 as being in the "testing phase".

Pillar 2: External agent backend — Offload only heavy tasks to subscription Claude

The concept — Don't make a 7–9B model on 16GB do "everything"

Last time, the goal was to "run it stably locally." But honestly, it is too much to ask a 7–9B model running on a 16GB machine to handle everything, including design and difficult bugs. Running it locally feels great for daily light editing, refactoring, and explanation generation, but when it comes to situations where you really need to "use your brain," you still want the judgment capabilities of an Opus/Sonnet-class model.

That is where the v2.7.7 external agent backend (`kind: "agent_cli"`) comes in. The goal is to handle light tasks locally and only offload heavy parts (design/difficult bugs) to Claude (Opus/Sonnet) using a single CodeRouter to switch between them.

The mechanism is simple: register `claude -p` (Claude Code CLI's headless mode) as one provider. From CodeRouter's perspective, it just hits the external agent as a one-shot "box that outputs text when you input a prompt." Since it reduces to the same form as the LLM providers you usually handle, you can inherit fallback chains, profiles, guards, and cost tracking as they are.

This is where it's most effective ── Works with Pro / Max subscription OAuth = No API key required, zero pay-as-you-go costs. No API charges are incurred per request (it just consumes the 5-hour window of your subscription). This enables you to "call smart cloud models for critical moments without additional billing."

Configuration — Add one to providers

In `providers.yaml`, keep your existing local providers as they are and add just one more. A complete working example is included in the repository at `examples/providers-agent-cli.yaml`, so I will only show the core lines here.

providers:
  # --- 既存のローカル(前回組んだもの) ---
  - name: ollama-qwen-coder-7b
    model: qwen2.5-coder:7b
    # ...

  # --- 今回足す: 外部エージェント backend ---
  - name: claude-agent
    kind: agent_cli            # ← 新しい種類の backend
    model: sonnet              # CLI へ渡す --model。軽用途は sonnet 推奨(理由は後述)
    paid: false                # サブスク=従量ゼロ。有料ゲートの対象外
    capabilities:
      streaming: false         # 擬似ストリームのみのため明示
    agent_cli:
      agent: claude            # Phase 1a は claude のみ実装済み
      # 既定で read-only(FS を書き換えさせない)

The `paid: false` setting is the key. Since the Claude backend called via subscription OAuth has zero pay-as-you-go costs, `paid: false` is semantically correct. The paid gate (`ALLOW_PAID`) is a mechanism to stop "backends that charge per call," so the logic is that subscription backends should be excluded from the gate entirely.

The basic way to switch between them is to separate profiles and switch using the `X-CodeRouter-Profile` header. Prepare a "local profile for daily use" and a "profile with claude-agent at the front for heavy tasks," then swap the header on the request side.

# 重い作業だけ、claude-agent を含む profile を指名する
curl -s localhost:8088/v1/chat/completions \
  -H "X-CodeRouter-Profile: claude-agent" \
  -d '{"model":"claude-agent","messages":[{"role":"user","content":"..."}]}'

Cost measurement — Approx. 25k prompt_tokens for "What is 1+1?"

I will show the numbers honestly here. This is an actual measurement (blurred) of just throwing the 4 characters "What is 1+1?".

# → "usage":{"prompt_tokens":25657,"completion_tokens":25},
#    "coderouter_cost_usd":0.2227

Prompt_tokens are about 25,000. You are consuming over 25,000 tokens for just 4 characters. The reason is clear: when you run `claude -p`, the entire set of Claude Code system prompts(tool definitions, various instructions, environment information, etc.) is loaded in its entirety.

In terms of cost, it is equivalent to $0.2 per call. Since it is subscription-authenticated, the charge is zero, but the Claude Code subscription has a 5-hour window rate limit, and this call definitely eats into that quota. "Zero charge" and "free" are two different things.

However, there is a silver lining. From the second time onwards, prompt caching kicks in, and the equivalent cost drops to about 1/4 of the first time ($0.22 → $0.05). The more you call it in the same session, the more the system prompt portion is amortized.

The conclusion from these numbers is clear.

• The external agent backend is heavy equipment to be used for "critical moments". It is not suitable for use as a calculator or for rapid-fire chatting.

• For light tasks, point to `model: sonnet`(this is why it is set to sonnet in the sample above). Prepare a separate profile that points to opus only for situations where judgment is truly required.

Honest status of verification

What has been confirmed on actual hardware is two systems: macOS and Ubuntu (Linux).

Also, the implemented backend is currently **`claude` only (Phase 1a)**. `codex` / `gemini` / `grok` are planned for future addition, and at this moment, if you write them in the configuration, it will be clearly rejected as "not implemented."

Authentication Tips

• macOS / Linux desktop: If you have already run `claude /login`, `claude-agent` will work as is. No additional token configuration is required.

• Headless server / container: Since there is no screen, issue a token with `claude setup-token`, place `CLAUDE_CODE_OAUTH_TOKEN` in `.env`, and forward it to the child process using `passthrough_env` on the CodeRouter side. This path is considered unverified, so please let me know if it works.

Quick Tips — Automatic Launcher Sync and Prompt Caching

Just two short points.

Launcher provider auto-sync (v2.7.4+). When you launch a model via the launcher, that model is automatically registered for routing. The hassle of manually editing `providers.yaml` to "match the launched model with the routing configuration" as before is gone. If you launch a model with MTP from the launcher as described in Pillar 1, it will connect directly as a backend for Claude Code.

Prompt caching effectiveness. The "1/4 cost from the second time onwards" mentioned in Pillar 2 is a general benefit that applies beyond just external agents. The more you use the same premises (system prompts or long context) repeatedly, the lower the actual consumption becomes — it is more advantageous the more you use it within a continuous work session.

Summary — What to do first (Priority Table)

I introduced one foundation and two main pillars, but you don't need to do everything at once. Starting with one thing that is "bothering you the most right now" is enough.

Here is the recommended order. First, Priority 1 (llama.cpp update) will be necessary sooner or later if you want to try new models, regardless of whether you use MTP or not. It is easier to clear the hard-to-diagnose `missing tensor` trap first.

Next is Priority 2 (distilled models). Just by downloading one GGUF file and swapping it out, you get a verified accuracy improvement in actual benchmarks. This is the most cost-effective move this time. Priority 3 (MTP) follows that; if the chosen model has an MTP variant, just pick that one. There is almost no risk because it won't break due to automatic fallback even if it doesn't work.

Priority 4-5 (external agents) should naturally be added once you start feeling that "this is tough for a 7-9B model" while running everything locally. If you throw everything at Opus right away, you will quickly use up your 5-hour window, so start with Sonnet. Once you get used to it, separate the Opus profile.

What they all have in common is that they are tools that you "install and let work when needed". Distilled models raise the baseline, MTP speeds things up in environments where it can, and external agents borrow a smarter model only when you need more brainpower. Within the 16GB constraint, stretching what you can without overdoing it — this is the view beyond the "stable operation" of last time.

Links

• Previous article (prerequisite for this article): Claude Code + Local LLM is seriously usable even on a 16GB laptop — Until Tool Calls stabilize with CodeRouter (July 2026 version)

• Details of Pillar 0 (Category-by-category model comparison): Which is the strongest LLM by category for a 16GB laptop? — Distilled/9B-class model head-to-head comparison (July 2026 version)

• Behind the scenes of the MTP "test phase" (Development Diary): Detection was correct but it couldn't launch — Why you should consider CodeRouter v2.7.6's MTP support to be in a "test phase"

• CodeRouter (GitHub/MIT License): https://github.com/zephel01/CodeRouter

CodeRouter is a personally developed OSS (MIT) that runs with `uv tool install coderouter-cli`. If these steps were helpful, GitHub stars help determine the priority for the next release. Feedback from actual machines like "It worked/didn't work like this on 16GB" is also welcome in Issues/Discussions. Now, have a great AI coding life.
Also, feel free to ask questions here or in the question box.