I tried NVIDIA's new LLM routing infrastructure NeMo Switchyard

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

I had just published two articles about running NVIDIA LLM Router on DGX Spark, when immediately afterward I was told that "a new routing platform has been released that also incorporates the LLM Router algorithm." NeMo Switchyard v0.1.0 was released on July 1, 2026, Japan time.

As someone who was preparing a continuation of those articles, I have mixed feelings, but as I actually tried it out, I kept discovering that "features I had struggled to build myself for LLM Router are already included from the start." In this article, I'll run Switchyard on my Mac and DGX Spark, and verify whether the pitfalls I encountered during LLM Router testing have truly been resolved.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3/

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3-training/

What Is NeMo Switchyard?

It is a routing proxy that distributes LLM traffic, published under the NVIDIA-NeMo organization on GitHub. It's a Python package installable via pip install nemo-switchyard, with a two-layer structure where Python wraps a Rust core built with maturin. The license is Apache 2.0, the version is 0.1.0, and the development status is explicitly listed as Alpha.

https://github.com/NVIDIA-NeMo/Switchyard

A documentation site is also available.

https://nvidia-nemo.github.io/Switchyard/

I had an opportunity to ask NVIDIA about the relationship with LLM Router, and received the response that it is not a simple successor, but rather "a more formal product that encompasses various routing technologies and the infrastructure to host and improve them." The algorithms from the LLM Router Blueprint are also currently being ported to Switchyard.

For those who have used LLM Router, here is a table comparing the differences in character.

I think the two major differences are "no trained router required" and "protocol conversion is built-in." LLM Router was designed to train a custom classifier that passes Qwen embeddings through PCA and MLP. Switchyard replaces that with signals obtained from LLM queries and agent tool execution history. Since GPU is no longer required, it runs as-is on a Mac.

Routing: Choose from 4 Methods

The documentation covers 4 routing methods.

llm-routing summarizes the last 4 turns of conversation, passes it to a classifier model, categorizes it into 4 categories — simple / medium / complex / reasoning — then maps to weak / strong tiers. It uses tool calling for classification, and uses a fail-open design where it falls back to the default tier if confidence falls below a threshold or classification fails. Three built-in classification policies are available: general, coding_agent, and openclaw, and it's interesting that policies for coding and for a resident assistant are prepared in advance.

cascade is even more sophisticated, evaluating tool execution result signals — such as error severity, test pass/fail, and number of file edits — across 3 layers. It first makes immediate decisions for clear-cut situations (critical errors go to strong, finishing work with all tests passing goes to weak), then uses weighted scores, and only consults the LLM classifier when it can't be confident. The only dial the user touches is confidence_threshold, and the recommended value of 0.5 is stated to have been calibrated on SWE-Bench Pro.

From Installation to Serve

Python 3.12 or later is required. I used uv to set up the environment this time.

uv init switchyard-handson && cd switchyard-handson
uv add "nemo-switchyard[server,cli]"

Wheels are available for Linux x86_64 / aarch64 as well as macOS arm64, so it installs directly on Apple Silicon Macs.

The configuration has a 3-layer structure: endpoints (provider connections), targets (upstream models), and profiles (routing policies presented to clients). I assigned GLM-5.2 to the strong tier and DeepSeek V4 Flash to the weak tier. The classifier that handles routing decisions references the same target as weak. I actually made a costly mistake once when choosing the classifier model, and the current configuration reflects that lesson — but I'll cover the full story later.

endpoints:
  openrouter:
    base_url: https://openrouter.ai/api/v1
    api_key: ${OPENROUTER_API_KEY}

targets:
  strong:
    endpoint: openrouter
    model: z-ai/glm-5.2
    format: openai
  weak:
    endpoint: openrouter
    model: deepseek/deepseek-v4-flash
    format: openai

profiles:
  fast:
    type: passthrough
    target: weak
  smart:
    type: llm-routing
    profile_name: coding_agent
    strong: strong
    weak: weak
    classifier: weak

There were only 2 stumbling points. The profile type name uses hyphens: random-routing, while random_routing (with underscore) that appears in quickstart examples is a separate system for the old route bundle format. Also, classifier in llm-routing takes the target ID as a string. Both gave error messages that explicitly told me what was expected — "expected one of strong, weak, ..." and "expected a string" — so I could fix them immediately. Incidentally, passing the same ID as weak to the classifier isn't laziness; defining two targets with the same model causes a duplicate registration error. The approach is to have one target referenced by both the routing role and the response role.

The naming difference between these two formats wasn't noted in the quickstart, so I submitted a PR upstream to add a note to the documentation.

https://github.com/NVIDIA-NeMo/Switchyard/pull/19

Starting the server is a single command.

uv run switchyard serve --config profiles.yaml --port 4000

This exposes all three APIs — OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses — on the same port. Even if the client-side format differs from the upstream format, they are mutually converted through an internal intermediate representation.

/v1/models Returns 3 Types of IDs

When you first call GET /v1/models after starting serve, it gives you the clearest picture of Switchyard's design philosophy. The returned model list contains IDs of different kinds coexisting.

{"id": "smart", "display_name": "llm-routing", ...}
{"id": "strong", "display_name": "z-ai/glm-5.2", ...}
{"id": "z-ai/glm-5.2", "display_name": "target strong", ...}

Specifying the profile ID (smart) in model activates routing, while specifying a target ID (strong) or an upstream model name bypasses routing and fixes to that model. In other words, clients can choose between "I want it routed" and "I want this specific model" simply by which model name they use.

Seeing this, I found myself with a distant look in my eyes. The biggest obstacle in the verification I had been preparing as a practical follow-up to the LLM Router series was exactly this.

Has the LLM Router's "Model Name Ignored" Issue Been Resolved?

LLM Router had a specification where it didn't look at the model field in the request body. Even if a client explicitly specified model: claude-opus-4-8, it would be overwritten by auto routing, making it incompatible with clients like Claude Code that use different model names per task type. In verification, I had worked around this by applying a model name bypass patch to my fork.

I ran the same verification against Switchyard. I sent 5 variations of the same prompt "Say OK only." to the llm-routing profile, changing only the specified model name.

When routing is desired, routing occurs; when a fixed model is desired, it is fixed. Behavior that required modifying forked code in LLM Router is officially supported from the start. For this single point alone, I think it's worth switching over.

One-Command Claude Code Connection with switchyard launch claude

Claude Code is an agent dedicated to the Anthropic API, so connecting it to an OpenAI-compatible routing proxy traditionally required inserting a conversion proxy like CCR (Claude Code Router). Since Switchyard has Anthropic Messages conversion built in, this becomes a single command.

switchyard launch claude

A proxy starts on an available port, and Claude Code launches with environment variables like ANTHROPIC_BASE_URL swapped out. The default configuration is a verified trio of Claude Opus 4.7 (strong), Kimi K2.6 (weak), and Gemini 3.5 Flash (classifier), with a status footer at the bottom of the screen showing real-time request counts and token counts per tier. When you send simple instructions versus heavy queries, watching the numbers in the footer reflect the routing is quite satisfying.

When I asked within a session "What model are you currently running on?", it returned the route ID switchyard-deterministic-.... Claude Code itself has no idea it's running in a routing configuration through a proxy, while behind the scenes Kimi K2.6 is responding. This transparency is the heart of the launcher.

For connection verification, a smoke test is provided. It automatically checks 8 stages from credential resolution through proxy startup to an actual Claude Code response.

[1/8] Resolving credentials...        OK
[2/8] Reaching backend...             OK (GET /models 200, 289ms)
[3/8] Probing /v1/messages support... OK (native passthrough)
[4/8] Starting proxy...               OK (127.0.0.1:51068)
[5/8] Locating claude binary...       OK
[6/8] Round-tripping chat completion... OK (reply='ok')
[7/8] Spawning claude with proxy env...  OK (10463ms)
[8/8] Tearing down proxy...           OK
verify claude: PASS (model=moonshotai/kimi-k2.6, 14322ms)

When you want to switch between multiple routing configurations, you pass a YAML called a route bundle. The registered routes appear in Claude Code's /model picker, allowing you to switch mid-conversation — for example, "classifier-assigned by default, but Sonnet fixed for difficult parts." One thing to note here: Claude Code's picker only displays models whose IDs start with claude or anthropic. Switchyard automatically generates aliases with the claude- prefix, but naming routes like claude-smart from the start will match the picker display and avoid confusion.

When I actually opened the picker, the backend catalog models were lined up with claude- prefixes below the registered routes. Over 340 on my end. This means you can switch directly to models you haven't defined as routes on a whim.

Note that route bundle is treated as a legacy format and displays a deprecated warning at startup. That said, there's no option yet to pass the new profile config format to the launcher, so for now this is the only way to switch between multiple routes in the picker. Bundle-based startup can take close to a minute to listen, and at first I thought it had hung. These are the kinds of gaps you'd expect from something labeled v0.1.0. This old/new relationship, by the way, would be completely reversed three weeks after this article was published. I'll touch on that in the postscript at the end.

Does /effort Finally Work?

Another pitfall I hit during LLM Router verification was actually on the CCR side. Claude Code can adjust thinking depth in 5 levels with the /effort command, but that value goes into output_config.effort in the body, while the thinking field always has {type: "adaptive"} attached regardless of the effort level. Since CCR's think judgment only checks for the presence of thinking, even lightweight /effort low requests would get sent straight to the expensive model. Working around this required writing a custom router to replace the judgment logic.

I reproduced the same situation in Switchyard. I sent Anthropic Messages requests to the llm-routing profile with thinking: {type: "adaptive"} attached while changing output_config.effort through 5 levels.

Since the prompt is "Say OK only." throughout, judging by content means all of them should correctly go to weak. Routing didn't misfire even with the thinking field attached. Switchyard's conversion layer drops requests into an intermediate representation first, and output_config.effort is treated as a first-class field there. It seems structurally impossible to have a place that makes the hasty judgment "there's a thinking field, therefore it's heavy processing."

Building a Fully Local Routing Setup on a Single DGX Spark

Since aarch64 wheels are available, I tried it on DGX Spark as well. Creating a venv and installing nemo-switchyard[server] is all it takes — the import works as-is even in the GB10 aarch64 environment.

Since I was at it, I set up a fully local configuration using no external APIs. I used two models loaded into ollama as stand-ins for the tiers.

targets:
  strong:
    endpoint: ollama # http://localhost:11434/v1
    model: qwen3.6:35b
  weak:
    endpoint: ollama
    model: qwen3:1.7b

When I asked the llm-routing profile "What is 2+2?", the 1.7B responded immediately. When I threw "Prove the halting problem is undecidable using the diagonal argument," it switched to the 35B. Including the routing judgment, everything runs entirely within a single DGX Spark unit. For those running local LLMs who have felt "it's wasteful to wake up the 35B for a simple question," this configuration should be quite appealing.

There was one lesson learned. Initially I assigned the 1.7B to the classifier as well, but the small model ignored the tool calling constraint (tool_choice) and replied in plain text, causing all classifications to fail. Since the design is fail-open, routing itself didn't stop and kept flowing to the default tier, but I was able to catch this because the classifier error count was visible directly in the stats API. Assigning a model of sufficient scale to reliably handle tool calling to the classifier seems to be the practical key point here.

Switching My Daily Hermes Agent Usage to Switchyard

I had been running Hermes Agent through LLM Router. Since I was already at it, I switched this everyday traffic to Switchyard as well. The change was simply swapping out the base_url in the connection settings file.

model:
  default: hermes # Profile name on the Switchyard side
  provider: custom
  base_url: http://localhost:4000/v1
  api_key: dummy
  api_mode: chat_completions

Switchyard's serve doesn't require authentication for clients, so a dummy API key works fine. I assigned the openclaw policy — meant for a resident assistant — to the profile. In the operations-side profiles.yaml, I embedded the model name into the tier name for readability when reviewing later.

targets:
  weak-ds:
    endpoint: openrouter
    model: deepseek/deepseek-v4-flash
    format: openai
  strong-glm:
    endpoint: openrouter
    model: z-ai/glm-5.2
    format: openai

profiles:
  hermes:
    type: llm-routing
    profile_name: openclaw
    strong: strong-glm
    weak: weak-ds
    classifier: weak-ds
    fallback_target_on_evict: strong-glm

I hit one pitfall here. If fallback_target_on_evict in llm-routing is omitted, it tries to find a target named strong. As soon as I changed the tier name to strong-glm, a startup error occurred, so explicit specification is required when using tier names other than the default strong / weak.

I ran everyday traffic through this configuration for one night, about 15 hours. Stats showed 56 routing decisions, 39 dispatches to the weak model itself, and 0 errors. The total DeepSeek V4 Flash for the period — as measured by OpenRouter — was 95 requests, approximately 2.53 million tokens, $0.25, and it was satisfying to see the Switchyard count (56 classification calls + 39 weak model calls = 95) match the billed request count exactly. GLM-5.2 on the strong side also ran with 0 errors including tool calling and streaming. The routing judgment overhead has a median of about 7.2 seconds, but since Hermes traffic is primarily cron jobs and async message responses, there's been no practical impact.

Additionally, I set scheduled delivery jobs where I don't want quality to degrade to bypass routing, directly specifying target IDs like strong-glm for the model name. The usage pattern from the first half of this article — "use the profile ID when you want routing, use the target ID when you want a fixed model" — works directly as an operational tool.

One more thing: there's an issue I found specifically because I put it into production. OpenRouter showed 2.53 million tokens, but the weak token count in Switchyard's stats was only a fraction of that. Tracing the cause, I found the implementation doesn't include streaming response usage in stats (only buffered responses are aggregated), and I was able to confirm that even when the upstream sends usage frames, they get dropped. Since agent traffic is almost entirely streaming, cost aggregation is completely invisible in real-world operations. I reported this as an issue with reproduction steps and the root cause identified.

https://github.com/NVIDIA-NeMo/Switchyard/issues/18

The follow-up on this issue is covered in the postscript at the end.

I Actually Made a Mistake in Choosing the Classifier Once

To be honest, the tier configuration I've been showing up to this point is the second version. Initially, to make it easier to compare with the model pool from the LLM Router article, I set strong to Claude Sonnet 4.6, weak to Nemotron 3 Nano, and set up Gemini 3.5 Flash as the classifier. My reasoning was that a model with "Flash" in the name would be a cheap model suited for the judgment role.

Running Hermes for half a day on this initial configuration showed 125 requests, 0 errors, 98.4% routed to weak, and routing judgment overhead with a median of about 2.1 seconds — the behavior was flawless. However, when I looked at OpenRouter's aggregation, weak itself was about 5.67 million tokens for $0.32, while classifier Gemini 3.5 Flash was about 590,000 tokens for $0.70. The judgment role was consuming more than double the cost of the main model. Since llm-routing sends the last 4 turns of conversation to the classifier every turn, the agent's long context loads directly onto it, averaging over 4,000 tokens per call. I understood in my wallet why sticky — which fixes the tier after the first judgment — and cascade — which avoids calling the classifier — are provided.

However, it wasn't purely a traffic characteristics problem. Laying out the unit price table revealed it was also a mistake in model selection itself.

Looking at OpenRouter's model information, Gemini 3.5 Flash has reasoning listed as mandatory — meaning there's no way to stop the thinking. Checking the stats from that time, 65% of the classifier's 32,360 completion tokens — 21,184 tokens — were reasoning. I was paying $9.00/M thinking costs every single time for a job that just needed to classify into 4 categories and return a tool call. Choosing by name impression without checking the unit price table was my downfall.

What stings is that this information was already in my local knowledge base. Two weeks prior in LLM Router verification, I had recorded "using a reasoning model as a judge makes the thinking unstoppable and judgment heavy," and just days ago in OCR model selection, I had summarized "Gemini 3.5 Flash has mandatory reasoning and is excessive for simple tasks." Recording it but failing to retrieve it in the moment of writing the configuration is meaningless. A painful lesson.

So I consolidated weak and classifier to a single model in 2 roles — DeepSeek V4 Flash — and re-selected GLM-5.2 for strong, giving me the current configuration used throughout this article. GLM-5.2 achieves scores comparable to Sonnet 4.6 on Artificial Analysis metrics, while being priced at about 1/3 for input and 1/5 for output. Here's a comparison of the judgment metrics before and after the switch.

Cost per judgment is approximately 1/12. Since I made weak and classifier the same model, I can no longer separate the judgment portion in OpenRouter billing, so the post-switch figure is an estimate multiplying the token count from stats by the official unit price. The estimate for 56 judgments over one night is $0.02, so the reversal where "the judgment role costs double the main model" has been corrected to "the judgment role costs 10% of the main model."

However, there was a trade-off. The median judgment latency grew from 2.1 seconds to 7.2 seconds — more than tripling. It was surprising to see it slow down after eliminating reasoning, but the underlying work of processing an average 4,500-token prompt every time hasn't changed, so the raw response speed of the model and provider shows through directly. If placed in front of an interactive agent, judgment cost and judgment latency need to be weighed on separate axes.

The tier swap itself required only a few lines of YAML change and a serve restart. Get a working configuration into production first, then swap models while watching the stats and billing. Being able to iterate this way naturally is a benefit of the routing proxy becoming a pip library.

Things I'm Concerned About

I've written a lot of positives, so let me honestly summarize the current caveats as well.

First, the development status is Alpha, and known issues are publicly listed. As of v0.1.0, there are 2 cases: token aggregation returning 0 for Codex integration, and requests with tools failing when routed to an upstream with a fixed tool schema. The latter can be encountered in agent operations that heavily use tools, so the safe approach is to ensure all tier models you route to support tool calling.

Missing format: specification also requires attention. If omitted, it defaults to OpenAI format, and the documentation explicitly states that cache_control for prompt caching gets stripped when sending to Claude-family models. Make sure to explicitly specify format: anthropic for targets using Claude as upstream.

My custom version of LLM Router was a multi-choice setup selecting one from a pool of 9 models using a trained classifier. All built-in routing in Switchyard is designed as a binary strong/weak choice with a classifier added, so if you want more options, you'd define multiple routes and have the caller explicitly select via model name or the /model picker. The role division is: automatic judgment handles binary choices, and multi-choice selection is left to explicit caller specification. This is similar to how Sakana Fugu, which I introduced previously, makes it the client's responsibility to call either fugu or fugu-ultra, with orchestration running internally within the called endpoint.

https://dev.classmethod.jp/articles/sakana-fugu-ga-first-touch/

So will multi-choice automatic judgment never come back? While reading the code, I made an interesting discovery. While the documentation covers 4 routing methods, the source already has a type implemented for integrating LMSYS's RouteLLM (a learning-based router using matrix factorization) as a profile. Even though the trained router appears to have disappeared, a receptacle for learning-based approaches is properly prepared. This aligns with NVIDIA's response that "LLM Router algorithms are being ported to Switchyard." For those of us with LLM Router training assets, this is a point I'd want to dig into in a follow-up... or so I wrote, but that expectation turned out to be wrong. As covered in the postscript at the end.

Post-Publication Updates (Added 2026-08-05 · Supplemented 2026-08-08 / 2026-08-12)

What Changed During This Month

The issue mentioned in the latter half of the article — streaming response tokens not appearing in stats — was fixed on July 14. Since agent traffic is almost entirely streaming, this makes cost aggregation usable in real-world operations.

There's also a follow-up on the documentation amendment PR I submitted about the random-routing vs random_routing naming difference. A maintainer indicated they'd prefer to unify it in the code rather than explain it in documentation, so I resubmitted it as a code fix (PR #22) that accepts both hyphens and underscores, and it was merged on July 7. The stumbling block described in the main article directly led to an upstream fix.

Configuration saw significant changes. The route bundle I described as "deprecated in old format" is the one that remained, while the profile config I had been using as the new format was removed. From late July onward, it was rewritten as an independent Rust server with TOML configuration. The RouteLLM integration I wrote about wanting to dig into in a follow-up was also deleted on July 16.

However, what installs via pip install nemo-switchyard is still v0.1.0, and the procedures in this article still work as written. If you look at the main branch on GitHub, it's a completely different thing, so if you're starting fresh, check first whether you're dealing with the PyPI version or main.

This Rust version was then officially released as v0.2.0 on August 10, 2026, and became installable from crates.io and PyPI. Since everything from the configuration syntax to the classifier algorithm is entirely different from v0.1.0 in this article, if you're doing a new installation please refer to the new article I wrote as a first-touch for v0.2.0 (article published 2026-08-12).

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-v020-rust-first-touch/

Trying opencode and Fireworks on an Internal Workload

At the time of writing the article, the only production use case I had on hand was the Hermes Agent, but since late July I've been running another workload internally. It's a configuration that places Switchyard behind the OSS coding agent opencode and routes to Fireworks AI models. It's a two-role setup with DeepSeek V4 Pro assigned to strong, and DeepSeek V4 Flash assigned to weak and classifier, with session affinity enabled.

I ran an A/B test on coding tasks. The same task set was run across 3 arms — auto (with routing), strong fixed, and weak fixed — for a total of 39 runs. All three arms achieved perfect scores with zero failures, and the cost for auto was approximately 27% cheaper than strong fixed, including the cost of the classifier. Looking at just the main model cost without the classifier, it's 40%.

However, I'll be transparent about the conditions. Since the task set was one where even weak fixed could achieve a perfect score, this is not proof that "routing protected quality," but rather proof that it "reduced cost without breaking anything." Measuring quality differences would require harder tasks involving design decisions spanning multiple files. Latency was also worse for auto, with a difference of 30 seconds vs. 21 seconds in real-time median.

Session affinity worked as expected. Classifier calls were reduced by 56%, and tier switching within sessions was zero. On the other hand, even for the same task, the pinned tier could split between weak and strong across different sessions, so there's some variance in cost estimates.

When using DeepSeek models with Fireworks, you need to write extra_body: {} empty in the configuration. This is because the vLLM parameters that Switchyard automatically adds are rejected by Fireworks, resulting in an HTTP 400 error. This has been reported upstream, but for now, working around it in the configuration is the safest approach.

The full configuration has been published as a Docker bundle. Those who want to try the same combination can run it from here.

https://github.com/himorishige/switchyard-opencode-bundle

Configuration updates have continued since then, with weak switched to the official DeepSeek V4 Flash-0731 and strong switched to Kimi K3. The current default for the bundle also uses this configuration. The overall picture as a team AI environment — from the thinking behind narrowing down to two models, to settings that prevent data from leaking externally, to observation through routing logs — is summarized in the next article (published 2026-08-08).

https://dev.classmethod.jp/articles/open-weight-team-ai-environment/

Summary

I ran NeMo Switchyard on Mac and DGX Spark and re-examined the two pitfalls encountered during LLM Router validation. The issue where model names were being ignored has been officially resolved as a distinction between profile/target ID usage, and the /effort misfiring no longer occurs at the design level of the conversion layer. Both the patch to the fork and the custom router built as a CCR are no longer needed.

It installs via pip, requires no GPU, connects to Claude Code with a single command, and can even be set up for fully local routing on a DGX Spark. While there are some rough edges befitting an Alpha version, the barrier to entry feels dramatically lower compared to the weight of "forking and nurturing a Blueprint" for an LLM Router.

The classifier cost dropped to approximately one-twelfth by reconsidering the model selection. Next time, I'd like to look at a comparison of session affinity and stage_router — which reduce the number of judgments themselves — using actual data from the internal workload mentioned in the addendum.