NVIDIA Dynamo is the first inference stack built for MoE from day one

vLLM wasn't built for MoE. Dynamo is. That changes who wins the inference layer.

vLLM wasn't built for MoE — and you can feel it

Every inference framework you reach for today started life serving dense transformers. When mixture-of-experts showed up, we papered over it. vLLM bolted on expert parallelism. SGLang hacked radix attention around the problem. TensorRT-LLM grudgingly grew MoE kernels. They all work. They all leak abstraction.

In June, NVIDIA shipped Dynamo: an open-source distributed inference runtime that puts MoE at the center of the design instead of treating it as an edge case. That sounds like marketing. It isn't.

What Dynamo actually does

Dynamo is a disaggregated serving platform with three pieces that matter:

1. A planner. It watches live traffic and decides which experts need to be hot on which GPUs. For DeepSeek-style models with hundreds of experts, this isn't optional. Without smart placement, you're paying H100 rent to keep cold experts resident.

2. A smart router. Sends tokens to the right expert replica, balancing locality with load. Dense-model runtimes don't have to think about this. MoE models can't afford not to.

3. NIXL (NVIDIA Inference Xfer Library). A high-bandwidth KV cache transfer layer that lets prefill and decode live on different machines and still hand off context without melting your interconnect. Disaggregated prefill/decode isn't new. Doing it without losing 30% throughput to serialization is.

The planner runs as its own process and you can swap the policy, but the shipped default uses real traffic patterns to make decisions. That's the part that made me pay attention — it's not a static config, it's a control loop.

The catch

It's a real-time system. The flexibility buys you operational baggage.

Hardware lock-in. This is NVIDIA-flavored. H100, H200, Blackwell only. If you're an AMD shop or running Trainium, look away. NIXL is the giveaway — it's not a generic RDMA library.
Young. First real release. The model zoo covers the usual suspects (Llama 4, DeepSeek V3, Mixtral variants). Bring your own tuning for anything else.
You need a real cluster. Dynamo earns its keep at 8+ GPUs. Below that you're paying orchestration cost to coordinate two boxes.
TensorRT-LLM is the favored child. Other backends work, but if you want the benchmark numbers in the README, you're using TRT-LLM. That constrains your model definition to what their compiler accepts.

Who actually needs this

If you're serving a dense Llama in one region for 200 users, you don't need Dynamo. You need vLLM and a Sunday afternoon.

If you're running inference for a 200B+ expert MoE model across multiple replicas and paying real money to NVIDIA Cloud or your own DGX stack, Dynamo is worth a serious look. The planner alone shifts expert-to-GPU placement continuously rather than relying on the operator's static guess.

The other audience: agent platforms serving big MoE models where time-to-first-token pays rent. Disaggregated prefill matters there. Every millisecond of TTFT on a 200B MoE model is real margin.

What this means for builders

Dynamo is the first credible signal that NVIDIA sees inference as its own market, not just a route to GPU sales. Open-sourcing it is deliberate: if Dynamo becomes the default MoE runtime, every hyperscaler building on NVIDIA hardware reasons about inference the NVIDIA way. That is not free for AMD. It is not free for in-house stacks either.

Practical takeaways:

Try it on DeepSeek V3 first. That's where the MoE placement story actually pays off. Dense models won't show you the wins.
Watch the planner logs. They will tell you whether your routing assumption matches reality. Mine never does.
Don't switch off vLLM yet. Dynamo is a year behind on tooling and benchmarks. Run both. The interesting future is hybrid — vLLM for the cheap dense stuff, Dynamo for the MoE-heavy paths.
Track NIXL upstream. That's the real prize. A fast open transfer library with NVIDIA's backing tends to become a default whether the committee likes it or not.

Dynamo won't eat the inference world this year. But the assumptions it makes about the shape of the workload are the right ones, and the rest of the stack will quietly bend toward them.