AWS shows how to route coding agent requests with NeMo Switchyard

AWS has published a sample deployment that routes coding agent requests between models on Amazon Bedrock using NVIDIA NeMo Switchyard and LiteLLM. The approach measures performance by cost per solved task, rather than by token price alone.
- AWS uses cost per solved task to evaluate coding agent routing.
- NVIDIA NeMo Switchyard supports Stage and Random routing in the LiteLLM integration.
- The sample deploys LiteLLM and Switchyard on Amazon ECS with AWS Fargate.
- LiteLLM routes requests to models through Amazon Bedrock.
- AWS labels the integration experimental and intended for non-production use.
Why task-level costs matter
Coding agents such as Claude Code and OpenAI Codex CLI may make a few model calls or dozens while completing one developer request. They can read files, edit code, run tests and retry work, meaning the total cost depends on the number of turns, the history replayed in each turn and whether the task finishes successfully.
AWS says developers pay for every attempt, including work on tasks that never finish. It therefore recommends evaluating routing by cost per solved task: the total cost of runs divided by the number of runs whose tests pass. This gives teams a way to compare a routed model group with individual model baselines on their own workloads.
How Switchyard selects a model
NVIDIA NeMo Switchyard is an open source library that lets an AI agent choose which model handles each request. In the sample's LiteLLM integration, Stage routing examines recent tool activity and results in the request history before choosing between two candidate models. The agent must send the full conversation history for this strategy to work.
Random routing instead selects from configured models, with optional weights. AWS describes it as useful for traffic splits and A/B comparisons. Stage requires exactly two candidates, while Random requires at least two. Neither strategy uses a judge model. Classifier-backed strategies such as Task and Composite routing require intermediate model calls that this integration does not support.
The AWS deployment architecture
The walkthrough uses the AWS Cloud Development Kit to deploy LiteLLM, an open source gateway for large language models, with the Switchyard routing plugin in an AWS account. LiteLLM calls models through Amazon Bedrock. LiteLLM and the plugin run together in one container on Amazon Elastic Container Service with AWS Fargate, using private subnets.
Requests use an OpenAI-compatible interface with the model field set to switchyard. AWS Certificate Manager, Amazon Route 53, AWS WAF and an Application Load Balancer provide the endpoint and network controls. The load balancer forwards only /v1/chat/completions, /v1/models and /health/liveliness to the ECS task. LiteLLM then calls the selected model through the Amazon Bedrock Converse or ConverseStream API.
The response includes an x-litellm-model-name header identifying the model that answered, while the response body's model field remains switchyard. Inference traffic uses an Amazon Bedrock Runtime interface endpoint in the Amazon VPC.
A sample for evaluation, not production
The sample requires access to Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B in Amazon Bedrock. AWS says the pair has been tested in US East (N. Virginia), and asks users to check regional availability and tool-calling support elsewhere.
AWS also cautions that the sample installs switchyard-server 0.3.0 and uses the published LiteLLM v1.102.0 image. It describes switchyard-server 0.3.0 as a demo and evaluation component, and says the experimental integration's configuration and APIs may change. The company recommends deploying in a non-production account, testing before reliance and working with security and legal teams on organizational requirements.
For users building coding agents, the example provides a concrete way to test whether routing reduces the cost of completed work. Its central measure, however, is tied to each team's workloads, model configuration and test outcomes, rather than to a token price considered in isolation.


