Artificial intelligence news.
Straight from the original announcements.

AWS Adds Cross-Region Inference Routing to SageMaker AI

November 19, 2026, 19:20By the AI News Bites Editorial TeamSource: AWS
AWS Adds Cross-Region Inference Routing to SageMaker AI
Image: AWS

AWS has introduced cross-Region inference routing in Amazon SageMaker AI, giving applications one invocation path across model endpoints and inference components in multiple AWS Regions and accounts. The new inference router keeps requests local when possible and shifts new traffic when a local destination cannot serve reliably.

Key points
  • AWS introduced cross-Region inference routing in Amazon SageMaker AI.
  • The inference router prefers local destinations and shifts traffic when they cannot serve reliably.
  • Routing targets can contain endpoints and inference components across Regions and accounts.
  • The router retries eligible real-time failures once on another destination.
  • Health states and routing decisions are available through Amazon CloudWatch and response data.

A managed path across Regions

Before this launch, customers running models on SageMaker AI managed endpoints had to build their own cross-Region routing logic. That meant detecting a struggling endpoint, deciding which failures were safe to retry, moving traffic back after recovery and managing access to endpoints in other accounts.

The inference router is a Regional SageMaker AI resource that applications invoke instead of calling a specific endpoint. Customers register the destinations that serve a model, including real-time endpoints, inference components or asynchronous endpoints. A routing target identifies one logical model, and a router can contain multiple routing targets that fail over independently.

Endpoints remain deployed as they are today, with their existing models, configurations, instance pools and autoscaling. The router does not create capacity, scale endpoints, deploy models or copy model artifacts and container images. Each destination must already be running in its Region.

How SageMaker AI chooses a destination

The router prefers destinations in its own Region. If multiple local destinations are healthy, it spreads requests across them evenly. Requests move to other Regions only when local destinations cannot serve reliably, and traffic returns gradually after recovery.

SageMaker AI calculates a health score from recent requests in each destination's own Region. Server errors, throttles and timeouts reduce the score, while platform errors such as a failed health check have the greatest effect. Errors caused by the request itself, such as invalid input, do not reduce the score. Model latency is not used as an input.

The service publishes one of three health states to Amazon CloudWatch: healthy, overloaded or degraded. An overloaded destination continues serving while some traffic moves away. A degraded destination receives most of its traffic elsewhere, while retaining a small share to detect recovery. AWS says traffic starts moving within about a minute of sustained errors, or longer for destinations with little traffic. Customers cannot set routing weights or a fallback order.

Retries, asynchronous requests and supported setups

For real-time requests, the router retries once on another destination when the first destination fails without running the model. AWS lists throttles, a 503 error, a model that is not ready, rejection by a cross-Region traffic cap and access denied by the destination as examples. It does not retry timeouts, 500 errors or model errors. Requests already in progress are not interrupted.

For asynchronous requests, the router tries each eligible destination once, starting with local destinations, until one accepts the request. Applications use InvokeInferenceRouterAsync with a routing target name and an Amazon Simple Storage Service input location.

AWS describes use cases including handling GPU capacity shortages, routing between endpoints owned by separate AWS accounts, serving several models from one router and placing a router behind an existing API or model gateway. Direct InvokeEndpoint calls continue to work, and real-time responses include the destination that served the request and the reason for the routing decision.

Related news

AWS SageMaker AI Adds Monitoring for Inference Endpoint Drift
Products and toolsNovember 19, 2026, 22:28

AWS SageMaker AI Adds Monitoring for Inference Endpoint Drift

Amazon SageMaker AI can now identify latency drift, underused GPUs and missed serving optimizations on eligible generative AI inference endpoints. The monitoring is…

AWS adds GPU diagnostics and observability to SageMaker HyperPod
Products and toolsNovember 19, 2026, 17:41

AWS adds GPU diagnostics and observability to SageMaker HyperPod

AWS has introduced enhanced infrastructure observability in Amazon SageMaker HyperPod, bringing GPU diagnostics and profiling together for distributed training workloads…

Salesforce uses SageMaker AI routing to manage global model capacity
Products and toolsNovember 18, 2026, 21:13

Salesforce uses SageMaker AI routing to manage global model capacity

Salesforce has integrated Amazon SageMaker AI’s cross-Region inference routing into its Internal Gateway to handle fragmented model capacity across more than 30 global…