Artificial intelligence news.
Straight from the original announcements.

Rafay adds budget-based routing for frontier and open-source AI models

November 5, 2026, 12:26By the AI News Bites Editorial TeamSource: PR Newswire
Rafay adds budget-based routing for frontier and open-source AI models
Image: PR Newswire

Rafay Systems has added Token Router and Token Budgets to Rafay Token Factory, allowing operators to direct inference requests across in-cluster and external models while enforcing spending limits. The capabilities are available today to Rafay Token Factory customers.

Key points
  • Token Router directs requests across in-cluster and external model providers.
  • Token Budgets support daily and monthly limits for organizations, projects and users.
  • Routing can fall back to lower-cost or self-hosted models when budgets are reached.
  • The capabilities are available today to Rafay Token Factory customers.

One endpoint for multiple model providers

Token Router sits between applications and the locations where inference runs. It can send requests to in-cluster deployments or external providers such as OpenAI, Anthropic and any OpenAI-compatible endpoint, using priority, provider health and cost as routing factors.

Applications use one API and one set of credentials, while operators can change the model source without requiring code changes. Requests can be sent to a highest-priority provider group first and then moved to another group if the primary is unavailable. Rafay cites configurations such as self-hosted models with cloud bursting, frontier models with in-cluster fallback, and multi-vendor routing.

Traffic is balanced across healthy providers in a group. Providers that fail health checks are removed from rotation automatically and return when they recover. Rafay also says operators can set separate rates for input, output, cached input and cache write tokens, with each request metered according to the group that served it.

Budgets can shift traffic instead of simply blocking it

Token Budgets let administrators set daily or monthly spending limits for an organization, project or individual user, using the currency supported by the service provider. A user's budget follows them across projects, and every request is checked against applicable budgets before admission.

When a limit is reached, an operator's policy can deny requests, continue serving them or fall back to a lower-cost model. Token Router can also move traffic to a lower-cost or self-hosted model when a tenant, team or API key approaches or reaches its budget. This lets users continue receiving answers while spending remains within the administrator's limit.

New projects and users inherit default budgets, while administrators can apply overrides for heavier workloads. Threshold alerts notify administrators before limits are reached. Developers can view their spending and remaining budget in Rafay Developer Hub, and budgets reset automatically each day or month.

Targeting neoclouds and enterprise AI teams

Rafay says the capabilities are designed for neoclouds offering open-source models as a service and enterprises operating internal token factories. A neocloud can add frontier models to its catalog through the same API and credentials, offer a daily allowance for frontier usage, then route additional traffic to open-source models running on its own GPUs.

For enterprises, a platform team can set a daily frontier quota for a team or application. Once that quota is used, requests can move to an open-source model on company GPUs, while applications continue calling the same endpoint. Rafay says a token usage dashboard with per-user filtering and CSV export supports internal showback and chargeback.

“Token Router and Token Budgets turn that tradeoff into policy,” Mohan Atreya, chief product officer at Rafay Systems, said in the announcement. Rafay also added support for LiteLLM Generic Guardrail API providers, separate cached input token pricing, and API key rotation policies with expiration, overlap windows and alerts.

Related news

JetBrains asks 12 AI models for 1,200 programming hot takes
ResearchNovember 15, 2026, 18:02

JetBrains asks 12 AI models for 1,200 programming hot takes

Twelve AI models overwhelmingly attacked Python, Rust and TypeScript in a JetBrains experiment that collected 1,200 opinions about programming languages. The models also…

OpenAI and Databricks bring governed enterprise data to AI tools
BusinessNovember 13, 2026, 16:30

OpenAI and Databricks bring governed enterprise data to AI tools

OpenAI and Databricks are expanding their partnership to help enterprises use ChatGPT, Codex and GPT models with governed business data. The companies describe three…

Rafay and Fortanix bring confidential AI to neoclouds and enterprises
Safety and ethicsNovember 12, 2026, 12:26

Rafay and Fortanix bring confidential AI to neoclouds and enterprises

Rafay Systems and Fortanix have announced a joint solution for delivering confidential AI as a service on neocloud, sovereign AI cloud and enterprise infrastructure. The…