Airbnb Uses Cost per Task to Tune Coding Agents on Amazon Bedrock

Airbnb is using cost per task, alongside model evaluations and production monitoring, to select models and manage coding agents running through Amazon Bedrock. The approach is designed to track the full cost of completing an engineering task rather than pricing individual model requests in isolation.
- Airbnb runs AirChat coding agents through an LLM gateway on Amazon Bedrock.
- The company evaluates models using its own engineering tasks before rollout.
- Cost per task includes all token costs across every request needed to finish a task.
- Airbnb uses CloudWatch metrics to monitor latency, throttling, quota use and spending.
- Production data can lead Airbnb to change model defaults and effort ceilings.
Airbnb runs coding agents through Bedrock
Thousands of engineers and engineering-adjacent people at Airbnb use AirChat, the company’s internal tool for running coding agents, every week. The agents send model requests through Airbnb’s large language model gateway, which runs on Amazon Bedrock.
Engineers can run AirChat on AirDev workspaces, which are personal development machines in the cloud. They can also hand a task to AirChat Remote, which runs cloud agents in separate cloud workspaces while the engineer works on something else. Both routes pass through the gateway to Amazon Bedrock.
AWS says Bedrock provides access to foundation models from multiple providers through a common API, while AWS manages the model infrastructure. It also integrates with AWS access controls and monitoring, and processes Airbnb’s code, prompts and model responses within AWS.
The pace of model releases makes choosing defaults more difficult. According to Airbnb, eight new models came out in about five weeks. By mid-October, about two thirds of requests sent by its engineers through coding agents went to those new models. Each model release can affect output tokens, cost, latency and capacity at the same time.
Cost per task includes every request
Airbnb evaluates models before rollout using its own engineering tasks. The company measures both a score and the cost per task for every model and every permitted reasoning effort. Reasoning effort controls how much a model reasons before responding.
Airbnb defines cost per task as the list-price cost of all tokens an agent uses to finish one task. Because a coding agent can make many requests and reread the context on each request, Airbnb adds together the costs of the complete task rather than looking at a single request.
The calculation includes uncached input tokens, cache reads, cache writes and output tokens for every request. Each type is priced at its own rate. Airbnb records the average cost beside the score for each model, reasoning effort and agent client, including low-scoring attempts.
The company uses these results to choose a default model and effort for its coding tools, as well as an effort ceiling, which is the highest effort allowed for a model. In one model test described by Airbnb, moving from low to high reasoning effort raised the score by 0.8 points, while cost per task was 2.5 times as high.
Production metrics show when choices need to change
Airbnb’s evaluation system is paired with production monitoring. The gateway records the model, client, team and user for every request, and Airbnb uses those records in aggregate for cost and capacity planning. CloudWatch supplies token counts, latency, errors and throttles.
AWS identifies several key signals in the AWS/Bedrock namespace. InvocationLatency, TimeToFirstToken, OutputTokenCount and derived output tokens per second help investigate latency. InvocationThrottles and request token reservations help track throttling. EstimatedTPMQuotaUsage measures estimated tokens per minute quota consumption across Bedrock inference APIs and can support proactive alarms before usage reaches a quota limit.
For spending, Airbnb monitors InputTokenCount, CacheReadInputTokenCount, CacheWriteInputTokenCount and OutputTokenCount. AWS says cache reads cost less than uncached input, while cache writes can cost more depending on the model, so a lower cache hit rate can increase cost per request.
Airbnb compares these production signals with evaluation scores and cost per task. When real usage does not match evaluation results, the company investigates the cause and can revisit model defaults or effort ceilings. The setup gives engineering teams a way to connect model quality decisions with the operational effects of latency, capacity, throttling and token consumption.


