AWS Launches SageMaker JumpStart Leaderboard for Model Comparisons

AWS has launched the Amazon SageMaker JumpStart Model Leaderboard, a SageMaker Studio view that compares open-weight models on quality, performance and estimated cost. The leaderboard covers more than 70 models at launch and uses AWS measurements to make comparisons more consistent.
- AWS launched the SageMaker JumpStart Model Leaderboard in SageMaker Studio.
- It covers 70+ models from providers including Meta, Google, Qwen, IBM and Arcee.
- AWS measures models on BIG-Bench Hard, GPQA Diamond and MMLU-Pro.
- Users can compare time to first token, throughput and estimated instance cost.
- Filters and deployment options connect model selection with SageMaker deployment.
A side-by-side view of JumpStart models
Amazon SageMaker JumpStart provides hundreds of open-weight foundation models that users can deploy quickly. AWS says the number of choices can make it difficult to decide where to start, particularly when model cards and benchmark results are published under different conditions.
The new leaderboard appears on the JumpStart base models page in SageMaker Studio, alongside the existing Categories and All models views. It opens with rankings for large language models and uses the same search bar and filters available elsewhere in JumpStart.
At launch, it covers 70+ models from providers including Meta, Google, Qwen, IBM and Arcee. AWS says more models will be added over time.
Quality, speed and cost in one place
The leaderboard reports quality scores for three public reasoning benchmarks: BIG-Bench Hard, GPQA Diamond and MMLU-Pro. AWS runs the tests itself with the same evaluation setup for every model, so the results can be compared using the same method. Higher scores are better, and the benchmarks remain separate rather than being combined into a single rating.
BIG-Bench Hard covers challenging tasks such as logical deduction, object tracking and multi-step arithmetic. GPQA Diamond consists of graduate-level science questions, while MMLU-Pro measures knowledge and reasoning across subjects including science, law and engineering.
AWS also measures performance and cost by sending a standardized workload to an endpoint using each JumpStart deployment configuration. The displayed metrics include time to first token, throughput per request and estimated instance cost per hour. Time to first token measures how long a user waits before a response begins, throughput measures generated tokens per second, and instance cost is based on the default instance type for the configuration.
The estimated cost covers the SageMaker instance cost at the on-demand hourly rate. AWS says it excludes data transfer, storage, inference component costs and discounts such as Reserved Instances or AWS Savings Plans. Actual billing can vary, and AWS directs users to its Pricing Calculator for billing estimates.
Profiles help users compare deployments
Users can select a use case profile from the View benchmarks for your use case menu. The default profile is Max Context, and changing it affects performance and cost metrics, but not the quality scores.
The Optimized for menu selects the deployment configuration used for the displayed metrics. The default Balanced option uses the configuration with the lowest estimated cost per token. Cost optimized selects the lowest hourly instance price, Latency optimized selects the lowest time to first token, and Throughput optimized selects the highest throughput. Some configurations can use speculative decoding.
AWS says users can start by applying requirements such as license, provider, and input or output modality filters. They can then sort benchmark columns, use a blue heatmap to identify stronger results, hide unnecessary columns and open a model to deploy it. For models that support serverless customization, the model page also provides a customization option.
The leaderboard is designed to connect model selection with deployment in SageMaker Studio. AWS says users should select the same use case and optimization goal when deploying that they used for comparison, because a different configuration can change quality, latency, throughput and cost.


