A CatBoost model predicts fine-tuning time far better than FLOPs

A CatBoost model predicted the end-to-end duration of fine-tuning jobs far more accurately than conventional FLOP calculations in a Hugging Face study of 272 production runs. The model reached an R² of 0.98 and a mean absolute error of 1.71 hours in 5-fold cross-validation.
- The study analyzed 272 production SFT runs on NVIDIA H200 GPUs.
- The best FLOP baseline had a mean absolute error of 12.41 hours.
- CatBoost reached an R² of 0.98 and a mean absolute error of 1.71 hours.
- Evaluation and checkpoint activity were among the model's most important features.
- The model uses information available before a job is dispatched.
Why FLOP estimates miss the full job
FLOP-based estimates focus on the compute performed during the optimization loop. For a dense Transformer, the source describes training costs as roughly 6·P·D floating-point operations, where P is the parameter count and D is the number of training tokens. That figure is then divided by the number of GPUs, their peak throughput and an assumed Model FLOPs Utilization, or MFU.
The approach does not capture the full wall-clock process. Before training starts, jobs may need environment setup, model downloads, loading and tokenization. During training, they can perform evaluations and upload checkpoints. At the end, the final model is sent to object storage. On a shared cluster, these activities compete with other jobs for network and storage I/O.
The study tested FLOP baselines with MFU values of 30%, 45% and 60%. The 30% setting performed best, but still had an R² of 0.31 and a mean absolute error of 12.41 hours. The 45% and 60% settings produced mean absolute errors of 13.01 hours and 14.62 hours, respectively.
A model built from information available before dispatch
The researchers treated end-to-end duration as a supervised regression problem. The target was the wall-clock time from the moment a job received its GPUs until it finished, excluding queue time. Every input had to be available before dispatch, including hyperparameters, dataset sizes, model identifiers and GPU allocation.
The feature set included configuration and model metadata, along with derived measures intended to represent operational overhead. These included num_eval_data, which counts samples processed across evaluation cycles, num_saves, num_params_saved and per-GPU token volumes. The FLOP estimate with MFU fixed at 0.45 was also included as a feature, allowing the model to learn adjustments to the compute-only estimate.
The team compared Linear Regression, Ridge, XGBoost and CatBoost. After feature selection and grid search using mean absolute error, the final CatBoost model used 24 features.
Results from 272 production runs
The data came from 272 supervised fine-tuning runs in a corporate production environment, representing about 7,382 GPU-hours. The runs covered several model families up to 15 billion parameters, LoRA and full fine-tuning, and a broad range of dataset sizes and hyperparameters.
Jobs used 1, 2 or 4 GPUs from pools of 8 to 32 NVIDIA H200 GPUs across 1 to 4 nodes, alongside other training and inference workloads. Their measured durations ranged from 1.6 to 172.4 hours. The mean duration was 23.43 hours and the median was 15.29 hours. The software stack included Hugging Face Transformers and Unsloth.
CatBoost achieved an R² of 0.98, an RMSE of 3.52 hours and a mean absolute error of 1.71 hours. XGBoost followed with an R² of 0.97 and a mean absolute error of 1.95 hours. Ridge reached a mean absolute error of 7.23 hours, while Linear Regression reached 7.52 hours. Calculon, a more detailed analytical performance model, had an R² of -0.51 and a mean absolute error of 20.36 hours in this comparison.
Why the findings matter for shared clusters
CatBoost's feature importance placed evaluation and checkpoint settings alongside model and workload variables. num_eval_data ranked first, followed by num_trainable_params, batch_eval_size_percent, num_all_params and num_gpus*batch_train_percent, with num_params_saved also ranking highly.
Across all out-of-fold predictions, the median absolute error was 0.51 hours. The source says 80% of predictions were within 2.20 hours and 90% were within 4.31 hours. These estimates could help shared-cluster operators plan job placement and ordering using details known before a run begins, rather than relying only on nominal compute.
The researchers describe the work as a preview of an upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models. The full version will be published on arXiv soon, according to the article.
Hugging Face: Why your fine-tuning job takes longer than the FLOPs say


