TurboPairFormer speeds OpenFold3 training by up to 1.73×

Lambda, Stevens Institute of Technology and the OpenFold team have built TurboPairFormer, an open source GPU implementation that trains OpenFold3 up to 1.73× faster than its original Triton backend. The system also targets gradient accuracy and reproducibility on NVIDIA Hopper GPUs.
- TurboPairFormer trains OpenFold3 up to 1.73× faster than its original Triton backend.
- It had the lowest GPU computation time per optimizer step across tested crop sizes.
- The implementation reduced several gradient errors through output residual compensation.
- Gradients were bitwise identical across all 600 fixed input test cases.
- The open source package supports NVIDIA H100 and H200 GPUs.
A faster implementation of triangle attention
TurboPairFormer is designed for triangle attention and triangle multiplication, operations used by biomolecular structure prediction models such as OpenFold3. These operations help models reason about pairs of molecular components through a third component, capturing geometric information needed to predict three dimensional structures.
Triangle attention is a major computational bottleneck because its cost scales cubically with training crop size. Doubling the crop size makes the operation about 8× more expensive. The researchers say TurboPairFormer combines hardware aware GPU optimizations with a redesigned attention calculation to address that cost while retaining numerical precision and reproducibility.
The package is open source, available on PyPI, and currently supports NVIDIA H100 and H200 GPUs.
Benchmarks show the largest gains at bigger crop sizes
The team integrated TurboPairFormer into OpenFold3 and compared it with seven other backend configurations: ByteDance Protenix, NVIDIA cuEquivariance, TriFast, MegaFold, DeepSpeed-Evo, ScaleFold and the original OpenFold3 Triton backend. Tests used 16 H100 GPUs across two nodes and crop sizes of 384, 640 and 768 tokens.
TurboPairFormer recorded the lowest GPU computation time per optimizer step at every tested crop size. At crop 768, it took 12.58 seconds per step, compared with 21.72 seconds for OpenFold3 Triton, making it 1.73× faster according to Lambda. On H200 GPUs, the system was 1.54× faster than OpenFold3 Triton at crop 768, reducing time per step from 22.53 to 14.67 seconds.
The measurements covered steps 21 to 200 and excluded data loading, host to device transfers and EMA updates. The performance gap grew with crop size, where triangle attention becomes more expensive.
Accuracy and reproducibility are part of the design
TurboPairFormer computes each attention probability tile once in the backward pass and uses the result for gradients for queries, keys, values and pair biases. The implementation also records the rounding information lost when FP32 outputs are converted to BF16, then uses that residual to improve backward pass gradients.
In controlled experiments, Lambda reports 28% to 47% less mean gradient error from this output residual compensation. Against NVIDIA cuEquivariance, TurboPairFormer reduced mean error by 31.4% to 44.4% for query gradients, 28.7% to 42.4% for key gradients and 16.9% to 37.9% for pair bias gradients. Value gradient accuracy was essentially tied.
For reproducibility, the researchers ran 600 fixed input cases five times each on an H100 GPU. TurboPairFormer produced bitwise identical gradients in all 600 cases. In a separate training test using the same checkpoint, random seeds and training examples, the team reports steady loss without significant fluctuations or NaN values.
These results could let researchers run more biomolecular modeling experiments on the same GPUs. Lambda says the kernels may support workloads including protein ligand binding prediction, structure guided drug design and research on disease related protein families. The package can be installed with pip install turbopairformer.


