PyTorch team integrates SGLang with IBM Spyre in less than a week

A two-person team integrated SGLang with IBM Spyre in less than a week, using AI coding agents to build an open source plugin of roughly 1,500 lines. PyTorch says SGLang delivered about 3.6× the throughput of a reference PyTorch inference implementation using torch-spyre.
- A two-person team integrated SGLang with IBM Spyre in less than a week.
- The open source plugin contains roughly 1,500 lines of code.
- SGLang delivered approximately 3.6× the throughput of a reference PyTorch implementation using torch-spyre.
- SGLang was on par with vLLM on measured Granite 3.3 8B and Gemma 4 26B A4B workloads.
- RadixCache prefix sharing worked without Spyre-specific changes.
A short integration built on existing software
The team wanted to make SGLang’s serving stack, including its built-in prefix sharing, available to users of IBM’s Spyre accelerator for enterprise AI inference. Spyre integrates with PyTorch through torch-spyre, which registers it as a PyTorch device and connects it to torch.compile.
Existing PyTorch hardware support and SGLang’s extension points limited the amount of new code required. AI coding agents used existing PyTorch implementations of Spyre operations as reference material, then adapted the integration code to SGLang’s backend interfaces. Their role focused on connecting established hardware support to the serving framework.
The result was a plugin of roughly 1,500 lines. Four small patches extended SGLang’s backend support. Two exposed existing execution paths without CUDA-specific dependencies, while two registered Spyre as a supported device and attention backend that does not use Triton. The compatibility changes could also support other accelerators, while the registration changes are specific to Spyre.
Performance compared with PyTorch and vLLM
According to PyTorch, SGLang on Spyre achieved approximately 3.6× the throughput of the reference PyTorch inference implementation using torch-spyre. The team also compared SGLang with vLLM on Spyre using Granite 3.3 8B and Gemma 4 26B A4B.
Across the workloads measured for those two models, SGLang’s performance was on par with vLLM. The post says the comparison used relative performance normalized to vLLM = 1.0, with lower values representing better performance.
The integration reused attention code from the existing vLLM Spyre backend, including its handling of input buckets and tensor shapes. It also reused the backend’s PyTorch-based approach to allocating the KV cache. Existing Spyre implementations for RMSNorm, SiLU, GeLU, rotary embeddings and linear layers were connected to SGLang’s operator dispatch.
What the approach means for Spyre users
Some SGLang features required no Spyre-specific changes. RadixCache, which provides prefix sharing, is handled by SGLang’s scheduler before execution reaches the accelerator, so the existing scheduling logic worked unchanged. SGLang’s device-agnostic KV pool likewise required only a single method override for buffer creation, although allocation still needed PyTorch-based handling where the existing path relied on Triton.
The plugin is available as open source alongside upstream SGLang. PyTorch says future Spyre software changes are expected to add dynamic shapes over the coming months, reducing some of the shape management required today. Later this year, the team also expects Triton support for Spyre and RadixAttention on top of it.
For users, the work provides a route to run SGLang’s serving stack on Spyre while retaining shared PyTorch support and features such as prefix sharing. For developers, the post presents the integration as a repeatable model: keep hardware support in PyTorch and connect serving frameworks through defined extension points.
PyTorch: Integrating SGLang into Spyre in Under a Week with AI


