Artificial intelligence news.
Straight from the original announcements.

vLLM Adds Day-One Support for Thinking Machines’ Inkling 2 Small

November 12, 2026By the AI News Bites Editorial TeamSource: vLLM
vLLM Adds Day-One Support for Thinking Machines’ Inkling 2 Small
Image: vLLM

vLLM has added day-zero support for Thinking Machines’ Inkling 2 Small, a multimodal mixture-of-experts model that accepts text and images and produces text. The 48-layer model has 594B total parameters, 17B active parameters and a context window of up to 1M tokens.

Key points
  • vLLM supports Thinking Machines’ Inkling 2 Small on day zero.
  • The multimodal MoE model has 594B total and 17B active parameters.
  • Its context window reaches up to 1M tokens.
  • Native DSpark drafts seven tokens per step.
  • vLLM reports up to 630 tokens per second for a single user at TP4.

A larger multimodal model with a revised architecture

Inkling 2 Small uses 384 routed experts with top-8 routing across all layers. Its 48 layers include 40 sliding-window attention layers and eight full-attention layers, while the model uses a 1,024-token sliding window.

The model retains parts of the original Inkling backbone, including grouped-query attention with learned relative bias, four short convolutions per layer and a sigmoid-routed MoE design. It also introduces block attention residuals, gated attention with a per-token attention sink, and a 23-layer ViT image encoder operating on 20×20 patches.

Compared with Inkling, the new model accepts text and images rather than text, images and audio. vLLM says the model can be served on a single node because its mixed-precision MoE uses NVFP4 in layers 0, 39 and MXFP8 in layers 40, 47.

Native DSpark support and vLLM optimizations

Inkling 2 Small ships with a native DSpark speculator. Its five-layer RNN head drafts seven tokens per step, replacing Inkling’s eight chained MTP heads. vLLM supports DSpark speculative decoding and also provides reinforcement learning support through SkyRL, along with LoRA support.

The integration combines kernel fusion, programmatic dependent launch and sequence parallelism. vLLM says its fused kernels combine communication and memory-bound operations around the model’s attention residuals, while sequence parallelism removes repeated work during prefilling. On an 8K-token prefill, this cuts the step by 22% at TP4 and 26% at TP8, according to the company, and frees 1.3, 1.9 GiB per GPU for KV cache.

vLLM also developed a warp-specialized Gluon kernel for decode. In its test, a batch-1 decode step at 100K context fell from 15.1 ms to 4.8 ms. The company attributes the change to packing six attention heads into one tile and reading the key and value data once.

Reported throughput and accuracy checks

In an agentic workload using TP8, DSpark with probabilistic drafting and prefix caching, vLLM measured output throughput per GPU at a concurrency where each user received 100 tokens per second. At 32K context, the result was about 1,070 output tokens per second per GPU across about 84 concurrent sessions, with a 91% prefix-cache hit rate and 3.3 acceptance length.

At 64K, 128K and 256K context, the reported throughput was about 915, 700 and 450 output tokens per second per GPU, respectively. The corresponding concurrent session counts were about 72, 55 and 37. For a single user at TP4, vLLM reports about 630 tokens per second with an acceptance length of 3.5.

vLLM says its implementation preserves the reference numerics, including FP32 attention output and router calculations and the same NVFP4 and MXFP8 activation quantization. The team verified accuracy with DSpark enabled on NIAH up to 1M tokens, MMMU-Pro and HLE, and says the scores matched runs without speculative decoding.

Related news

PyTorch team integrates SGLang with IBM Spyre in less than a week
For developersNovember 12, 2026, 15:45

PyTorch team integrates SGLang with IBM Spyre in less than a week

A two-person team integrated SGLang with IBM Spyre in less than a week, using AI coding agents to build an open source plugin of roughly 1,500 lines. PyTorch says SGLang…

Vercel adds Thinking Machines’ Inkling 2 Small to AI Gateway
ModelsNovember 12, 2026

Vercel adds Thinking Machines’ Inkling 2 Small to AI Gateway

Vercel has added Thinking Machines’ Inkling 2 Small to AI Gateway, bringing a multimodal reasoning model for coding agents, visual understanding and professional…

vLLM adds day-0 support for Mistral Large 4, a 1-trillion-parameter MoE model
For developersOctober 29, 2026

vLLM adds day-0 support for Mistral Large 4, a 1-trillion-parameter MoE model

The vLLM inference project now supports Mistral Large 4 from day 0. According to vLLM, the model is Mistral's new frontier mixture-of-experts, natively multimodal model…