SGLang Reports DeepInfer Benchmark Results

DeepSeek open-sourced DeepInfer, the inference engine behind its service, and its kernels. SGLang measured its own performance against DeepInfer using DeepInfer’s benchmarks on the same hardware. SGLang reports results on par with or ahead of DeepInfer: up to 1.04x prefill throughput on 4x GB300 and 4x B300, and decode steps up to 7.4% shorter on 64x GB300, with results within 1% at the other points.

SGLang integrates DeepSeek kernels including FlashMLA, DeepGEMM, DeepEP, DeepSelect, TileLang and the Mega series: MegaAttention, MegaMoE, MegaGate and Mega-mHC. MegaMoE combines token dispatch, expert matrix calculations and the combine operation in one kernel. MegaAttention is the default attention path on B300 and GB300, for single servers and prefill and decode servers. SGLang uses data parallel attention with expert parallelism; the tests use DEP4 for prefill and DEP64 for decode across 64 GB300 GPUs. It also runs DSpark speculative decoding, which proposes a block of tokens for the target model to verify together, and SWA bounded replay, which processes only the last 128 prompt tokens in late decoder layers. Its Engram tables can stay in host memory or be split across GPUs for direct reads over NVLink.

In prefill tests on 4x GB300, SGLang achieved up to 1.04x DeepInfer’s throughput, leading at 4K, 32K and 64K new tokens per GPU per step and matching it at 8K and 16K. On 4x B300, it led at 4K, 8K and 32K, matched at 16K and 64K, and reached up to 1.04x throughput. In decode tests on 64x GB300, SGLang’s steps were 1.1% to 7.4% shorter at 16 to 256 requests per GPU and matched DeepInfer at 8. The comparison used DeepInfer’s measured curve for prefill, selecting its fastest tested configuration at each point. The benchmark used 128K-token contexts; prefill tested 4K to 64K new tokens per step, while decode used 3 draft tokens per step and converted step time to tokens at 3.25 accepted per step.

SGLang says its speedups come from reducing intermediate memory traffic and kernel launches. It feeds Mega-mHC directly into MegaMoE, uses MXFP8 output from attention for the output projection, and uses fused kernels for prefill compression and request-window KV copies. Its scheduler has prepared the next step on the CPU while the GPU runs the current step since v0.4 in December 2024; DeepInfer’s documentation credits SGLang with that design. SGLang also provides a unified radix cache to reuse KV across requests and conversation turns, prefill and decode on separate servers with selectable KV transfer backends, and a model gateway that balances requests across replicas, including by cached prefixes. The system serves as an API server on NVIDIA GB300, B300, B200 and H200, and AMD MI355X.

DeepSeek open-sourced DeepInfer, the inference engine behind its service, and its kernels. SGLang measured its own performance against DeepInfer using DeepInfer’s benchmarks on the same hardware. SGLang reports results on par with or ahead of DeepInfer: up to 1.04x prefill throughput on 4x GB300 and 4x B300, and decode steps up to 7.4% shorter on 64x GB300, with results within 1% at the other points.

SGLang integrates DeepSeek kernels including FlashMLA, DeepGEMM, DeepEP, DeepSelect, TileLang and the Mega series: MegaAttention, MegaMoE, MegaGate and Mega-mHC. MegaMoE combines token dispatch, expert matrix calculations and the combine operation in one kernel. MegaAttention is the default attention path on B300 and GB300, for single servers and prefill and decode servers. SGLang uses data parallel attention with expert parallelism; the tests use DEP4 for prefill and DEP64 for decode across 64 GB300 GPUs. It also runs DSpark speculative decoding, which proposes a block of tokens for the target model to verify together, and SWA bounded replay, which processes only the last 128 prompt tokens in late decoder layers. Its Engram tables can stay in host memory or be split across GPUs for direct reads over NVLink.

In prefill tests on 4x GB300, SGLang achieved up to 1.04x DeepInfer’s throughput, leading at 4K, 32K and 64K new tokens per GPU per step and matching it at 8K and 16K. On 4x B300, it led at 4K, 8K and 32K, matched at 16K and 64K, and reached up to 1.04x throughput. In decode tests on 64x GB300, SGLang’s steps were 1.1% to 7.4% shorter at 16 to 256 requests per GPU and matched DeepInfer at 8. The comparison used DeepInfer’s measured curve for prefill, selecting its fastest tested configuration at each point. The benchmark used 128K-token contexts; prefill tested 4K to 64K new tokens per step, while decode used 3 draft tokens per step and converted step time to tokens at 3.25 accepted per step.

SGLang says its speedups come from reducing intermediate memory traffic and kernel launches. It feeds Mega-mHC directly into MegaMoE, uses MXFP8 output from attention for the output projection, and uses fused kernels for prefill compression and request-window KV copies. Its scheduler has prepared the next step on the CPU while the GPU runs the current step since v0.4 in December 2024; DeepInfer’s documentation credits SGLang with that design. SGLang also provides a unified radix cache to reuse KV across requests and conversation turns, prefill and decode on separate servers with selectable KV transfer backends, and a model gateway that balances requests across replicas, including by cached prefixes. The system serves as an API server on NVIDIA GB300, B300, B200 and H200, and AMD MI355X.