SGLang Unifies Memory for Model State
- •SGLang engineering post describes unified GPU memory for full-attention KV, sliding-window KV and recurrent state
- •A 32K-token request with a 4K window and 64-token pages uses 512 full-KV pages and 64 window pages
- •Allocator supports speculative draft KV, Kimi K3.1 DSA indices, pipeline parallelism and in-flight GPU work
SGLang’s unified-memory design lets full-attention KV, sliding-window KV and recurrent state share one GPU-memory arena, instead of assigning each type a fixed capacity at startup. The system adjusts how much space each type can use as demand changes. It is described in a SGLang engineering post published on November 11, 2026.
In the earlier setup, each state type had a separate pool sized through startup settings such as --swa-full-tokens-ratio and --mamba-full-memory-ratio. A long-context workload could run out of full-attention space while bounded-state capacity sat unused; many short concurrent requests could create the opposite imbalance. Unified memory keeps each type’s page or slot size, but lets them draw from shared free space. Compaction moves live allocations toward opposite ends of the arena, leaving a free region between them.
The allocator stores each token’s rows across all layers in one entry, so allocation, eviction and page moves cover those rows together. Rows can have different shapes and data types. That lets compatible speculative-decoding draft KV share entries with target KV, and lets Kimi K3.1 place each DSA layer’s top-k index beside its MLA KV. State slots likewise hold the convolution and temporal state for state layers. The design supports models with all three state types, including Inkling, without a separate allocator for each combination.
For a 32K-token request with a 4K sliding window and 64-token pages, the stated steady-state footprint is 512 full-KV pages and 64 window pages. Requests with recurrent state also need a slot. Admission estimates joint demand, including page rounding, expected generation and window rows that may persist temporarily; it is used for generation limits, prefill admission, chunk commitments and prefill–decode disaggregation preallocation. To determine whether that demand fits, the planner checks reuse and in-place growth, compaction of the end bands, and relocation of floating bands. It can assess whether a layout is feasible before choosing a concrete realization plan. With pipeline parallelism, stages exchange arena geometry at loop start and independently check the same logical request counts.
The overlap scheduler can prepare the next step while a previous forward pass is running, so pages cannot always be overwritten as soon as requests release them. A logical free counts toward capacity immediately, while physical reuse waits until unfinished readers or writers are processed or settled. Moves copy data and redirect virtual-to-physical mappings; old readers can finish on the source, and copies must be ordered after old writes. The design also describes pending pages, reusable holes, moved-source protections and fences for bytes beyond a retreating frontier. It works with overlap scheduling, Radix cache, HiCache, prefill–decode disaggregation, speculative decoding, pipeline parallelism, decode context parallelism and data-parallel attention. The available article text ends after mentioning focused tests and a limitation, without giving their details.
The fastest, most reliable AI news, straight to your inbox