HiCache Coordinates Distributed Model Serving

The LMSYS team describes how HiCache coordinates reusable key-value (KV) cache data across context parallelism (splitting work across ranks), pipeline parallelism (splitting model layers across stages), speculative decoding, and prefill-decode disaggregation. For production workloads from Iflytek Astron MaaS, requests typically carry 30K to 200K input tokens, with some reaching 1M. The team reports that adding HiCache with external storage raised the cache hit rate from around 30% to 95% and brought median time to first token below 3 seconds.

In context parallelism, ranks keep the same logical radix tree and page IDs while each GPU stores only its assigned KV pages. A page’s logical ID determines its owner rank and local address through arithmetic, without a per-page lookup table. When a request extends a cached prefix, the allocator preserves its page ownership rotation and rejects insertions that would mix incompatible rotations. Before attention reads a cached prefix, ranks gather their pages one layer at a time into temporary buffers; two alternating buffers let the next layer’s gathering overlap with the current layer’s computation. On supported configurations, GPU copy engines perform the gathering instead of NCCL SM kernels.

HiCache backups and restores each page on its owner rank, even if its physical address changes. For each layer, gathering waits for restore completion, and attention waits for the gathered data. In a production comparison using GLM-5.2-NVFP4, TP4/CP4, EAGLE MTP, and a static memory fraction of 0.9, each deployment used two prefill engines with four NVIDIA GB300 GPUs apiece. The baseline and sharded setup each had a target host-cache budget of 960 GB per engine; the sharded setup allocated 240 GB per rank across four ranks. Over a three-hour traffic window, each processed approximately 24.5K requests with known cache-tier attribution. Mean input lengths were 136K–137K tokens, and total input volume differed by 1.8%. With sharding, GPU token hit rate rose by 40.48 percentage points while the combined GPU and host hit rate stayed near 93%. Host-served tokens fell from 1.712B to 0.322B, an 81.2% reduction; mean prefill queue time was 56.2% lower and mean prefill forward time was 31.3% lower. The team says per-layer gathering adds overhead and that CP sharding remains under optimization.

In pipeline parallelism, stages must apply cache changes in the same order despite transfers completing at different times. The pp_sync protocol has the first stage agree on the count of ready transfers, then sends that count to later stages; each stage waits for its selected transfer before updating cache records. For external storage, background threads agree on the shortest prefix loaded across ranks and publish progress through ordered acknowledgment queues. Stages consume the same acknowledgments at each cache check, so they use the same cached prefix for a request even when later read progress is already available.

Speculative decoding also requires restoring the draft model’s KV, indexer buffers, and separately owned recurrent state alongside the target model’s prefix. HiCache offers a packed layout, which shares slot mapping when row shapes, data types, and token indexing are compatible, and a sidecar layout, which keeps separate host pools and derives transfer indices from a declared source such as target KV or SWA. Both include draft-state copies in transfer-completion events before buffers are read.

The LMSYS team describes how HiCache coordinates reusable key-value (KV) cache data across context parallelism (splitting work across ranks), pipeline parallelism (splitting model layers across stages), speculative decoding, and prefill-decode disaggregation. For production workloads from Iflytek Astron MaaS, requests typically carry 30K to 200K input tokens, with some reaching 1M. The team reports that adding HiCache with external storage raised the cache hit rate from around 30% to 95% and brought median time to first token below 3 seconds.

In context parallelism, ranks keep the same logical radix tree and page IDs while each GPU stores only its assigned KV pages. A page’s logical ID determines its owner rank and local address through arithmetic, without a per-page lookup table. When a request extends a cached prefix, the allocator preserves its page ownership rotation and rejects insertions that would mix incompatible rotations. Before attention reads a cached prefix, ranks gather their pages one layer at a time into temporary buffers; two alternating buffers let the next layer’s gathering overlap with the current layer’s computation. On supported configurations, GPU copy engines perform the gathering instead of NCCL SM kernels.

HiCache backups and restores each page on its owner rank, even if its physical address changes. For each layer, gathering waits for restore completion, and attention waits for the gathered data. In a production comparison using GLM-5.2-NVFP4, TP4/CP4, EAGLE MTP, and a static memory fraction of 0.9, each deployment used two prefill engines with four NVIDIA GB300 GPUs apiece. The baseline and sharded setup each had a target host-cache budget of 960 GB per engine; the sharded setup allocated 240 GB per rank across four ranks. Over a three-hour traffic window, each processed approximately 24.5K requests with known cache-tier attribution. Mean input lengths were 136K–137K tokens, and total input volume differed by 1.8%. With sharding, GPU token hit rate rose by 40.48 percentage points while the combined GPU and host hit rate stayed near 93%. Host-served tokens fell from 1.712B to 0.322B, an 81.2% reduction; mean prefill queue time was 56.2% lower and mean prefill forward time was 31.3% lower. The team says per-layer gathering adds overhead and that CP sharding remains under optimization.

In pipeline parallelism, stages must apply cache changes in the same order despite transfers completing at different times. The pp_sync protocol has the first stage agree on the count of ready transfers, then sends that count to later stages; each stage waits for its selected transfer before updating cache records. For external storage, background threads agree on the shortest prefix loaded across ranks and publish progress through ordered acknowledgment queues. Stages consume the same acknowledgments at each cache check, so they use the same cached prefix for a request even when later read progress is already available.

Speculative decoding also requires restoring the draft model’s KV, indexer buffers, and separately owned recurrent state alongside the target model’s prefix. HiCache offers a packed layout, which shares slot mapping when row shapes, data types, and token indexing are compatible, and a sidecar layout, which keeps separate host pools and derives transfer indices from a declared source such as target KV or SWA. Both include draft-state copies in transfer-completion events before buffers are read.