DeepSeek Open-Sources DeepInfer Inference Framework

DeepSeek open-sourced DeepInfer on GitHub on the 20th (local time), describing it as an inference framework for running its AI models in production settings. The company said it was sharing technology used in its own operations to help outside developers lower model-running costs and use computing resources more efficiently. DeepInfer supports production deployment of DeepSeek-V4.1-Flash and can run on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The public release supports offline inference only.

The current version lacks a KV cache system (memory that stores earlier computation results), separate processing for prefill and decoding, and load balancing, so users need to build additional infrastructure. In DeepSeek’s measurements, prefill was tested on 4 GPUs and decoding on 64 GPUs. The company measured performance separately for prefill, which processes input data, and decoding, which generates response tokens. In a decoding test with 256 simultaneous requests per rank, one GPU processed more than 25,000 output tokens per second. DeepSeek said a single cluster could process 155 trillion tokens per day, but this was its own measurement and has not been independently verified against other inference frameworks under the same conditions.

Increasing batch size raised overall throughput, but gains slowed beyond a certain point while latency rose sharply. When the data unit increased from 4,096 to 65,536, throughput per GPU per second rose from 60,000 to 89,000, while latency increased from 70 milliseconds to 730 milliseconds. This result measures the input stage locally; it differs from the full service time between receiving a request and delivering the first token to a user. In actual services, tokenization, queues, cache lookups, network transfers, and whether past input data can be reused also affect efficiency.

DeepInfer combines GPU kernels and system optimizations, including FlashMLA for attention operations, DeepGEMM for matrix operations, and DeepEP for communication between GPUs. It distributes Mixture of Experts (a model design that activates selected specialist networks) computations across multiple GPUs and overlaps CPU task scheduling with GPU computation. Speculative decoding (drafting candidate tokens for a main model to verify) can speed up response generation; DeepInfer adjusts verification work based on the likelihood that each token will be accepted. The size of the performance improvement can vary with the number of simultaneous requests and working conditions.

DeepSeek said DeepInfer reduces prefill computation for long contexts by using the most recent 128 tokens in sliding-window attention to reconstruct the required state. It also provides controls for output formats using regular expressions, JSON schemas, and EBNF grammars. DeepSeek said on X that it was sharing its production stack so the community could reduce computing costs and energy use and make AI more affordable. Released under the Apache 2.0 license, DeepInfer lets developers and companies review and modify the code for their own inference environments.

DeepSeek open-sourced DeepInfer on GitHub on the 20th (local time), describing it as an inference framework for running its AI models in production settings. The company said it was sharing technology used in its own operations to help outside developers lower model-running costs and use computing resources more efficiently. DeepInfer supports production deployment of DeepSeek-V4.1-Flash and can run on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The public release supports offline inference only.

The current version lacks a KV cache system (memory that stores earlier computation results), separate processing for prefill and decoding, and load balancing, so users need to build additional infrastructure. In DeepSeek’s measurements, prefill was tested on 4 GPUs and decoding on 64 GPUs. The company measured performance separately for prefill, which processes input data, and decoding, which generates response tokens. In a decoding test with 256 simultaneous requests per rank, one GPU processed more than 25,000 output tokens per second. DeepSeek said a single cluster could process 155 trillion tokens per day, but this was its own measurement and has not been independently verified against other inference frameworks under the same conditions.

Increasing batch size raised overall throughput, but gains slowed beyond a certain point while latency rose sharply. When the data unit increased from 4,096 to 65,536, throughput per GPU per second rose from 60,000 to 89,000, while latency increased from 70 milliseconds to 730 milliseconds. This result measures the input stage locally; it differs from the full service time between receiving a request and delivering the first token to a user. In actual services, tokenization, queues, cache lookups, network transfers, and whether past input data can be reused also affect efficiency.

DeepInfer combines GPU kernels and system optimizations, including FlashMLA for attention operations, DeepGEMM for matrix operations, and DeepEP for communication between GPUs. It distributes Mixture of Experts (a model design that activates selected specialist networks) computations across multiple GPUs and overlaps CPU task scheduling with GPU computation. Speculative decoding (drafting candidate tokens for a main model to verify) can speed up response generation; DeepInfer adjusts verification work based on the likelihood that each token will be accepted. The size of the performance improvement can vary with the number of simultaneous requests and working conditions.

DeepSeek said DeepInfer reduces prefill computation for long contexts by using the most recent 128 tokens in sliding-window attention to reconstruct the required state. It also provides controls for output formats using regular expressions, JSON schemas, and EBNF grammars. DeepSeek said on X that it was sharing its production stack so the community could reduce computing costs and energy use and make AI more affordable. Released under the Apache 2.0 license, DeepInfer lets developers and companies review and modify the code for their own inference environments.