DeepSeek open-sources its production inference engine
Why it matters
DeepSeek released DeepInfer on GitHub on Oct 20 under Apache 2.0: the in-house inference framework that powers DeepSeek V4.1 Flash in production, not a demo. DeepSeek says a single cluster serves 155 trillion tokens a day, at about US$38 revenue per GPU-hour at off-peak ("valley") pricing. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with optimized kernels (FlashMLA, DeepGEMM), MoE expert parallelism, structured outputs, deterministic sampling and DSpark speculative decoding, which verifies about 30% fewer draft tokens for +4-7% throughput at the production operating point. deepseek-recipe keeps token handling aligned with the official DeepSeek API. Caveat: the KV cache system and load balancing are not included yet, and the smallest efficient deployment is 64 GPUs for prefill plus 64 for decode.
Read the source: DeepSeek (opens in a new tab)
All hits from 23 Oct 2026More Open ModelsGet the daily hits by email