DeepSeek open-sources DeepInfer, an inference tool to cut AI running costs

Chinese AI company DeepSeek released DeepInfer on GitHub on October 20, 2026, making public the inference framework it says already powers its own large-scale production services. The software is published under the Apache 2.0 license at github.com/deepseek-ai/DeepInfer, according to SiliconANGLE and TechTarget.

Inference is the stage after a model has been trained, when it actually generates answers to user prompts. DeepSeek said in its release notes, as quoted by Gizmodo, that it is "sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable."

What DeepInfer does

DeepInfer is an optimization layer built specifically for DeepSeek's V4.1-Flash model, a system the company markets for coding and agentic tasks, according to Gizmodo and TechTarget. SiliconANGLE reports that one of DeepSeek's own production clusters uses the framework to serve 155 trillion tokens a day.

The framework relies on speculative decoding, a technique in which a smaller "draft" model proposes several tokens ahead of time while the main model checks them in a single pass rather than generating every token one by one. SiliconANGLE, citing DeepSeek's technical documentation, describes DeepSeek's specific version of this system, called DSpark, in which a draft model proposes five tokens while a separate confidence mechanism predicts whether the main model will accept them; DeepSeek says this cuts unnecessary verification by about 30%, producing throughput gains of 4% to 7% under production-scale loads.

The released code supports Nvidia's SM100 (Blackwell) GPUs and Huawei's Ascend 950 neural processing units, but not older Nvidia Hopper-generation chips, per SiliconANGLE. TechTarget notes that enterprises running AI models other than DeepSeek's can already use comparable tools such as Nvidia's TensorRT-LLM, and that DeepSeek's continued reliance on Nvidia hardware alongside Huawei chips indicates Huawei still trails Nvidia in some respects, according to the report.

What is left out

DeepSeek has not released everything needed to run a full production service. SiliconANGLE reports the company withheld its key-value cache management system and load-balancing code because they are tightly coupled to DeepSeek's own internal infrastructure, and that an efficient deployment still requires a minimum of 128 GPUs — 64 dedicated to processing prompts and 64 to generating responses. Deterministic-output features, which make the model return consistent answers for the same input, currently work only on Nvidia hardware, the outlet adds.

Cost comparison and reaction

Gizmodo reports DeepSeek prices V4.1-Flash at $0.035 per million input tokens and $0.12 per million output tokens, compared with the figures it cites for OpenAI's GPT-6 Astra model of $10 and $50 respectively for the same volumes; Gizmodo frames the release as part of DeepSeek applying pressure on OpenAI and Anthropic. SiliconANGLE separately reports that off-peak pricing on a DeepInfer-run cluster can generate roughly $38 in revenue per GPU per hour.

Omdia analyst Lian Jye Su told TechTarget there is considerable industry discussion about optimizing token usage, noting that token consumption has risen sharply as AI adoption grows. TechTarget describes analysts as viewing the release both as a cost-optimization move and as a geopolitical positioning effort that also promotes Chinese chip alternatives alongside Nvidia hardware.

Background

SiliconANGLE reports that DeepInfer was originally forked from the open-source project vLLM in April 2025, after the company found a full release impractical at the time. DeepSeek has previously drawn attention for releasing low-cost, competitive AI models, and the DeepInfer release continues that pattern by opening up infrastructure software rather than only a model.

DeepSeek has said it plans to add support for additional models to the DeepInfer framework in the future, per Gizmodo, though no timeline has been given.