Friday, Nov 13 | --:--
Times of AI

Regions

China Europe India Japan Taiwan United Kingdom United States

Platforms

Alibaba Qwen Amazon Anthropic DeepMind Google Groq Meta Microsoft MiniMax Mistral Moonshot AI NVIDIA OpenAI PrismML Replit Thinking Machines xAI

Sections

Business Engineering Enterprise Frontier Labs Hardware Industry Infrastructure Models Policy Products Research Safety Security Work All articles → Subscribe by email
Back to home
Infrastructure Open Source China

DeepSeek Open-Sources DeepInfer as an Offline-Inference Release for Nvidia SM100 and Huawei Ascend 950

DeepSeek released DeepInfer under Apache 2.0 on Oct. 20, describing it as the inference framework behind its large-scale production services. The repo currently supports only DeepSeek-V4.1-Flash on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, and omits KV-cache, prefill/decode disaggregation and replica load balancing needed for a full online service. We omit DeepSeek's speed figures.

Times of AI Desk 3 min read Hangzhou View as Markdown
Engraved inference rack spanning an Nvidia GPU plate and a Huawei Ascend plate, teal intaglio.

Serving software usually stays secret; DeepSeek has published an offline-inference release of the framework it says runs its production services, with Huawei Ascend as a supported target. DeepSeek announced DeepInfer on Tuesday, October 20 and released it on GitHub under Apache 2.0. The README describes it as the high-performance inference framework behind DeepSeek's large-scale production services. For now it supports only DeepSeek-V4.1-Flash, loading the V4.1-Flash-1017 checkpoint by default.

What is in the release#

DeepInfer runs on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. Documented features include DeepSeek's optimised kernels (FlashMLA, DeepGEMM, DeepEP and others), attention data-parallel and MoE expert-parallel scheduling, DSpark speculative decoding, an asynchronous scheduler, batch-invariant inference, deterministic sampling and structured outputs. DeepSeek says the project was originally built mainly on vLLM, with inspiration from SGLang and TensorRT-LLM.

It is an offline-inference release. The README lists what a complete inference service still needs and this release does not include: a KV-cache system, VE/prefill/decode disaggregation, and load balancing across data-parallel replicas.

We leave DeepSeek's published throughput and latency figures out of this piece; The Next Web, citing Bloomberg, notes they have not been independently tested. We covered the 1017 open-weight checkpoint separately. DeepInfer arrives a day after MeriTalk reported the DHS FASCSA order removing DeepSeek from civilian federal IT (Federal Register notice filed Monday for Tuesday publication), and after reporting that Huawei engineers are co-designing Ascend chips with DeepSeek.

Limits#

  • Performance numbers in the README and secondary coverage are DeepSeek's; we do not treat them as verified here.
  • Offline only: no KV cache, prefill/decode disaggregation or DP load balancing in this release.
  • TNW says V4.1-Flash launched in September; the 1017 checkpoint we covered shipped October 17.

Sources#

Related Coverage

Scroll to continue reading