DeepSeek has open sourced DeepInfer, the inference stack behind V4.1 Flash, and published production figures of 155 trillion tokens per day from one cluster with 96% input-cache hits. The company reports 400 ms average time to first token and 230 tokens per second generation, rising to 300 tokens per second for coding. At off-peak prices it claims $38 of revenue per accelerator-hour, which matters because it reframes inference as a high-margin operation rather than a cost center.
How DeepInfer serves V4.1 Flash in production
DeepInfer covers distributed inference, speculative decoding, SWA Bounded Replay and structured output on NVIDIA and Huawei Ascend platforms. A companion deepseek-recipe component handles production token encoding and decoding plus processing compatible with the official API. DeepSeek says future model inference implementations will also appear in the same repository. The release was timed to PyTorch Conf and described as a customization of vLLM with contributions from SGLang and TRTLLM work.
The disclosed architecture separates vision encoding, prefill and decode so each stage scales against its own bottleneck. Recommended shapes are eight-accelerator machines for prefill and 64-128-accelerator supernodes for decode, with wide expert parallelism spreading MoE experts across devices. Prefill scheduling allocates work by estimated attention-compute cost rather than raw request length, using variable chunk sizes fitted to a time budget. Decode routing balances both running-request count and accumulated KV-cache length, while asynchronous CPU scheduling removes idle gaps between decode steps.
The cache hierarchy explains much of the reported throughput. Persistent block KV resides in a separate 3FS storage cluster with about 100 PiB of SSD capacity, with retention of three days for agent traffic and seven days for chatbot traffic. Because cached prefixes are pooled in shared storage, a request can go to any available prefill worker instead of the worker that served its prefix. That design suits agents that repeatedly resend growing histories of instructions, repository content, tool outputs and prior responses. Popular experts are also replicated and periodically redistributed to reduce waiting from routing skew.
What cached-token economics means for business
For companies running agents and assistants, the practical effect is lower cost per completed task when prompts repeat. Production data cited in the report points to 1.5 million cache-inclusive input tokens per second per prefill GPU and 14,000 output tokens per second per decode GPU across roughly 2,400 GPUs. Off-peak pricing is listed at $0.15 per million input tokens and $0.60 per million output tokens. Large deployments with steady agent traffic gain most, while small firms with varied prompts will see smaller savings from caching.
The limits deserve equal attention before procurement decisions. Although described as fully open source, KV-cache management and load balancing are not yet fully released because they remain coupled to internal infrastructure. The 155 trillion tokens, margin above 80% at $5-8 all-in cost per card per hour, and 81-day payback are first-party claims, not independently audited measurements. The vLLM team reported decode throughput within 6% of DeepInfer on GB200 in a matched configuration, so efficiency alone does not prove a unique stack. Buyers should ask which costs are included and how cache-hit accounting affects billed tokens.
The marker to watch is reproduction and deployment beyond DeepSeek. If independent teams replicate the vLLM comparison and publish stable recipes for eight-GPU prefill and large decode supernodes, cached-prefix serving will become a standard procurement requirement. If retention of three to seven days proves economical on commercial storage, vendors will compete on cache lifetime rather than raw tokens per second.
