A few weeks ago, during Open Source Week, we open-sourced several libraries. The response from the community has been incredibly positive - sparking inspiring collaborations, productive discussions, and valuable bug fixes. Encouraged by this, we’ve decided to take another step forward: contributing our internal inference engine back to the open-source community.
We are deeply grateful for the open-source ecosystem, without which our progress toward AGI would not be possible. Our training framework relies on PyTorch, and our inference engine is built upon vLLM, both of which have been instrumental in accelerating the training and deployment of DeepSeek models.
Given the growing demand for deploying models like DeepSeek-V3 and DeepSeek-R1, we want to give back to the community as much as we can. While we initially considered open-sourcing our full internal inference engine, we identified several challenges:
- Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago. Although structurally similar, we’ve heavily customized it for DeepSeek models, making it difficult to extend for broader use cases.
- Infrastructure Dependencies: The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications.
- Limited Maintenance Bandwidth: As a small research team focused on developing better models, we lack bandwidth to maintain a large open-source project.
Considering these challenges, we’ve decided to collaborate with existing open-source projects as more sustainable alternatives.
Moving forward, we will work closely with existing open-source projects to:
- Extract Standalone Features: Modularize and contribute reusable components as independent libraries.
- Share Optimizations: Contribute design improvements and implementation details directly.
We are profoundly grateful for the open-source movement - from operating systems and programming languages to machine learning frameworks and inference engines. It’s an honor to contribute to this thriving ecosystem and to see our models and code embraced by the community. Together, let’s push the boundaries of AGI and ensure its benefits serve all of humanity.
Note
To clarify, this article outlines our approach to open-sourcing of our DeepSeek-Inference-Engine codebase only. Regarding future model releases, we maintain an open and collaborative stance towards both the open-source community and hardware partners. We commit to proactively synchronizing inference-related engineering efforts prior to new model launches, with the goal of enabling the community to achieve state-of-the-art (SOTA) support from Day-0. Our ultimate aim is to foster a synchronized ecosystem where cutting-edge AI capabilities can be seamlessly implemented across diverse hardware platforms upon official model releases.