Overview

DwarfStar is a high-performance, native inference engine built for running cutting-edge Large Language Models (LLMs) like DeepSeek V4 Flash/PRO, GLM 5.x, and Qwen3.8 Flash Next on consumer-grade hardware. It prioritizes efficient local inference across diverse platforms, including Apple Silicon (Metal), NVIDIA CUDA, and AMD ROCm. DwarfStar is a self-contained, specialized engine, not a general-purpose GGUF runner, focusing on delivering exceptional speed and capability for supported models.

Key Features

• Broad Hardware Support: Runs efficiently on Metal (Macs with 96GB+ RAM, SSD streaming for smaller systems), NVIDIA CUDA (including multi-GPU setups and Ada Lovelace cards), and ROCm (Strix Halo systems).
• Optimized for Top Models: Specifically engineered for DeepSeek V4 Flash & PRO, GLM 5.2 & 5.3, and Qwen3.8 Flash Next, including experimental vision models.
• Advanced Inference Techniques: Leverages SSD streaming for models larger than RAM, supports tensor and pipeline parallelism for distributed inference across machines, and features speculative decoding for performance gains.
• Integrated Testing: The codebase includes comprehensive tools for GGUF handling, imatrix, quality assurance, and speed benchmarking, ensuring robust model integration.
• Native Coding Agent: Offers a streamlined ds4-agent for direct inference without an HTTP server, preserving token history and model state for a smoother coding experience.
• Flexible Model Loading: Supports running models directly from disk using techniques like Engram tables, with fast local SSDs being crucial for optimal performance.
• AI-Assisted Development: Developed with strong AI coding assistance, this project embraces the evolving landscape of software development and user customization through AI agents.

Typical Use Cases

• Local LLM Powerhouse: Run highly capable LLMs on your personal workstation, MacBook, DGX Spark, or Framework Laptop, achieving impressive inference speeds even with limited RAM via SSD streaming.
• Multi-User LLM Server: Deploy multiple CUDA cards to create a powerful, multi-user LLM server, supporting newer Ada Lovelace GPUs for models that might otherwise require more advanced hardware.
• Distributed LLM Inference: Combine multiple machines using RDMA for tensor parallelism or pipeline parallelism to sum RAM and run exceptionally large models that exceed single-system capacity.
• Customizable AI Development: Leverage the project's design as a template for fine-tuning and adapting LLM inference for specific hardware, models, and tasks, potentially with the help of AI coding agents for rapid customization.
• Enhanced Coding Assistance: Utilize the ds4-agent for a more integrated and responsive coding experience, managing conversation history and model state seamlessly.