Overview

Colibrì is a revolutionary inference engine designed to bring the power of frontier Mixture-of-Experts (MoE) models, ranging from 744 billion to a staggering 2.8 trillion parameters, directly to your existing hardware. Written in pure C with zero external dependencies, Colibrì treats your storage, RAM, and VRAM as a unified memory hierarchy, enabling the streaming of experts from disk on demand. This "tiny engine, immense model" approach democratizes access to cutting-edge AI, making massive models runnable on consumer and heterogeneous systems.

Key Features

• Massive Model Support: Run MoE models with up to 2.8 trillion parameters, including families like GLM, Inkling, Kimi K3, DeepSeek, Qwen, and OLMoE.
• Pure C, Zero Dependencies: A lean and efficient engine that avoids complex build environments and runtime dependencies.
• AI Memory Multitiering: Seamlessly integrates VRAM, RAM, and disk storage as a single hierarchy for inference, drastically reducing hardware requirements.
• Streaming Experts from Disk: On-demand loading of model experts from storage, making immense models feasible on limited fast memory.
• Hardware Agnostic: Designed to run on a wide range of consumer and heterogeneous hardware, from high-end GPUs to CPU-only systems.
• Reproducible Research Platform: Focuses on measurable end-to-end performance and semantic guarantees, fostering open research into inference optimization.
• Live Visualization Dashboard: An intuitive web interface (./coli web) provides real-time metrics on token generation, hardware utilization, and expert residency.
• Optimized I/O and Execution: Employs techniques like batched expert unions, overlapped reads and compute, O_DIRECT, and heterogeneous execution (CPU/GPU overlap).
• Faithful Model Representation: Maintains model precision and semantics without silent changes, offering compressed state options like a 57x smaller KV cache.
• Local Cluster Mode: Distribute inference tasks across multiple machines in your local network.

Typical Use Cases

• Personal AI Exploration: Run and experiment with state-of-the-art large language models on your own hardware, bypassing the need for expensive cloud resources.
• Research and Development: A robust platform for investigating and optimizing inference-side performance across the entire hardware-software boundary, from model formats to scheduling and kernels.
• On-Premise AI Deployments: Deploy and manage massive AI models within private infrastructure where data privacy or network latency is a concern.
• Democratizing AI Access: Enable individuals and smaller organizations to leverage the capabilities of enormous AI models that were previously inaccessible due to hardware constraints.
• Prototyping and Benchmarking: Quickly test and measure the performance of different MoE models and configurations on diverse hardware setups.