Overview

oMLX is a highly optimized LLM inference server designed exclusively for Apple Silicon Macs. It brings continuous batching and a sophisticated tiered KV cache (RAM + SSD) to your local machine, all managed seamlessly from a user-friendly macOS menu bar application. This empowers you to run complex LLMs, VLMs, and embedding models efficiently and conveniently on your Mac, bridging the gap between raw performance and effortless usability.

Key Features

• Menu Bar Control: Manage your LLM server directly from your macOS menu bar – start, stop, monitor, and configure without ever touching the terminal.
• Continuous Batching & Tiered KV Caching: Experience boosted throughput with continuous batching and an intelligent KV cache that spans RAM (hot tier) and SSD (cold tier) for persistent, reusable context, even across server restarts.
• Vision-Language Models (VLMs) & OCR Support: Run cutting-edge VLMs and OCR models with the same optimized inference pipeline, supporting multi-image chat and image inputs.
• Multi-Model Serving: Load and manage a variety of model types (LLMs, VLMs, embeddings, rerankers) simultaneously, with intelligent eviction, pinning, and per-model settings.
• Admin Dashboard: A rich, offline-capable web UI for real-time monitoring, model management, direct chat, benchmarking, and integrations with popular tools.
• OpenAI/Anthropic API Compatibility: Seamlessly integrate with existing workflows by using oMLX as a drop-in replacement for popular API endpoints.
• Experimental Multi-Mac Inference: Distribute inference workloads across multiple Macs for enhanced scalability and performance.
• Optimized for Apple Silicon: Leverages the full power of Apple's M-series chips, including optional native custom kernels for significant performance gains on supported models.

Typical Use Cases

• Local AI Development: Run and test LLMs, VLMs, and embedding models on your Mac for rapid prototyping and development without relying on cloud services.
• Enhanced Productivity Tools: Integrate oMLX with tools like Claude Code, OpenClaw, or Copilot for intelligent code completion, analysis, and agent-based workflows directly on your machine.
• Offline AI Capabilities: Deploy powerful AI models for use in offline environments or situations with limited internet connectivity.
• Personal AI Assistant: Build and run custom AI agents and assistants tailored to your specific needs, managed conveniently from your menu bar.
• Research and Experimentation: Explore the latest in LLM and VLM technology with a flexible and performant inference server optimized for Apple hardware.