Overview

Experience the cutting-edge of local AI with Bonsai Demo, featuring the powerful Bonsai 2 27B model. This repository provides everything you need to run a highly intelligent, compact, and versatile AI directly on your hardware, supporting a wide range of backends including Metal (Mac), CUDA, Vulkan, ROCm, and CPU. Designed for efficiency, Bonsai 2 27B achieves remarkable performance, retaining over 98% of FP16 intelligence while significantly reducing model size.

Key Features

• Bonsai 2 27B On-Device: Run a full 27B-class model locally, offering advanced reasoning capabilities without the need for cloud infrastructure.
• Exceptional Compression: Achieves near-FP16 performance with just ~1.75 bits per weight, drastically reducing VRAM and storage requirements (e.g., 5.9 GB for the PQ2_0 packing).
• Vision Capabilities: Process and understand images, screenshots, and PDFs directly within your conversations.
• Agentic Tool Calling: Seamlessly integrate with external tools using native OpenAI-style tool_calls, enhancing the model's practical application.
• Adaptive Thinking: Control the reasoning effort per chat or per request, balancing speed and accuracy for diverse use cases.
• Massive Context Window: Leverage up to 262K tokens for extensive context management, made practical through a hybrid-attention backbone.
• Broad Platform Support: Runs smoothly on Mac (Metal), Linux, and Windows, with optimized binaries for various GPU architectures and CPU-only operation.
• Simplified Setup: A single ./setup.sh script automates dependency installation, model downloads, and environment configuration for quick deployment.
• Optional Open WebUI Integration: Easily deploy a feature-rich ChatGPT-like interface for chat, vision, and code interpretation.

Typical Use Cases

• Local AI Development & Experimentation: Test and integrate advanced AI models on your own machine, ensuring privacy and full control.
• On-Device Content Analysis: Analyze images, documents, and code without uploading sensitive data.
• Personalized AI Assistants: Build custom assistants capable of understanding context, performing actions via tool calls, and processing visual input.
• Efficient LLM Deployment: Deploy powerful AI models on hardware with limited resources, such as laptops or single GPUs, thanks to aggressive quantization.
• Real-time Interactive Applications: Power applications requiring fast, local AI responses with dynamic reasoning and broad context.
• Agentic Workflows: Automate complex tasks by enabling the AI to interact with external services and tools.