Overview

MiniMind is a groundbreaking open-source project that democratizes Large Language Model (LLM) development. It empowers individuals to train a 64M-parameter LLM from scratch in just 2 hours and for as little as $3. This project breaks down the barriers of cost and time typically associated with LLM training, making advanced AI accessible to everyone. Beyond training, MiniMind provides a comprehensive, from-scratch codebase covering the entire LLM lifecycle, serving as both a practical implementation and an educational resource.

Key Features

• Ultra-Fast & Low-Cost Training: Train a 64M-parameter LLM from zero in approximately 2 hours on a single GPU (NVIDIA 3090), with estimated GPU rental costs around $3.
• Lightweight & Accessible: The core MiniMind model is roughly 1/2700th the size of GPT-3, designed for easy training and replication on personal GPUs.
• End-to-End Training Pipeline: Features a complete, from-scratch codebase covering Pretraining, Supervised Fine-Tuning (SFT), LoRA, RLHF (DPO), RLAIF (PPO/GRPO/CISPO), Tool Use, Agentic RL, Adaptive Thinking, and Model Distillation.
• Pure PyTorch Implementation: All core algorithms and modules are natively implemented in PyTorch, avoiding high-level framework abstractions for deeper understanding.
• Extensible Architecture: Supports expansion into multimodal models (MiniMind-V, MiniMind-O), diffusion models, and linear attention models.
• Comprehensive Dataset: Provides fully open-sourced datasets for all stages of training, including collection, distillation, cleaning, and deduplication.
• Community & Education Focus: Designed as a tutorial and practical guide for LLM newcomers, encouraging hands-on learning and community contribution.
• Broad Compatibility: Seamlessly integrates with popular frameworks like transformers, trl, peft, and inference engines such as llama.cpp, vllm, and ollama.

Typical Use Cases

• Personal LLM Development: Train and fine-tune custom LLMs on your own hardware or cloud instances without prohibitive costs.
• AI Education and Research: A perfect platform for students and researchers to understand LLM internals by building and modifying models from the ground up.
• Rapid Prototyping: Quickly experiment with different LLM architectures, training strategies, and fine-tuning techniques.
• Building Lightweight AI Applications: Deploy efficient LLMs for applications where computational resources are limited.
• Exploring Advanced AI Concepts: Experiment with cutting-edge techniques like RLHF, RLAIF, Tool Use, and Agentic RL in a reproducible environment.