Overview

Miles is a cutting-edge, enterprise-grade reinforcement learning framework purpose-built for post-training large-scale language and vision-language models. Forked from and co-evolving with slime, Miles integrates seamlessly with SGLang for high-throughput rollout and Megatron-LM for scalable training, delivering the precision, stability, and observability essential for trillion-parameter model training. It offers a PyTorch FSDP2 backend as an alternative for those preferring to train HuggingFace implementations, though core recipes and large-scale model support are optimized for Megatron-LM.

Key Features

• Fully Asynchronous RL: Decouples rollout and training workers with configurable on- and off-policy schedules, optimized pipelines for reduced latency, and flexible async rollout and evaluation modes.
• High-Performance Agentic Rollout: Leverages SGLang with a smart router for efficient request distribution, preserves per-request metadata, and ensures fleet health for demanding multi-turn agentic workloads.
• Rapid Weight Updates: Achieves near real-time weight propagation to engines, even for models with trillions of parameters, utilizing P2P RDMA for disaggregated setups.
• Low-Precision Training: Supports advanced formats like MXFP8 and NVFP4 with numerically stable RL recipes to mitigate precision-induced divergence, alongside FP8, INT4 QAT, BF16, and FP16.
• Efficient LoRA & Multi-LoRA: Enables training of frontier-scale models on significantly fewer GPUs with Low-Rank Adapters, which can be directly loaded into SGLang for deployment.
• End-to-End Correctness: Implements Token-in-Token-out (TITO) for seamless data flow without detokenization/retokenization, and Rollout Routing Replay (R3) to eliminate MoE routing mismatches for stable training.
• Robust Fault Tolerance: Automatically recovers and resumes runs in place when SGLang engines fail, ensuring uninterrupted training processes.
• Day-0 Model Support: Offers immediate compatibility with the latest frontier models, alongside extensive hardware support for both NVIDIA and AMD platforms.
• Versatile Recipe & Environment Support: Accommodates a wide range of RL algorithms (GRPO, GSPO, PPO, REINFORCE++), SFT, on-policy distillation, and integrates with numerous agentic environments and task sandboxes.
• Diffusion Model Training: Extends capabilities to diffusion models through the miles_diffusion repository, supporting flow-GRPO, DiffusionNFT, and SFT.

Typical Use Cases

Miles is designed for organizations pushing the boundaries of AI through sophisticated post-training techniques. Its enterprise-grade features make it ideal for:

• Large-Scale LLM and VLM Fine-tuning: Enhancing the capabilities of massive language and vision-language models through reinforcement learning for specialized tasks or improved general performance.
• Agent Development: Training complex AI agents for coding, computer interaction, and other sophisticated tasks that require nuanced decision-making and adaptation.
• Research and Development: Providing a robust and scalable platform for researchers to experiment with novel RL algorithms and distributed training techniques on state-of-the-art models.
• Production Model Optimization: Delivering high-performance, stable, and resilient RL pipelines for companies needing to continuously improve their deployed AI models in demanding environments.
• Multimodal AI Advancement: Post-training vision-language models to achieve greater accuracy and understanding in handling both visual and textual data.