LitGPT
llm-training | llm-from-scratch
Overview
LitGPT is a PyTorchLightning-based framework for pretraining, finetuning, and evaluating large language models. It provides reproducible training recipes for many open-source LLMs.
Key Features
- 20+ pretrained models: LLaMA, Mistral, Phi, Qwen, etc.
- Training recipes: SFT, DPO, RLHF, LoRA, QLoRA
- Benchmarking: Built-in evaluation on downstream tasks
- Flash attention: Native FlashAttention-2 support
Relationship to Other Projects
- Alternative to nanoGPT for production-scale training
- Related to litgpt (same project, alternate naming)