torchtitan 0.4.0

A PyTorch-native platform for pretraining and post-training generative AI models at scale

Quick start · Models · Features · RL · Docs · Contributing

torchtitan is a clean-room, PyTorch-native implementation of large-scale training. One readable codebase takes a model from pretraining through supervised fine-tuning to reinforcement learning, composing FSDP, tensor, pipeline, context, and expert parallelism without rewriting the model.

• Composable parallelism. Models declare how they shard; FSDP2, TP, PP, CP, and EP stack into N-D layouts through DeviceMesh and DTensor.
• Built for research. Minimal, hackable code; every run is a Python recipe you can read, copy, and modify.
• Pretraining to RL in one stack. TitanRL runs the same model definitions and kernels in the trainer and in vLLM generation, with a bitwise-reproducible batch-invariant mode.
• Frontier models and techniques. DeepSeek, Qwen, Kimi, gpt-oss, Llama, and FLUX, with MXFP8/NVFP4 low precision, Dist-MoE, CUDA graphs, and more.

Latest news

• [2026/08] TitanRL is a hackable RL stack for scaling and debugging. It reuses TorchTitan model definitions and kernels across training and vLLM generation and supports batch-invariant mode.
• [2025/11] AMD released an optimized fork of torchtitan for AMD GPUs.
• [2025/10] We released torchtitan v0.2.0.
• [2025/10] SkyPilot now supports torchtitan! See the tutorial.

Quick start

1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build.

git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt

2. Train a debug model. It uses the tokenizer bundled with the tests, so no download is needed.

NGPU=1 ./run_train.sh   # Llama 3 debug model, 10 steps

3. Train a real model. Download the tokenizer (Llama weights require access on Hugging Face), then select a recipe by module and function name:

python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh   # 8 GPUs

4. Make it yours. A recipe is a Python function returning a complete config, so changing a run means writing a function, not passing flags:

# my_recipes.py
from torchtitan_recipes.models.llama3 import llama3_8b

def llama3_8b_tp2():
    config = llama3_8b()
    config.parallelism.tensor_parallel_degree = 2
    return config

MODULE=my_recipes CONFIG=llama3_8b_tp2 ./run_train.sh

See the configuration guide for the full model, and torchtitan_recipes for the verified recipes.

Supported models

Every model has debug configurations exercised in CI. Verified recipes have been run in training, convergence, or performance workflows. To add a model, see the model guide.

Features

We report performance on up to 512 GPUs and verify loss convergence across parallelisms and techniques.

Test status

Hardware	Integration tests	Unit tests
CPU
NVIDIA GPU
AMD GPU (ROCm)
RL (NVIDIA GPU)

Installation

The quick start installs from source, which is the recommended way to get the latest features. Other options:

pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132 --force-reinstall
pip install --pre torchtitan --index-url https://download.pytorch.org/whl/nightly/cu132

pip install torchtitan
# or
conda install conda-forge::torchtitan

#SBATCH --ntasks=2
#SBATCH --nodes=2

srun torchrun --nnodes 2 ...

Documentation

Contributing

We look forward to your contributions!

• New ideas should start in the experiments folder; follow the experiments guidelines.
• For fixes and contributions to core, follow the contributing guidelines.

Citation

Our paper gives a detailed look at the parallelisms and optimizations in torchtitan, with advice on when to use each technique: TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training.

@inproceedings{
   liang2025torchtitan,
   title={TorchTitan: One-stop PyTorch native solution for production ready {LLM} pretraining},
   author={Wanchao Liang and Tianyu Liu and Less Wright and Will Constable and Andrew Gu and Chien-Chin Huang and Iris Zhang and Wei Feng and Howard Huang and Junjie Wang and Sanket Purandare and Gokul Nadathur and Stratos Idreos},
   booktitle={The Thirteenth International Conference on Learning Representations},
   year={2025},
   url={https://openreview.net/forum?id=SFN6Wm7YBI}
}

License

Source code is made available under a BSD 3 license. You may have other legal obligations that govern your use of other content linked in this repository, such as the license or terms of service for third-party data and models.