torchtitan

TorchTitan is the PyTorch team's framework for training large language models at scale. It combines PyTorch's distributed training primitives into a working system: data parallelism, tensor parallelism, pipeline parallelism, and activation checkpointing. Fully free, BSD-licensed, no cloud requirement.

This is not a weekend project. You need multi-GPU clusters (H100 or equivalent) and familiarity with SLURM or cloud HPC to orchestrate across nodes. The project supports multiple architectures including Llama 3 and 4, DeepSeek V3, Qwen3, and Flux for image generation. It ships with configuration examples but expects you to already understand distributed training before you start.

ML researchers and infrastructure teams training large-scale models from scratch have the cleanest PyTorch-native starting point available. Solo developers building on top of existing models have no use for this. It is infrastructure for teams running their own training clusters who want to stay in the PyTorch ecosystem.

The catch: TorchTitan is a reference implementation, not a hardened production system. The APIs are bleeding-edge and change frequently.