Torchtitan

Pretrain LLMs at scale with PyTorch 4D parallelism.

By Orchestra Research · Official Docs · Date unknown

TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.

Platforms: linux, macos.

hermes skills install official/mlops/torchtitan

• #skills
• #optional
• #mlops
• #model architecture
• #distributed training
• #torchtitan
• #fsdp2