fsdp
Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan
TL;DR Scaling pretraining for a ~1-trillion-parameter Mixture-of-Experts (MoE) model on shared, public infrastructure was previously considered unfeasible due t…






