Announcing Long-Context Training on TPUs with MaxText Context Parallelism- Google Developers Blog
AI-generated summary Google announces long-context training support in its MaxText framework using four context-parallel strategies on Cloud TPU hardware. Google says Llama 3.1 8B achieves 70 percent Model FLOPs Utilization on TPU 5p for sequence lengths up to 4 million tokens.