4-bit training: Six times faster on paper, 8 percent in practice
AI-generated summary Thinking Machines Lab explained in its blog how it trained Inkling 2 Small, a mixture-of-experts model with 594 billion parameters. The lab says the model was trained using a native 4-bit forward pass from the first step of pretraining to the last reinforcement learning run.