From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning
AI-generated summary Researchers analyze mixture-of-experts models that share experts across layers or computational steps to study how local prediction error reduction affects final outputs. The study examines linear regression models and trained Transformers to determine when sharing improves local estimation and when those gains survive to the final output.