Mixture of Experts

AIAI-generated summary Hugging reports that Mixture-of-Experts models scale large language models to hundreds of billions of parameters while consuming significant inference memory. Hugging says its new Higher-Order Pruning of Experts method accounts for pairwise expert interactions and achieves gains on agentic coding tasks.

AIAI-generated summary Thinking Machines has unveiled Inkling 2 Small as its second model generation, featuring 17 billion active parameters out of a total 594 billion. Thinking Machines says the model is strongest in text recognition, infographic interpretation, and visual reasoning.

AIAI-generated summary Thinking Machines released Inkling 2 Small, a 594B-total and 17B-active multimodal MoE trained in 68 hours on 12T tokens. The model features a 1M context window and is available on Tinker with Apache 2.0 weights promised next week.

AIAI-generated summary Baidu has released the ERNIE 5.5 Flash Base open-source foundation model featuring a mixture-of-experts architecture with 280 billion parameters. According to Baidu, the model competes with similarly sized open foundation models in reasoning, code, and long-context tasks.