Benchmarks & Evaluation
Benchmarks and evaluation encompass the standard tests, datasets and metrics used to measure and compare artificial intelligence system capabilities and performance. Truncated covers research papers, testing methodologies, score debates and industry standards for evaluating models.
GlobeBench: A Benchmark for Text-Based World Models
patronus.ai — GlobeBench tests whether language models can stand in for an environment by predicting its response to an agent's next action.
A $100 budget buys 40 Intelligence Index tasks (a mix of task archetypes) with Opus 5.5 at medium reasoning effort, or…
In the last 14 days, nearly 30,000 tasks ran for over an hour on Agent Arena.
As tasks get more complex, @GoogleDeepMind and @XiaomiMiMo are the biggest winners in the Agent Arena.
Agent Arena measures complex, real-world tasks.
I gave Claude, ChatGPT, and a local 9B model the same 20 tasks, and only one was worth paying for
XDA — Claude needed less hand-holding once the tasks got complicated.
StateBench: Testing how language models track state changes over long documents
huggingface.co — A Blog post by Ai2 on Hugging Face
No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark
The Decoder — SemiAnalysis is putting AI agents through simulated knowledge work in its new GDPx benchmark, covering everything from chip design to quant trading. No model breaks 12 percent. Claude Opus 5.5 leads narrowly over Gemini 4 Carbon. Critics point to the small number of tasks per domain and uneven testing conditions.
Why AI's Next Breakthrough Depends On Measurement, Not Just Scale
Forbes — The gap between what benchmarks measure and what businesses need is not theoretical, and it does not live where most leadership teams expect.
VB Pulse Agentic Reliability and Evals Report - September 2026
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline perf…
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
The Decoder — CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects.
Exciting news from Agent Arena: Hy4 by @TencentHunyuan just landed #14 overall.
In AA-PersonalityBench, the top open weights models on average rank lower than the top proprietary models on truthfulne…
We’ve updated the six Artificial Analysis Capability Indexes to v2.0, aligning them with evaluations from the Artificia…
We’ve worked with the CritPt authors to finalize the CritPt v2 grading server and will be making this available to the…
Rapid LLM development is outrunning the research meant to explain it, study finds
The Decoder — Stanford researchers tested 50 widely cited claims about LLMs against 20 current models. 44 percent of the findings are already "expired," and another 34 percent have weakened significantly. The average lifespan is just 15.7 months. Claims about what models can't do expire fastest.
CWE-bench v2: More coverage, quality, and transparency
huggingface.co — We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Introducing Physical-Engineering’s Last Exam for agents
Google Cloud Blog — Discover PLE, a new benchmark evaluating frontier AI agents on real-world mechanical, electrical, civil, and structural engineering tasks.
Dismiss email signup
Subscribe to 
The top AI news in your email delivered each morning. Choose categories