Agent Evaluation
Agent evaluation is the process of assessing autonomous artificial intelligence systems for task completion, tool use, and reliability. Truncated covers testing methodologies, benchmark results, failure analysis, and validation standards for deployment.
What Is Eval-Driven Development? How to Improve AI Agents
www.langchain.com — Eval-driven development helps you build AI agents against a consistent set of metrics. Define good behavior, write the eval first, then compare experiments.
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test
The Decoder — Alibaba's new coding-agent benchmark, Co-Design Bench, tests AI agents on real user requests that already tripped them up in practice. Even GPT-6 Astra, the top performer, fully solves only 26.4 percent of tasks. On older benchmarks, the same models score above 99 percent.
AI auditors catch most agent misbehavior but flood reviewers with false alarms, new study finds
The Decoder — AI models can often spot problematic behavior by other AI agents in logs, according to a study by Anthropic researcher Andy Hall. GPT-5 found about 98 percent of reward-hacking cases, but the more exhaustively the search is tuned, the more false alarms pile up. That's not yet enough for reliable safety checks.
🚨 Science's Last Exam is out and it's not another Q&A benchmark the agent gets a research goal + tools, picks its own…
Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts' bar
The Decoder — The "Into the Unknown" benchmark tasks nine AI agents with tackling open research questions from U.S. national laboratories. None of the 135 submissions meets all requirements. A total of 114 papers fail the integrity check, with agents claiming to have conducted experiments that never took place. GPT-6 Astra performs best but still doesn't meet the experts' standards.
Measuring how AI agents handle everyday wet-lab science
Two Agent Studies Test Different Kinds of Deception
Forkast — One measured what an incentive-driven game elicited without asking models to deceive; the other measured outcomes after explicit influence prompts. Neither gives a deployment rate.
Visa Finds AI Models Fail 43% of Shopping Test Runs
PYMNTS.com — Visa found that when 22 artificial intelligence models from 12 labs were put through realistic shopping scenarios, 57.2% of the test runs passed and 42.8%
Interactive Visual Gym: Evaluating Visual Interaction Models Through Simulation
Salesforce — Learn how Salesforce AI Research built Interactive Visual Gym, an open benchmark using dynamic simulation to evaluate real-time visual AI assistants.
In the last 14 days, nearly 30,000 tasks ran for over an hour on Agent Arena.
As tasks get more complex, @GoogleDeepMind and @XiaomiMiMo are the biggest winners in the Agent Arena.
Agent Arena measures complex, real-world tasks.
Why human insight is a strategic advantage for building better agents
langchain.com — Explore 3 principles for effective human-in-the-loop workflows and how human-review is a strategic imperative for production-ready agents.
No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark
The Decoder — SemiAnalysis is putting AI agents through simulated knowledge work in its new GDPx benchmark, covering everything from chip design to quant trading. No model breaks 12 percent. Claude Opus 5.5 leads narrowly over Gemini 4 Carbon. Critics point to the small number of tasks per domain and uneven testing conditions.
VB Pulse Agentic Reliability and Evals Report - September 2026
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
The Decoder — CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects.
Exciting news from Agent Arena: Hy4 by @TencentHunyuan just landed #14 overall.
Introducing Physical-Engineering’s Last Exam for agents
Google Cloud Blog — Discover PLE, a new benchmark evaluating frontier AI agents on real-world mechanical, electrical, civil, and structural engineering tasks.
The top 10 agents in Agent Arena process about 93 input tokens for every token they generate, according to our analysis…
Dismiss email signup
Subscribe to 
The top AI news in your email delivered each morning. Choose categories