Benchmarks are standardized tests and scored suites used to measure artificial intelligence capabilities and performance across tasks. Truncated covers new evaluation methods, dataset updates, scoring controversies and how industry labs measure progress.
www.unite.ai — Class imbalance occurs when one target class is much rarer than another, allowing aggregate metrics and standard training to neglect the cases that may matter most. This guide explains the mechanism, trade-offs, evaluati...
hgpu.org — Large Language Models (LLMs) have shown strong capabilities in code generation and reasoning, motivating extensive benchmarks for evaluating functional correctness. However, performance optimizatio…
The Decoder — Alibaba's new coding-agent benchmark, Co-Design Bench, tests AI agents on real user requests that already tripped them up in practice. Even GPT-6 Astra, the top performer, fully solves only 26.4 percent of tasks. On older benchmarks, the same models score above 99 percent.
The Decoder — Thinking Machines' AI model Inkling 2 Small swears during its internal reasoning, 316 times across 2,655 benchmark runs, according to an analysis by Vals AI. The profanity typically shows up late in long runs that fail anyway. But it doesn't work as a warning signal: 96 percent of failed runs contained no cursing at all.
The Decoder — In the Vals AI benchmark, OpenAI's GPT-6 Sol Daybreak hits 80.8 percent of attack objectives in simulated corporate networks. Worse, the open-weight model DeepSeek V4.1 Flash reaches 72.8 percent at a tenth of the cost. Because open models can no longer be restricted, analysts say the window for defenders is shrinking fast.
The Decoder — The "Into the Unknown" benchmark tasks nine AI agents with tackling open research questions from U.S. national laboratories. None of the 135 submissions meets all requirements. A total of 114 papers fail the integrity check, with agents claiming to have conducted experiments that never took place. GPT-6 Astra performs best but still doesn't meet the experts' standards.
South China Morning Post — Before deciding how to respond to developments in artificial intelligence, Beijing and Washington need a better sense of just how quickly the tide is rising.
patronus.ai — Three open model families cut filler words from their reasoning by a quarter to a half in 2026. Shorter wording did not mean fewer tokens.
Command Line — In our second post on ThinkingBox, we look at expanded model evaluations, the price of dependable runs, and reinforcement learning on terminal-state rewards.
huggingface.co — A Blog post by Ai2 on Hugging Face
Tech Times — AI backtest safeguards fail a new test. The FinDendro benchmark, posted to arXiv on Nov. 4, finds that telling GPT-5 to answer as of a past date cuts its knowledge claims from 39% to 1% while accuracy holds near 75%. The authors say vendor cutoffs, self-reports and date prompts cannot certify a
dwarkesh.com — AIs are now good enough to run a conspiracy
PYMNTS.com — Visa found that when 22 artificial intelligence models from 12 labs were put through realistic shopping scenarios, 57.2% of the test runs passed and 42.8%
The top AI news in your email delivered each morning. Choose categories