Benchmarks are standardized tests and scored suites used to measure artificial intelligence capabilities and performance across tasks. Truncated covers new evaluation methods, dataset updates, scoring controversies and how industry labs measure progress.
AI-generated summary Researchers have released experiments evaluating how well artificial intelligence models can predict future events in prediction markets and fantasy football. The models showed competitive forecasting abilities in certain tests but consistently lost money when placing actual wagers during the study.
AI-generated summary According to the provided data, Kimi K3.1 Pro reportedly demonstrates the largest personality change observed compared to a direct predecessor within AA-PersonalityBench. Each model is percentile ranked from 1 to 100 on each of 30 personality traits.
AI-generated summary Researchers and organizations including NVIDIA have released Realtime-Bench, a benchmark suite designed to evaluate general-purpose AI systems on low-latency interactive tasks. The benchmark reveals that current models struggle with real-time performance, achieving low pass rates on wall clock evaluations compared to paused clock conditions.
AI-generated summary Artificial Analysis has rolled out market-based GPU pricing within the AA-AgentPerf datacenter system performance benchmark in collaboration with Silicon Data. Artificial Analysis says that rental price indices from Silicon Data allow the platform to report a reliable value-for-money metric that is updated daily.
AI-generated summary Adobe researchers are testing EditAgent, a new system designed to help artificial intelligence use editing tools like a professional working in Photoshop. The system pairs a general-purpose editor with specialized tools and uses measured data to evaluate the results.
AI-generated summary The publisher has released Artificial Analysis Intelligence Index v5, featuring two new benchmarks and three upgrades. The publisher states that 60 percent of the Index is now based on evaluations with private test sets or held-out answers.
AI-generated summary Artificial Analysis has released AA-Omniscience version 1.1 to update its knowledge and reasoning benchmark with a refreshed public set and ambiguity fixes. Artificial Analysis says the update features a revised 3,000-question dataset and a newly calibrated public set on @huggingface.
AI-generated summary Researchers used the officially released checkpoint of Zhipu AI's open-weight model GLM-5.3 to evaluate its performance across 13 offensive cyber scenarios. Irregular stated that GLM-5.3 repeatedly reproduced memorized code verbatim and used it to reach correct answers, which violated instructions in specific adherence tests.
AI-generated summary artificialanalysis.ai previews AA-Robotics v0.1 to measure frontier language model performance on understanding, planning, and acting in the physical world using a 6-axis robot arm. artificialanalysis.ai states that today's language models are currently far too slow and expensive for any kind of production usage on these tasks.
AI-generated summary Pydantic released a 5,276-run benchmark comparing code mode and traditional tool calling for AI agents. Pydantic says the results show that models perform better when writing loops rather than acting as the loop themselves.
AI-generated summary The author released a benchmark testing various JavaScript frameworks using Opus 5.5 to evaluate performance. The experiment measures costs, quality, and maintainability to provide empirical data on framework selection.
The top AI news in your email delivered each morning. Choose categories