Benchmarks

AIAI-generated summary The Media Rating Council announced that the first phase of its draft standards for artificial intelligence in media measurement is planned for public comment in the first quarter of 2027. A second phase of the draft standards is scheduled to follow by mid-2027 according to the organization.

AIAI-generated summary South Korea ranked 10th out of 70 countries in a global artificial intelligence and digital competitiveness evaluation, according to the science ministry citing an international report. The country climbed eight positions compared to the prior year, securing third place in technology infrastructure and efficiency metrics as well as eighth place in the artificial intelligence adaptation index.

AIAI-generated summary NextLadder Ventures and Vals AI announced the Access to Justice Benchmark, reporting that none of 23 tested artificial intelligence models reached an 80 percent reliability threshold across legal-aid scenarios. Vals AI reports that the models satisfied up to 87 percent of expert criteria for general legal explanations, but only up to 50 percent for identifying specific local resources.

AIAI-generated summary The artificial intelligence model MiniMax H3-Image has been added to the Arena platform for evaluation. Users can access the new model in Battle Mode to test prompts and participate in voting while scores are pending.

AIAI-generated summary Artificial Analysis announced a partnership with GoogleDeepMind to bring MKA Benchmark to its platform. The collaboration makes the benchmark available through the Artificial Analysis platform.

AIAI-generated summary Artificial Analysis benchmarked both modes of Keenable Search 1.0 using the Stirrup agent harness to complete 1700 search-focused tasks. Keenable Search 1.0 achieved first and second place on the Artificial Analysis Search Index with scores of 83 for pro and 82 for realtime.

AIAI-generated summary Skillprint released findings from Signal, its human-AI evaluation platform, showing that humans outperform artificial intelligence sixfold in teamwork. Skillprint says the experiment tested models from OpenAI, Anthropic and Google in a cooperative platform game to examine collaboration capabilities.

AIAI-generated summary The Frontier Model Forum proposes a new Jailbreak Rating Framework to track how artificial intelligence models bypass safeguards. The proposed system grades artificial intelligence jailbreaks across capability gains, scope, and ease of deployment.

AIAI-generated summary Apple’s updated Mac Studio with the M5 Ultra SoC features an 80-core GPU and options for up to 512GB of unified memory for local artificial intelligence tasks. In testing against Nvidia DGX Spark and Dell Pro Precision GB10 systems, the Mac Studio was evaluated using models such as Qwen 3.8 27B, GLM-5.3 Flash, and DeepSeek V4.1 Flash 1017.

AIAI-generated summary TimeOmni-Live is a model evaluated using LiveTS-Suite, a benchmark designed for Continuous and Live Time Series Understanding. The benchmark contains 5,249 streams, 15,555 annotated response events and 2.2 billion timesteps organized into 18 tasks across six capability groups.

AIAI-generated summary InfoQ reports on reliability layers tested with language models on the CSIRO Adverse Drug Event Corpus to measure accuracy, yield, and verification limits. The evaluation found that the tested reliability layers sorted answers by trustworthiness but corrected almost none of them.

AIAI-generated summary Anthropic released Claude Fable 5.2 and Claude Mythos 5.2, models it says show improvements in coding, reasoning, and vision. On CyScenarioBench, Anthropic says Claude Fable 5.2 completed 76.3 percent of challenges compared to 61 percent for OpenAI's GPT-6 Cyber.

AIAI-generated summary Octomind reports that its client uses 25 percent fewer tokens than OpenAI's Codex CLI while solving the same coding tasks using the gpt-6.1-sol model over five paired rounds. Octomind states that benchmarking against Codex CLI helped uncover bugs caused by its tools asking for different message shapes than the model learned to use.

AIAI-generated summary @ArtificialAnlys announced an event in San Francisco on Tuesday, November 17, to discuss the quality of AI-generated images and video. The organization says the gathering will address the challenges of measuring subjective media generation models and feature a panel of industry leaders.

AIAI-generated summary Salesforce introduces GIFT-Eval-CTX as an open-source benchmark for contextual forecasting to evaluate how forecasting systems turn future context into numerical forecasts. Salesforce says incorporating structured future context yields significant error reductions across frontier LLMs, with tool-augmented agentic architectures achieving superior accuracy.

AIAI-generated summary Researchers at the University of the Pacific School of Pharmacy published a study in Scientific Reports comparing the accuracy of five large language models on 227 infectious disease examination questions. ChatGPT-4o-mini achieved the highest accuracy at 91.2 percent, while the evaluation showed that model performance declined on questions containing images.

AIAI-generated summary Anthropic has announced the public release of Claude Fable 5.2 on its blog along with benchmark details. Anthropic states that Claude Mythos 5.2 is available to all paid users, while eligible paid plans receive extra usage.

AIAI-generated summary Dosu says that building production-grade language model products is difficult because ensuring reliability is far from straightforward. Dosu says that LangSmith has allowed the company to scale evaluation driven development by enabling easy monitoring and searching of activity.

AIAI-generated summary Chad Dodd, Vice President of Product Management at athenahealth, discusses the evidence clinicians and practices need before trusting an AI-assisted tool in healthcare. Dodd explains why governance should extend across the product lifecycle, including testing for bias, accuracy, and data validity, as well as validation across specialties and datasets.

AIAI-generated summary In AA-PersonalityBench, artificial intelligence models that rank high on sincerity tend to rank low on slop according to the data. The evaluation also shows that models ranking high on creativity tend to also rank high on humor and eloquence.