Anthropic and OpenAI both shipped on Tuesday, September 22, and Artificial Analysis published its own measurements of both within hours. Two days later OpenAI switched an entire modality off its API, and on Friday a new industry alliance released the first version of a benchmark for defensive security work.

Read the announcements and you get four capability stories. Read them next to the measurements and you get one story, and it is not about capability.

The one thing that matters

A benchmark score measures a configuration. The model is only one part of it.

Anthropic’s announcement puts Claude Opus 5.5 at 66.4% on Terminal-Bench 4.0. Artificial Analysis, testing the same model on the same benchmark in the same 24 hours, puts it at 59.6%.

Neither party is careless and neither hides the reason, which sits in Anthropic’s own footnote. What separates the two numbers is which harness ran the task and how much effort the model was told to spend. Neither is a property of the model.

This Digest argued in Issue 004 that the party doing the work should not also write the account of it. That still holds, and this week is the harder case. Both accounts are honest and both disclose their settings. They still disagree by almost seven points, and the ranking flips.

Stories worth your time

Two labs shipped, and the independent numbers split

Anthropic released Claude Opus 5.5 on September 22 at $4 and $20 per million input and output tokens, down 20% from Opus 5’s $5 and $25, with cache reads cut 60% to $0.20. It claims roughly 40% lower cost on typical workloads, output more than 30% faster, and a model that attempts to circumvent its boundaries about 85% less often than Opus 5.

The benchmark table leads with Terminal-Bench 4.0: Opus 5.5 at 66.4%, Fable 5.1 at 55.8%, Opus 5 at 52.3%, GPT-6 Astra at 57.9%. The footnotes are careful. Standard error is given as plus or minus 2.6 points. Anthropic notes that the public leaderboard, at 5 trials per task on the Claude Code harness, reports Opus 5 at 51.8% while its own setup reproduces 52.3%, within noise. And it says plainly that its Opus 5.5 figure is at xhigh effort while the GPT-6 Astra figure is at high effort, as reported by OpenAI.

That last sentence is the story. Artificial Analysis ran the same benchmark on Stirrup, its open-source agent harness, and got 59.6%, which it calls level with GPT-6 Astra at xhigh rather than ahead of it. An 8.5-point lead in the vendor’s table becomes a tie once both models are measured at the same effort by the same third party.

The rest of the independent read favors Anthropic. Opus 5.5 scores 58 on the Artificial Analysis Intelligence Index, the highest the firm says it has measured by several points, and gains 143 Elo over Fable 5.1 on its private AA-Briefcase knowledge-work evaluation. It trails on CritPt, AA-LCR, and GDP.pdf, and it spends roughly 119,000 output tokens per index task against about 73,000 for Opus 5, which is why a 20% price cut lands at level cost per task rather than cheaper.

OpenAI’s release the same day went the other way. GPT-6 Sol and GPT-6 Luna arrived with what the company describes as 50% lower API prices than GPT-5.6 promotional pricing, and the pricing page bears that out exactly: Sol at $2 and $10 against GPT-5.6 Sol’s $4 and $20, Luna at $0.10 and $0.50 against $0.20 and $1.20. GPT-6 Astra sits at $10 and $50.

Artificial Analysis measured the pair and found that Intelligence Index and Coding Agent Index scores remain level with GPT-5.6. Sol gains two points on the Coding Agent Index to reach 57 and Luna loses two to land at 41. Both regress on GDPval-AA v2.1 at max effort, attributed to shorter deliverables that more often omit required elements, and Luna drops about 45 Elo on AA-Briefcase v1.1. What moved is the bill: Sol costs $1.06 per task at max effort against $1.99 for GPT-5.6 Sol, and Luna $0.07 against $0.18.

So one lab shipped a real capability gain with a headline number inflated by an effort mismatch, and the other shipped a real price cut with flat capability and a regression in knowledge work. Both are useful. Neither is what the announcement led with.

This continues Issue 002, which reported that the strongest version of each frontier model had moved behind an eligibility check, so what you can buy depends on who you are. The new part is that approval is not the end of the question. Once you are in, the model you get is still underdetermined, because the number that convinced you to pick it came from a harness and an effort level you are probably not running.

Builder implication: comparing two vendor tables means comparing two configurations. Pin the harness and the effort setting before you read the percentages, and treat any number reported at a different effort than its comparison as no comparison at all.

A new cyber index found the frontier refuses the work

On September 25, Artificial Analysis launched the Cyber Index Alliance with Collinear AI, IBM, NVIDIA, and Vercel, and published version 1 of a composite benchmark for enterprise cyber defense. It is built from three evaluations. CWE-Bench-AA, from Collinear AI, gives an agent 120 held-out tasks across all ten OWASP Top 10 (2025) categories in C and C++, Go, Java, JavaScript and TypeScript, Python, and Rust, and asks it to audit a real open-source repository and patch what it finds without breaking working behavior, with programmatic verifiers deciding the outcome. DeepsecBench-AA, from Vercel, isolates discovery and scores findings against expert-verified ground truth on an F2 score that weights recall over precision. CyberGym-E2E-AA, from Berkeley RDI, takes 131 filtered tasks on memory-safety bugs in C and C++ and requires the agent to locate the bug, produce a crashing proof of concept, and write a patch that still passes existing tests. All three run on Stirrup, in sandboxes with no internet access, and turning a found vulnerability into a working exploit is out of scope.

The headline finding is not a score. Because the work is dual-use, the index tracks tasks a model or provider declines on safety grounds and reports them separately. On the end-to-end benchmark that accounting swallows the result: “Refusals are a significant factor on CyberGym-E2E-AA: GPT-6 Astra, GPT-6 Sol, Claude Fable 5.1, Claude Opus 5.5, Qwen3.8 2.4T A95B, and Qwen3.8 27B refuse at least 98% of tasks, making it difficult to assess frontier model performance.”

Read that with the setup in mind. The task is defensive. The agent has the source. It runs in a sandbox with no network. It is asked to find a memory-safety bug, reproduce it, and fix it, which is the daily work of everyone maintaining C. Six models across three vendors decline nearly all of it.

Where models do engage, the numbers are sobering rather than alarming. On discovery, the best model identifies only 41% of the expert-verified issues, and what it finds is legible: flaws with a direct path from untrusted input to consequence. On the end-to-end benchmark, 31% of passing attempts patch a real crash other than the target, and those off-target passes skew shallow, with null-pointer crashes making up 22% of them against 9% of on-target passes. A pass rate here is partly a measure of how often the agent fixed something true and irrelevant.

Builder implication: refusal rate is a capability number, and it is the one number no vendor benchmark publishes. If any part of your product touches security, abuse, malware, or vulnerability work, measure declines as their own metric before you measure quality, because a 98% refusal rate is not something you prompt your way out of.

OpenAI removed the Videos API and named no successor

On September 24, per OpenAI’s own deprecations page, the Videos API and every Sora 2 model left the platform: sora-2, sora-2-pro, and the dated sora-2-2025-10-06, sora-2-2025-12-08, and sora-2-pro-2025-10-06. The notice went out on March 24, 2026, so developers had six months. The Sora web and app experiences had already closed on April 26.

The replacement column is empty. Not deferred, not “use this other endpoint.” A frontier lab removed a modality from its developer platform and named no migration target, and its deprecation page is the document of record.

Video generation is still available elsewhere, so this is not a capability regression at the frontier. It is the first case in this Digest’s coverage of a whole class of generation being withdrawn from a major platform with no path forward inside it. Six months of notice is generous by the standards of most API deprecations. It is also six months, which is shorter than the life of most products built on top of one.

Builder implication: read the deprecation page before you take a dependency, not after. The useful question about a vendor capability is no longer only what it costs and how good it is, but how long the vendor has committed to keeping it.

950 agents found a new enzyme system, and the laboratory has the verdict

On September 23, Anthropic published an account of Claude identifying a previously uncharacterized enzyme system it calls array-associated reverse transcriptases, or ART. The configuration is published in full: 950 agents, 210 million tokens, 21 hours. The agents gathered over 200,000 reverse transcriptases, picked out 3,500 new candidate systems, and narrowed those to the 20 most compelling. The system pairs a reverse transcriptase with a partner gene and a long array of evenly spaced DNA repeats, and it turns up mainly in bacteriophages. A group formed in spring 2026 runs a BSL-1 and BSL-2 laboratory in the Bay Area, where the protein is expressed in standard laboratory strains and characterized biochemically and structurally.

The most interesting sentence in the post admits what is not known. Anthropic does not yet know what ART does, the accessory protein is uncharacterized, and the finding is out as a preprint with the main functional question open.

That shape is the point. Issue 003 covered an 11-day formalization run and argued that an agent’s output is worth what something outside it can reject. There the rejecter was a proof assistant: total, mechanical, instant. Here it is a wet laboratory: partial, slow, expensive, and the run was designed around that constraint rather than against it. The agents were not asked to decide anything. They were asked to reduce 200,000 candidates to 20 worth the bench time, a job where a false positive costs an experiment and a false negative costs an opportunity nobody will ever notice.

Builder implication: when your verifier is slow and costly, stop trying to make the agent right and make it a good filter instead. Publish the funnel with the result, because the ratio from 200,000 to 3,500 to 20 tells a reader more about whether to trust the output than any claim about the model does.

The pattern

Four events, and in each one the variable that moved sits outside the model.

In the benchmark split it was the harness and the effort setting. In the cyber index it was a refusal policy, which removed a task category more completely than any capability limit would have. In the Sora removal it was a deprecation calendar. In the enzyme run it was a pipeline whose shape, 200,000 to 3,500 to 20 with a laboratory at the end, is what makes the result mean anything.

None of those appears on a model card, and three of them are not published at all unless someone builds the instrument to look. That is what the Cyber Index actually contributed this week. Its scores are early and narrow. Its decision to count refusals separately from failures is the durable part, because it turned an invisible constraint into a number.

This is where the argument from Issue 004 has to change shape. That issue was about provenance: move the record outside the thing being recorded. This week both records were written honestly with settings disclosed, and they still disagreed. So the fix is not only independence. A number without its configuration is not a measurement, and the only configuration that predicts your results is yours. The evaluation harness in your own repository, pinned to your own effort settings and task distribution, is not a redundant copy of the vendor’s work. It is the only instrument that reports all four of those variables at once.

What the evidence does not support is a claim that vendors are inflating results. Anthropic’s disclosure was adequate, and the inference required reading a footnote, which is a documentation problem rather than a credibility one. Artificial Analysis is a commercial benchmarking firm whose composite index includes private evaluations, so its numbers are not independently reproducible either. Two careful sources disagreed, which means the disagreement is structural.

Builder’s corner

The artifact to read this week is the Cyber Index announcement, and not for the leaderboard.

Read it as a model for how to report an agent evaluation. It names the harness and open-sources it. It gives task counts, 120 and 131, rather than a vague corpus. It says which languages and vulnerability categories are covered, what is out of scope, and what the index does not yet reach, which includes incident response, writing vulnerability-free code in the first place, and anything without source access. It uses programmatic verifiers instead of a judge model where it can. And it separates three outcomes most evaluation reports collapse into one: the model succeeded, the model failed, the model declined.

Then read the failure taxonomy, which is the part worth stealing outright. Instead of a pass rate it reports which kinds of bugs models find, that the best discovery score is 41% of expert-verified issues, that 31% of passes patch a real crash other than the target, and that those off-target passes are 22% null-pointer crashes against 9% for on-target ones. That is a report you can act on. A single composite score would have told you almost nothing and would have looked better.

Ideas worth stealing

Store the configuration next to the score

A score is a tuple, not a number: model version, harness, effort setting, trial count, prompt, scoring method. Anthropic’s footnote reproduced Opus 5 at 52.3% against a public leaderboard’s 51.8% and correctly called that within noise, which only means something because the harness and trial count were stated. Any result in your repository that does not carry its configuration will be uncomparable to its successor within a quarter.

Count refusals as their own outcome

Most harnesses have two buckets, pass and fail, and a refusal lands in fail. That hides the most consequential fact about an entire task category, as this week’s 98% figure shows. Three buckets cost almost nothing and turn a policy constraint into something you can see, trend, and route around.

Build the funnel, publish the funnel

When verification is expensive, the agent’s job is reduction, not judgment. State the ratio. Going from 200,000 to 3,500 to 20 with a laboratory at the end is a claim a reader can evaluate. “The model discovered a new enzyme system” is not.

Experiments worth running

Reproduce one vendor number on your own harness

Pick the single benchmark claim that most influenced a model choice you have already made. Run it in your own harness, at the effort setting your product actually uses, with the trial count you can afford, and write down the gap. You are not auditing the vendor. You are finding out how far your configuration sits from the one that sold you the model, and whether the ranking survives the move.

Add a refusal bucket to your evaluation

Separate declines from failures everywhere you score model output, and backfill the last month if you keep transcripts. If the rate is near zero you have spent an hour and gained a baseline. If it is not, you have found a ceiling no amount of prompt work will raise, and you would rather know before it is in a roadmap.

What I’m watching

  • Whether any vendor begins publishing refusal rates alongside capability numbers, and whether a provider-level decline is reported distinctly from a model-level one.
  • Whether the Cyber Index adds the categories it says are missing, especially compiled and live systems without source access, which is where most defensive work happens.
  • Whether effort settings become a standard part of benchmark reporting, or stay in footnotes where a cross-vendor comparison can quietly mix them.
  • Whether ART’s function comes back from the laboratory and the 20-candidate shortlist holds up. A funnel is only as good as its hit rate, and that number does not exist yet.
  • Whether other labs follow OpenAI in retiring a whole modality, and whether any offers a migration path rather than a date.

Take this with you

One thing to learn: a benchmark score is a measurement of a configuration, and this week the same model on the same benchmark read 66.4% from its vendor and 59.6% from a third party without anyone misreporting anything.

One thing to build: a refusal bucket in your evaluation harness, separate from failures, because six frontier models decline at least 98% of a defensive security benchmark and no vendor reports that.

One strategic question:

If a vendor’s number and your own disagree, which configuration is the one your product is actually running?

Sources

  1. Claude Opus 5.5, Anthropic, September 22, 2026, for pricing, the Terminal-Bench 4.0 table, the standard error and leaderboard reproduction footnotes, the xhigh and high effort disclosure, speed, and the safety claims.
  2. Claude Opus 5.5 takes the top spot on the Artificial Analysis Intelligence Index, Artificial Analysis, September 22, 2026, for the Intelligence Index score of 58, Terminal-Bench 4.0 at 59.6%, the AA-Briefcase Elo gain, Humanity’s Last Exam and SciCode results, token use, and the Stirrup harness.
  3. Announcing GPT-6 Sol and GPT-6 Luna in the API, Codex and ChatGPT, OpenAI, September 22, 2026, for the release and the 50% price reduction.
  4. OpenAI API pricing, OpenAI, for per-token prices across GPT-6 Astra, Sol, and Luna and the GPT-5.6 comparisons.
  5. GPT-6 Sol and Luna push the cost efficiency frontier, Artificial Analysis, September 22, 2026, for the level Intelligence Index and Coding Agent Index results, the GDPval-AA v2.1 and AA-Briefcase regressions, and cost per task.
  6. Announcing the Artificial Analysis Cyber Index Alliance, Artificial Analysis, September 25, 2026, for the alliance partners, the three component benchmarks with their task counts and scope, the Stirrup harness, the refusal accounting and the 98% figure, the 41% discovery result, and the 31% off-target patch result.
  7. OpenAI deprecations, OpenAI, for the March 24, 2026 notice, the September 24, 2026 removal of the Videos API and the Sora 2 model family, and the absence of a listed replacement.
  8. Claude discovers a novel enzyme system, Anthropic, September 23, 2026, for the 950 agents, 210 million tokens, 21 hours, the 200,000 to 3,500 to 20 funnel, the ART description, the laboratory characterization, and the open functional question.

Evidence note: Anthropic’s Terminal-Bench 4.0 figure and Artificial Analysis’s figure for the same model were produced under different harnesses and, for the cross-vendor comparison, different effort settings, both of which the publishers disclose. The gap is a configuration difference, not evidence that either party misreported. Artificial Analysis is a commercial benchmarking firm and its Intelligence Index includes private evaluations such as AA-Briefcase, so its numbers are not independently reproducible either. OpenAI’s own announcement page for GPT-6 Sol and Luna could not be retrieved directly during research, so the release claims here come from OpenAI’s developer-platform documentation and from OpenAI’s own announcement in its developer community; a factuality claim carried by third-party coverage was excluded because it could not be checked against a primary source. The Cyber Index is a version 1 release covering only vulnerability work with source-code access, excludes exploit realization, and reports that refusals make frontier performance on one of its three components difficult to assess, so its scores are not a general measure of security capability. The Sora and Videos API removal is documented on OpenAI’s deprecation page; live API behavior after September 24 was not independently tested. The ART result is Anthropic’s own account of its own model’s work, published as a preprint that has not been peer reviewed, and the system’s primary function is unresolved. The reading of a benchmark score as a property of its configuration, the argument that independence is insufficient without configuration, the funnel-over-judgment framing, and the proposed experiments are Gecko Road analysis.