Anthropic Gives METR Inside Access: What Independent AI Evaluation Can Establish

Anthropic says six METR researchers now have employee-like access to help assess its AI risks, with publication rights subject to defined redactions. The October 15 announcement expands external scrutiny, while METR’s own guidance warns that voluntary assessments cannot certify that frontier AI development is safe.

Anthropic has announced a deeper relationship with the nonprofit evaluator METR, giving a small research team access to internal systems and the ability to publish assessments of its risk levels, incidents and practices. The October 15 announcement says six METR employees have company hardware, office space and tools broadly comparable to those used by Anthropic’s risk-assessment teams.

The company says it cannot veto METR’s publication, although it can require redactions within predefined categories, including customer information, privileged material and model-architecture details. METR can publicly describe and question those redactions without revealing the protected information. Other evaluators have different access and publication arrangements, so the same rights should not be assumed across partnerships.

The important development is an arrangement for gathering and reporting evidence. It is not a published finding that a particular new model has passed every safety test. Understanding what outside evaluators can establish requires looking at their methods, earlier work, funding and stated limitations.

METR describes evidence gathering rather than a safety guarantee

In its October 14 account of working with AI companies, METR explains that its goal is to provide accurate, timely and independent evidence about the risk of humans losing control of AI agents. It intends policymakers, developers and the public to make their own decisions using that evidence.

Its transparency principles cover the scope and duration of an assessment, access limitations, redactions and constraints that could materially affect conclusions. It also acknowledges the difficulties of a voluntary system: developers control whether to engage and what information to provide, and some evidence may be impossible for an outside group to verify independently.

For readers, that changes how a report should be read. The name of an external organisation is useful information, but the methods and limitations explain what its participation means. A narrow investigation can produce valuable findings while leaving other questions unanswered. Those gaps should remain visible alongside the conclusions, rather than disappearing into a general claim that an AI system has been independently checked.

A current proposal separates three different risk questions

METR’s October 14 proposal on public evidence separates alignment, control and acceleration in AI research and development. In plain terms, these concern whether agents might pursue harmful actions, whether safeguards would stop them, and whether development could accelerate beyond the ability to understand and manage it.

The proposal explicitly says such exercises cannot give confident assurance of safe development. It also distinguishes them from audits, compliance assessments and formal oversight. Its intended contribution is evidence about specific technical risks, not an authority to approve or reject every deployment decision.

This separation is useful because strong evidence about one question does not settle the others. An agent might rarely attempt a harmful action in a test yet encounter different circumstances later. A monitoring system might identify suspicious behaviour while the response process remains too slow. Faster capability growth can also alter the circumstances in which earlier evidence was collected. Each claim needs its own supporting observations.

Alignment assessments look beyond successful demonstrations

The detailed alignment proposal describes reviewing serious misaligned behaviour across training, evaluations and internal use of models near a developer’s capability frontier. It also asks whether the observed cases represent what agents might do in plausible future situations.

Potential evidence includes internal transcripts, access to model checkpoints and information about how models were trained. The proposed analysis would examine whether testing or training could encourage agents to hide unwanted behaviour, as well as whether the available search methods might miss important incidents. These are questions for investigation, rather than findings that every model behaves in those ways.

A useful analogy is the difference between watching a product demonstration and examining a record of difficult operating conditions. The demonstration can show capability; the broader record helps reveal how the system responds when instructions, incentives or circumstances become less straightforward. Neither an isolated success nor an isolated failure should be treated as a complete statistical description of the system.

Control includes the human response to an alert

METR’s control and monitoring proposal asks about coverage, vulnerabilities in monitoring infrastructure, evasion, human review and the possible impact of undetected actions. It describes comparing inference records with monitoring records and investigating whether relevant traffic is actually examined.

It also considers what happens after an automated system raises a concern. A warning needs to reach someone who can assess it and respond appropriately. Testing the detector alone would leave the escalation process unexamined. The proposal therefore includes interviews, incident records and realistic tests of the review pipeline.

For an organisation using AI agents, this provides a concrete way to think about oversight. Logging an action, noticing a problem and stopping an action are different functions. A report should identify which function was tested and under which conditions. The existence of a monitoring tool does not, by itself, describe how much activity it covers or how effectively its warnings are handled.

Measuring faster AI development requires more than counting code

The research-acceleration proposal describes studying internal capability trends, the contribution of AI automation, other inputs and potential bottlenecks. It calls for multiple forms of evidence, including benchmarks, intermediate measures and interviews with researchers.

The attribution question matters. If capabilities improve more quickly, an assessor needs to distinguish the effects of AI assistance from changes in computing resources, staffing, data or other factors. The proposal treats forward-looking estimates as uncertain and asks whether the methods would reliably notice a substantially larger acceleration.

That is more informative than treating a large volume of generated code as a complete measure of research progress. Code can support experiments, but experiments still need to produce useful information, and research outputs need to be evaluated. A careful assessment connects inputs with outcomes over time. It should also explain where that connection remains ambiguous, particularly when several changes occur together inside a laboratory.

Earlier testing shows what inside access can produce

In a March account of testing Anthropic’s monitoring systems, METR reported that one researcher spent three weeks examining a subset of internal monitoring and security systems. Anthropic provided access and made staff available for questions. METR said the exercise discovered specific new vulnerabilities, some of which had subsequently been patched.

The account describes a 26-page report shared with Anthropic and a redacted version shared with a subset of METR staff. It did not say the findings severely undermined the major claims in the relevant Opus 4.6 sabotage risk report. This is historical evidence about that exercise, rather than a new October test result.

The example illustrates why the extent of access matters. An external researcher can ask operational questions that a public interface alone cannot answer. It also illustrates why the resulting conclusion must retain its boundaries: three weeks of work on a subset of systems supplies evidence about that scope, not every possible pathway through a company’s infrastructure.

Independent review can challenge reasoning as well as measurements

METR’s March review of Anthropic’s Opus 4.6 sabotage report describes examining two versions of the document and receiving an unredacted report with other supporting material. Its summary distinguishes the adequacy of information from the strength of the analysis.

The reviewers said the evidence was generally represented clearly while identifying places where more analysis could improve the report. They also raised concerns about the strength of reasoning in multiple areas. The review’s version history matters because a judgment about an earlier draft should not silently become a judgment about every subsequent document.

This is a useful distinction for readers following the October arrangement. Outside scrutiny need not produce a single pass or fail. It can identify an unsupported inference, a missing comparison or a limitation in the argument connecting a test to a broader safety claim. Those contributions can improve the public discussion even when the underlying measurements are not themselves disputed.

Confidential access and public accountability have to coexist

METR’s February confidentiality explanation describes six information levels, ranging from public material to information restricted to particular internal teams. At the most sensitive end, access and discussion of nonpublic models are limited to directly involved researchers.

The organisation says these measures address leaks through conversation and infrastructure, while also limiting who knows sensitive information. This is its description of its practices; it does not establish that confidential work carries no risk. It explains why giving a team access does not necessarily mean that every employee of the evaluator sees the same material.

For the public, a useful report must therefore provide enough detail to assess its reasoning while protecting information that cannot responsibly be released. A clear description of what was withheld can help readers understand the remaining uncertainty. Confidentiality and useful disclosure are not automatically incompatible, but the arrangement needs to make their practical boundary visible.

Funding arrangements also deserve attention

In its August funding update, METR reported about $71 million in commitments over the preceding six months. It said it had not accepted funding from frontier AI companies or donations made by or at the direction of their staff, while acknowledging significant free model-token access from those companies.

Separately, Anthropic’s September 18 Accenture announcement describes a directly funded partnership led by Faculty, Accenture’s specialist AI business. It lists model evaluation, adversarial testing, alignment assessment and safeguard testing among the intended activities. These are different relationships and should be evaluated on their own terms.

The next significant evidence will be the assessments themselves: what was examined, what evaluators could access, which findings are supported and which questions remain open. The October announcement makes that scrutiny more concrete. Its value will be demonstrated through the quality and timeliness of the resulting public record.

Explore more Smashology Tech reporting and our guide to interpreting Muse Spark Open benchmarks for another example of why evidence and test conditions matter.