METR

Today, METR is sharing its vision for its version of third-party “embedded assessments” for AI loss of control. The goal of these exercises is to make sure the public receives good evidence about whether or not the AI industry in general is on track to retain control of its AI agents as they become better at improving themselves.

If an AI developer was on track to lose control of its internal agents, would anyone outside the company know? Today, we are sharing METR’s vision for how we would work with AI developers to openly assess and share evidence about this. We describe three critical areas where AI developers should provide better open evidence: (1) alignment, (2) control, and (3) AI capability acceleration. In each area, we detail the specific questions an independent assessor could seek to answer and how. (1) Alignment - The public should know if agents would attempt actions like those seen in incidents like the hack of Hugging Face. Our proposal is to analyze a developer’s internal incidents and whether training may suppress evidence of worse misalignment. (2) Control - Agents that attempt harmful or unwanted actions are not always noticed. Our proposal is to analyze whether a developer’s internal agents could get away with serious harm without being caught. (3) AI R&D acceleration - If AI automates AI R&D, capability growth could outpace what alignment and control methods can handle. Our proposal is to analyze how much AI automation is accelerating capability growth internally. This proposal is our recommendation for how we could give the public visibility into risks occurring within AI development. We stand ready to conduct all three of our proposed exercises with frontier AI developers. Our proposed exercises would provide evidence, not confident safety assurance or formal oversight. We also encourage developers to work with other external groups to complement these efforts, and to unilaterally share evidence with the public. 1. The main proposal: https://lnkd.in/gncy4Krq 2. Detail on our alignment assessment proposal: https://lnkd.in/gwU4PN43 3. Detail on our control assessment proposal: https://lnkd.in/gDdcseTb 4. Detail on our AI R&D acceleration assessment proposal: https://lnkd.in/gtcrdTam 5. A companion piece, in which we explain our approach to working with AI developers in exercises like these. Our goal is to get the public accurate, timely, and independent evidence so that they can make their own informed decisions: https://lnkd.in/g2j4RkSi

METR is working hard to ensure that the world can track the risks and capabilities of advanced AI systems. As the stakes of this work rise, we're also scaling our operations team with exceptional people who will guide our next phase of growth and beyond. I'm extremely excited to start with these two roles: - Head of People: https://lnkd.in/gXN5yNVe - System Administrator: https://lnkd.in/gF2SsmPP If our mission resonates with you or you know someone who would be a great fit, apply or reach out; we really value getting trusted leads. All roles: https://metr.org/careers

In order to notice when AI agents misbehave, AI companies often log the actions and reasoning steps their agents take. However, misaligned AI agents may be able to hack the software that humans use to review and understand these logs, hiding misbehavior. Here’s a simple example: a METR researcher found a bug in the transcript viewer of Inspect, a popular evaluation framework, that would have enabled an agent to show the user reviewing its transcript a fake (or edited) version. This demonstration used a JavaScript injection via MathJax rendering, which an agent could trigger from anywhere in its output, including its reasoning. For clarity, we haven't seen agents exploit this in our evaluations, and Inspect was patched within a day of our report. This bug did not change the underlying logs, just what the viewer showed. More capable agents could target multiple oversight systems or deeply compromise their company’s infrastructure, creating a sort of “Potemkin village” to deceive the company’s employees. Treating agent observability as security-critical infrastructure means handling all AI outputs (transcripts, reasoning, actions, etc.) as untrusted inputs: ideally we can make it very difficult for agents to influence the systems we use to supervise them. https://lnkd.in/gRMHYgPb

It was an honor to testify in the U.S. Senate today before Chairman Hawley, Ranking Member Kim, and the subcommittee on METR's work, recent AI agent incidents, and the importance of transparency in frontier AI development. My written remarks are available here: https://lnkd.in/g9pTrzU3

We run lots of evals at METR. Sometimes, agents attempt harmful actions. To help make these evals safer, I built a monitor that blocks suspicious tool calls until a human reviews them. Writing out an argument for why this monitor is effective, and checking what evidence we actually have and don't have for each claim, helped to surface hidden assumptions we'd made. One assumption was that blocking a tool call would be enough to prevent it from running until a human reviews it. We found that this wouldn't always hold. For example, if someone launches an eval using a coding agent, that agent might "approve" actions blocked by the monitor without being asked to do so. Doing this exercise helped us improve the system a lot, and I'd recommend it to anyone else building monitors! https://lnkd.in/g4RSsBwS

I'm joining METR to work on more investigations like our Hugging Face report. Currently, tons of even basic information about AI development that's highly relevant to catastrophic risk isn't public. I used to be more skeptical of the value of public info, but recent events have changed my mind. Getting verified information about what's going on inside AI companies seems particularly urgent now. The limited public evidence we have seems consistent with the possibility that imminent recursive self-improvement could massively accelerate capabilities progress, which could then potentially yield extremely superhuman general capabilities within 6 months or a year. If this occurred, there would be a correspondingly large risk of worst-case outcomes. This uncertainty about extreme outcomes could be substantially resolved with more verified public information: we could either build more consensus about near-term risk or learn that such extreme outcomes are less likely in the near term. Beyond AI capabilities and takeoff, the state of public evidence is also highly limited for alignment, security, control, and risk-relevant internal processes at AI companies. This makes it hard to determine exactly how well or poorly these key areas will go in the near future. (METR plans to focus, at least initially, on just capabilities/takeoff, alignment, and control; I hope other groups cover security, internal processes, and other important areas.) While I'm no longer working at Redwood, I think the work they are doing is very important; I'm excited about Redwood's ongoing contributions to R&D on technical mitigations and better public interpretation of risk-relevant evidence.

METR has been in the news a lot lately, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.