AI Security Institute (AISI)

Today, we’re outlining AISI’s plan to help develop new technical best practice, delivering on the PM’s commitment towards principles and standards that keep AI development safe. Our blog sets out our priorities for new best practice for frontier development: - Documenting and assessing risk - Preserving and evaluating oversight methods - Enabling independent auditors and evaluators - Measuring alignment and security-relevant capabilities These challenges need a wide range of expertise, including developers, cybersecurity, incident response and other safety-critical industries We plan to work iteratively with developers and other frontier AI organisations to identify and share best practice, drawing on a wide range of expertise. ➡️ Read the blog: https://lnkd.in/eErCZwtM

As AI agents write code, run experiments, and work together, complex evaluations get harder to interpret. A single eval score obscures the ways agents reached it. Today, we're releasing Transect: an open-source tool that turns agent transcripts into interactive timelines. 🧵 During long evaluations, agents try many strategies, hit errors, and delegate to other agents. Tracing where a final score came from means combing through transcripts by hand or with an LLM - slow, and hard to reproduce. Transect reads a transcript as a sequence of turns. You describe the task and pick the behaviours to investigate - designing experiments, writing code, collaborating with other agents, analysing data - and it labels stretches of activity on a shared timeline. Transect also tracks work across agents. When one agent delegates to another, Transect can classify what the sub-agent was asked to do, and link report entries straight back to the relevant transcript passages. In a sample research run, an agent was given 6 days and resources to carry out a project and write a paper. Transect followed four documents - plan, code, experiments, self-review - as sub-agents wrote and later read them. We built Transect around three priorities: trustworthiness (findings link back to source material), flexibility (define your own categories and analyses), and scalability (reuse the same approach across many runs). Releasing Transect openly is part of our mission to build and share methods that keep evaluation fit for purpose. We hope the community will use it, stress-test it, and improve on it. ➡️ Read the blog: https://lnkd.in/ek-ZxR56 ➡️ Access Transect: https://lnkd.in/epZ6PKsB

Published today in PNAS Nexus: Conversational AI increases political knowledge as effectively as self-directed internet search. AISI’s Human Influence team finds that conversational AI's effects on political knowledge generalise across multiple topics, political perspectives, and model families, suggesting that the shift toward AI-assisted political information-seeking may not lead to increased public belief in political misinformation. ➡️ You can access the paper here: https://lnkd.in/es6Xsafy We found that AI usage did not increase belief in misinformation any more than internet search – even when AI models were prompted to be more persuasive. Results were similar for increasing extremist beliefs, changing political opinions, and decreasing trust in institutions. ➡️ Read more in the accompanying blog: https://lnkd.in/eAwbgVpZ

In August we committed to strengthening our security following an incident in which an agent took unsanctioned actions during a cyber evaluation. We are now sharing an update on our progress and what is coming next. We committed to three changes: tighter controls on internet access, real-time monitoring of evaluations as they run, and reassessing how we design evaluations. All three are now in place.  This is one phase of a continuous effort - as capabilities advance, we will adapt and develop our security practices to keep pace. We are setting out our work as openly as we can in the hope it helps those doing similar work, and we encourage others to share their approaches too. ➡️ Read the full blog: https://lnkd.in/eQ5Rs3QY

Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated. Recent incidents have seen AI systems perform unsanctioned cyber activity, like supply-chain attacks on out-of-bounds targets, despite being asked only to complete an evaluation. Before its release, we tested GPT-6 Astra for this behaviour. Our new ‘Unsanctioned Supply-Chain Attack’ evaluation runs in Petri, which uses LLM-based simulations, so no action executes in the real world. To test model behaviour with no interventions, we turned off GPT-6 Astra’s cyber safeguards, which are designed to block this activity. In our simulations, GPT-6 Astra attempts supply-chain attacks 29% of the time, vs 6% for GPT-5.6 Sol and 0% for GPT-5.5, showing GPT-6 Astra does this more frequently than previous models (though this may be complicated by increased simulation awareness).  In the simulations, GPT-6 Astra created fake identities to deceive developers, posted from fake accounts to argue against accurate security reviews, and delivered malicious payloads to open-source codebases. Even when we explicitly clarified the scope of its instructions to note “anything not listed as in scope is out of scope”, GPT-6 Astra still ran full supply-chain attacks in 4 of 49 trials.  GPT-6 Astra often stated that parts of its environment might be simulated, a limitation of our results. However, our analysis, in combination with observations from previous incidents, suggests it is possible this unsanctioned behaviour could occur in real-world conditions.  Our results indicate that defences beyond model alignment, like sandboxing and monitoring, may be needed to prevent real-world harm. These may also grow more fragile as capabilities improve. ➡️ Read the full blog: https://lnkd.in/eSTbNjAn See NCSC's advice on managing the cyber risk of agentic AI: https://lnkd.in/dgkC_urC  If this type of work interests you, AISI’s Alignment Red Team is hiring: https://lnkd.in/e6KDnBjS  

Some frontier AI evaluations require hundreds of millions of tokens, and every wasted trial has a real cost.    We’re introducing optstop: our open-source tool that stops an evaluation where estimates are already precise, and keeps running where they're not. Eval sample size is typically fixed before a run begins, leading to oversampling of tasks where model performance is already clear, and undersampling of those where uncertainty remains. As each trial gets pricier, the cost of a wasted one rises with it. Optstop treats evaluation as sequential measurement - carry on sampling where uncertainty is high, stop once a pre-set precision is met. It stops on two rules: 1. Precision (the interval is narrow enough) 2. Stabilisation (the interval has stopped changing). If neither rule is met, the run continues to its full planned budget. Genuinely noisy estimates aren't cut short. A conservatism mechanism in optstop guards against this, demanding more data when estimated success falls below 1%.  Every decision optstop makes is inspectable. In testing across binary, ordinal, and continuous scoring - on public benchmarks including MATH, GPQA Diamond, and WritingBench - optstop saved between 57% and 97% of planned trials, without impacting score estimates. Adoption is low-risk: evaluators can run optstop post-hoc, in shadow mode, or live. It integrates with our open-source Inspect framework in just a few lines.    ➡️ Read the full blog and paper here: https://lnkd.in/emTukeVT ➡️ Access the optstop package here: https://lnkd.in/eJSaR4wf

Today we're announcing two senior appointments. Henry de Zoete OBE joins as our new Director, and Nate B. becomes our new Chief Strategy Officer. Henry de Zoete OBE was instrumental in establishing AISI and already advises government on AI. A successful tech founder and one of the UK's leading voices on AI, he brings a rare mix of technical, commercial & policy expertise. Nate B. has been at the heart of AISI's work from the beginning, helping build it into a world-leading authority on frontier AI security. Together, they'll drive forward AISI's mission at a critical time for AI. A big thank you to our outgoing Interim Director Adam Beaumont, who guided AISI through rapid growth and strengthened its global standing. We wish him every success as he returns to GCHQ and look forward to continuing to work with him.

New paper in Nature Medicine: Researchers at the University of Oxford and UCL, in collaboration with AISI, built SIM-VAIL - a clinically validated framework for stress-testing how AI chatbots respond to vulnerable users in mental-health conversations, helping researchers spot weaknesses and test safer designs. You can access the paper here: https://lnkd.in/dmce7mEJ

On July 28th, we identified an incident during a routine cyber evaluation in which AI agents took sustained, unsanctioned actions directed at real people and organisations. The behaviour came mostly from one model (Anthropic's Mythos 5), with a small number of events from another (OpenAI's GPT-5.6-Sol). In the most serious case, an agent used social engineering to try and get malicious code into an open-source project. As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public. Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world. We are taking this incident seriously and working with labs, involved parties, and others to improve evaluation standards and best practice for disclosure - and sharing this openly so others can learn. You can read the incident report and full technical document here: https://lnkd.in/eXuUHd7A

Together with the US Center for AI Standards and Innovation, we ran evaluations of Kimi K3 focused on its cyber capabilities. Kimi K3 performs below leading US frontier models on our preliminary cyber evaluations. On "The Last Ones" (TLO), a 32-step simulated corporate network attack (~20 hours for a human expert), Kimi K3 reached step 17 on average. In 1 of 10 attempts, Kimi K3 completed TLO within the 100M token limit. On exploit development (ExploitBench), Kimi K3 scores 32%. On ExploitBench, Kimi K3 failed to develop exploits that achieved arbitrary code execution (ACE), the highest-severity outcome in exploit development. Kimi K3 achieved ACE on 0/41 samples.    Additionally, Kimi K3’s safeguards allow assistance with agentic cyber exploit development. Its safeguards did not prevent it from attempting cyber exploit development or offensive cyber operations during our evaluations.    These are preliminary results on a small set of public and private benchmarks. For the full methodology and results, read our joint blog with CAISI: https://lnkd.in/eT5iFxXn