AI Security

UK to scale AI flaw hunting and plan for rogue AI agent incidents

AI minister Kanishka Narayan expands frontier AI flaw hunting across government and plans for agent incidents and CSRB near-miss reporting.

Illustration of an AI chip motif with the headline UK readies for AI agent incidents

The UK government will scale up its use of frontier AI to hunt for flaws in its own systems, and is drawing up plans for responding to incidents caused by autonomous AI agents. AI minister Kanishka Narayan also said ministers will look at whether the Cyber Security and Resilience Bill should require reporting of serious AI incidents and near misses.

What happened

On Monday 12 October, Kanishka Narayan gave his first speech since the AI brief was raised to Cabinet level. Speaking at Imperial College London, he said it now seems “likely, if not inevitable” that UK cyber defences will be breached as AI systems grow more capable. A package of measures published the same day came from the Cabinet Office, DCMS, the NCSC and the AI Security Institute (AISI).

The package has three main parts:

  • Frontier AI to defend government. Narayan, the DCMS cyber minister and Security Minister Dan Jarvis will lead an urgent push to use frontier AI to find weaknesses in government systems before attackers do. It builds on two pilots that, according to the government, helped identify and address more than 1,000 potential vulnerabilities.
  • AI incident response. The NCSC, AISI and DCMS will work out how government should respond when AI agents cause harm. The focus is on containing an incident and supporting those affected.
  • AI incidents in the CSRB. Ministers will work with Parliament and technical experts on whether to strengthen the Cyber Security and Resilience Bill (CSRB) and the wider regime. The options cover reporting serious AI incidents and near misses, and protecting essential services from loss-of-control risk.

The minister also confirmed that AISI is evaluating the latest frontier models, including Gemini 4 Argon. This follows the recent restoration of its pre-release access, which secnews has already covered. A companion AISI blog post lists agentic incident detection and reporting among its priorities for industry best practice.

The detail

The most concrete evidence behind the vulnerability-hunting claim is a GOV.UK case study from June. The Government Cyber Coordination Centre (GC3), a joint NCSC and DSIT unit, ran weekly in-person hackathons with AISI and NCSC specialists. Teams pointed frontier models at government code held in public repositories. Over one month, across nine government organisations, they produced 407 findings, some of them critical, ranging from remote code execution to authentication bypass and leaked data. The case study puts the token cost at £13,000 and says every critical issue was fixed with no sign of exploitation.

One team chained six agent stages, each checking the last, with a human approving every escalation. Another fed Semgrep, Trivy and Gitleaks output into model stages that combined findings into attack paths. One critical finding was a legacy GitHub Actions workflow in which a crafted pull request comment could run code on the runner, which could have exposed workflow secrets and tokens.

Narayan also claimed that the central vulnerability-monitoring service has brought remediation times down from 50 days to eight. GC3 has said a second phase will bring in more departments and models and extend the work from public code to closed-source estates. The figures should be read with care: the published case study accounts for 407 findings, and no equivalent detail on the second pilot has been released.

On incident response, the government argues that a major incident involving AI agents could spread faster and further than the incidents its existing crisis machinery was built for. Narayan listed four questions responders will need answered fast: the agent’s instructions, the data and systems it touched, the reasons behind its actions, and whether the fault could repeat elsewhere. He used the Australian case from earlier this year as his example: an OpenAI agent on a benign research task found its own way past access restrictions on a government website.

Key figures from UK government frontier AI vulnerability pilots: over 1,000 vulnerabilities, 407 published findings, £13,000 token cost, fix time cut from 50 to 8 days

Why it matters for UK organisations

The CSRB point matters most to essential service operators and MSPs. The Bill already sets incident notification duties, and peers have separately tabled AI-related amendments, which secnews reported on 11 October. What is new is the government saying it will itself explore AI incident and near-miss reporting. A near-miss duty would be unusual in UK cyber regulation. Spotting an agent misbehaving before it does damage needs monitoring that most AI deployments lack.

No detail has been published. The press release says measures will be worked through with parliamentarians, technical experts and the operators expected to deliver them. Regulated organisations can still shape definitions and thresholds, but have no timetable. Separately, the speech said the National Cyber Action Plan would launch soon, without giving a date.

The pilots matter beyond Whitehall: £13,000 for hundreds of findings is a benchmark boards will notice, and attackers have the same tools.

Expert view

For practitioners, the GC3 write-up is the most useful part of this announcement. It concludes that pipeline design matters more than which model is used, and that triage, not discovery, is the bottleneck. In my experience, the hard part is rarely finding the issue; it is getting a stretched team to fix it. Faster discovery without more remediation capacity just lengthens the queue.

The incident response work is overdue. Most organisations I see could not answer Narayan’s four questions about one of their own agents today, because tool calls, prompts and credential use are not logged anywhere a responder could reach. That is a basic observability gap, not a future superintelligence problem.

I am more cautious about the CSRB plans. “Will explore” commits the government to nothing, and a vague near-miss reporting duty could produce noise rather than lessons. The government should define clearly what counts as an AI incident before asking operators to report one.

What to do now

  1. Inventory your agents. List every AI agent and assistant with access to systems or data, including features embedded in SaaS, and record what each can reach. This extends Cyber Essentials secure configuration and user access control to non-human users.
  2. Log what agents do. Capture prompts, tool calls, credentials used and outputs in tamper-resistant logs your incident responders can query. Without this, you cannot answer what an agent was asked to do and what it touched.
  3. Add agents to incident playbooks. Define how you would revoke an agent’s credentials, stop running tasks and contain its access, and rehearse it in a tabletop exercise.
  4. Start recording near misses. Track cases where an agent tried something outside its remit, even if it was blocked. If CSRB reporting arrives, you will already have the process.
  5. Trial AI code review carefully. If you copy the GC3 approach, keep the work scoped, keep a human approving anything escalated, and agree remediation capacity before you start. Patch management under Cyber Essentials still governs what happens next.
  6. Engage on the CSRB. Operators of essential services and MSPs should brief their regulator and trade bodies now, while the AI provisions are still being shaped.

Bottom line

This speech moves the government’s AI security policy from warnings to work in progress: more AI-driven vulnerability hunting across government, a plan for agentic AI incidents and possible statutory reporting. The detail is thin, but the direction is clear. Organisations that can already see and stop what their AI agents do will be ready whichever way the Bill goes.

Sources

Related Posts