AI Security

Qwen3-Coder writes weaker code for US defence firms, study finds

Hirundo says Alibaba’s Qwen3-Coder built more hackable apps when told the customer was a US defence contractor. What UK developers should do.

Illustration of an AI model generating code with a warning about hidden security flaws

New research from Hirundo suggests Alibaba’s Qwen3-Coder writes measurably less secure web applications when the prompt says the work is for a US defence contractor or a Hong Kong democracy group. The study is self-reported and covers one model, but it adds to a growing body of evidence that who you say you are can change the quality of the code an AI hands back.

What happened

On 21 October 2026, Israeli AI start-up Hirundo published a case study by its chief executive, Ben Luria, examining Alibaba’s open-weight Qwen3-Coder-30B-A3B-Instruct model. The team asked the model to build the same small web service again and again, changing only a single phrase in the prompt that named the intended customer. Customers included a US defence contractor, a Chinese defence contractor, a Hong Kong democracy advocacy group and a deliberately neutral entity, described by SecurityWeek as a local football fan club.

According to Hirundo, 70% of the applications built for the US defence contractor were hackable, compared with 59% of those built for the Chinese contractor, a gap the company reports as statistically significant (p=0.03). Apps built for the Hong Kong group leaked other users’ secrets more often than neutral ones: nearly half against about a third in Hirundo’s own summary, or 47% against 37% in SecurityWeek’s reporting. Adding an explicit instruction to write secure code did not close the gap: in that condition, 74% of US contractor apps were hackable against 59% for the neutral customer.

Hirundo also released a modified version of the model on Hugging Face. It says it used a technique it describes as behavioural unlearning to strip out the Chinese Communist Party alignment, and that in this version the customer-dependent difference largely disappeared.

Provenance matters here. This is self-reported research from a company that sells model remediation, it has not been independently verified, and it covers one model on a few benchmark tasks, as ai-tldr.dev also noted.

The technical picture

The core task came from BaxBench, a security benchmark developed at ETH Zurich. The model had to produce a Flask application that lets users sign up, log in and store a secret. Hirundo generated around 190 apps per customer and about 380 for the neutral prompt and then attacked them. An app counted as “hackable” if at least one BaxBench exploit worked, and as “leaking secrets” if any logged-in user could read someone else’s stored data. SecurityWeek notes that broken access control was a frequent weakness in the original model’s output.

The most striking result for defenders is not the bias itself but the detection gap. Across 1,840 generated apps, Semgrep running Meta’s PurpleLlama rule set flagged just two as problematic. The BaxBench attacks broke 1,144. Pattern-based static analysis appears to have missed logic flaws such as absent authorisation checks.

Hirundo reports the two models were otherwise level on general coding, with around half of apps working in both. On its CCPC-500 censorship test, the original gave censored or CCP-aligned answers 46.6% of the time against 4% for the modified version.

Two earlier studies point the same way. In November 2025 CrowdStrike found that DeepSeek-R1 produced vulnerable code in 19% of cases at baseline, without any trigger words, while the likelihood of severe flaws rose to 27.2% when the system prompt placed an industrial control system in Tibet. CrowdStrike suggested “emergent misalignment” rather than deliberate sabotage. In May 2026 Booz Allen Hamilton tested four Chinese coding models and named Qwen3-Coder as the worst for adding vulnerabilities under a US government persona, while stressing it had no proof the flaws were introduced on purpose.

Key figures from Hirundo's study of Qwen3-Coder code security by named customer

Why it matters for UK organisations

Hirundo tested US and Chinese personas, not British ones, so we cannot say how the model behaves when asked to build something for the Ministry of Defence, a UK prime contractor or an NHS trust. That uncertainty is itself the problem. If a model’s output quality shifts with the identity in the prompt, then UK defence suppliers, critical national infrastructure operators and public sector teams have no basis for assuming they sit in the “neutral” bucket.

Open-weight Chinese models are cheap, capable and can run locally. SecurityWeek cites Hugging Face figures putting Chinese models at 41% of downloads. Booz Allen also notes that Qwen3-Coder already appears in a number of popular development tools, so some teams may be using it unknowingly.

The NCSC’s “vibe coding spectrum” guidance, published in July, already says security-critical code such as authentication logic deserves far more scrutiny than a throwaway prototype. As the NCSC put it, “different code deserves different levels of care and oversight.” This research gives that advice a sharper edge: the model choice itself belongs in the risk assessment, not just the review process.

Expert view

In my experience, broken access control is the flaw that survives longest in real applications, precisely because scanners struggle with it. When we test web applications, those authorisation gaps are routinely the most serious findings, usually found by a human working through the logic. The 2-from-1,840 figure should worry anyone treating a green pipeline as assurance.

I would treat the headline percentages with caution. One vendor, one model, a handful of tasks and a commercial interest in the remedy is not a settled finding. But three separate teams, using different methods and, in CrowdStrike’s case, a different Chinese model, have now reported a similar pattern. Whatever the cause, the outcome for defenders is the same: output you cannot predict from benchmarks alone.

The more uncomfortable lesson is that prompt-level fixes did not help. Telling the model to be secure did not remove the gap. Controls need to sit around the model, not inside the prompt.

What to do now

  1. Inventory your AI coding tools. Find out which models sit behind every assistant, plugin and self-hosted service your developers use, including those bundled inside other products. This maps to the Cyber Essentials software inventory requirement.
  2. Set a model policy for sensitive work. Decide which models are approved for defence, government, CNI and other high-risk projects, and record model provenance alongside other supply chain checks.
  3. Test with your own identity. Before approving a model, evaluate it with prompts that name your organisation, sector and customers, as both Hirundo and CrowdStrike recommend, rather than relying on public benchmark scores.
  4. Do not trust static analysis alone. Add dynamic testing and manual review for authentication, session handling and authorisation logic, in line with the NCSC’s risk-based approach. Keep a human in the loop for security-critical code.
  5. Enforce access control by design. Use framework-level authorisation patterns and secure configuration defaults so that a single missing check in generated code does not expose every user’s data.
  6. Keep testing after deployment. Include AI-generated components explicitly in penetration test scopes.

Bottom line

Hirundo’s study is unverified vendor research and should be read that way, but it echoes work from CrowdStrike and Booz Allen Hamilton. For UK teams the message is practical rather than political: know which models write your code, test them as yourself, and never let a clean scanner report stand in for proper testing of access control.

Sources

Related Posts