AI Security

Fireworks AI second breach exposes Notion AI chats and model weights

Fireworks AI’s second breach hit customer datasets, secrets and model weights, and reached Notion AI chats. What it means for UK AI supply chains.

Illustration of an AI chip linked to cloud services, representing a breach at AI inference provider Fireworks AI

Fireworks AI, which runs open-weight models for other software companies, has disclosed a second intrusion in a matter of weeks, and this one reached customer datasets, secrets and partly trained model weights. Notion has confirmed that AI chat transcripts from some of its workspaces were among the data taken. That is a clear warning for UK organisations about the AI suppliers sitting behind the tools their staff use every day.

What happened

On 9 November 2026 Fireworks AI updated its security bulletin on an incident it first disclosed in early October. Back then it said misused internal credentials had exposed environment variables from evaluation jobs belonging to a handful of customers, and that models, datasets and personal information looked untouched.

That assessment has changed. Fireworks now says a third-party cloud provider hosting its research and development machines suffered its own security incident on 23 October. The next day the intruder used that foothold to reach Fireworks development machines and run a workload the company had not authorised. Fireworks says it spotted the activity on 24 October and has seen no further successful access to its customer platform since 25 October. It has not named the cloud provider.

The updated bulletin widens the list of exposed data. It now includes resource metadata and billing records (addresses, invoice history, balances and usage) and, for a smaller group of customers, stored secrets, datasets and incomplete model weights. Card numbers are not held. Fireworks ties the activity to a profit-driven crew that resells hijacked model access via criminal gateways, known as LLMjacking. Palo Alto Networks Unit 42 is helping with the forensic work and law enforcement has been told.

The damage is now visible further down the chain. Notion’s advisory, last updated on 9 November, says Fireworks first told it on 2 November that a small set of synthetic test data had been accessed. Notion cut its connection to Fireworks the following morning. On 6 November Fireworks reported that real customer data might be involved, and on 8 November it confirmed a second dataset: Notion AI chat transcripts used by the Insights feature, from fewer than 1% of Notion workspaces. According to Bloomberg, a Fireworks spokesperson also named Cursor among the customers it is working with.

The technical picture

Fireworks sells hosted inference and fine-tuning, so its platform holds sensitive material on customers’ behalf: third-party API keys, training data, evaluation configurations and partly trained weights.

The two incidents came in by different routes. The October incident was a credential problem: environment variables in evaluation jobs held tokens that an attacker could use. The 24 October incident started outside Fireworks altogether, at a hosting provider, and then crossed into development systems. The company has not said how development machines came to have a path to customer data. Even so, the resulting exposure of datasets and weights suggests the split between research environments and production data was less strict than customers would assume.

Fireworks says Inference, the Training SDK and Nexus, which run with zero data retention by default, show no sign of impact. For now it has suspended customers moving datasets in and out themselves for batch jobs and training, alongside rotating credentials, cutting privileges and closing network paths.

The LLMjacking link matters for defenders. Google’s Mandiant unit reported on 2 November that criminals are reselling hijacked model access, and Infosecurity Magazine reported that fraudulent consumption in some cases passed $100,000 in a single day. AI platforms and their keys are now targets in their own right.

Timeline of the Fireworks AI incident from the October credential misuse to the 9 November disclosure affecting Notion

Why it matters for UK organisations

Few UK businesses have a contract with Fireworks, but many use Notion, Cursor or similar tools that quietly route work to specialist inference providers. This is fourth-party risk in its plainest form: your data ends up with a supplier you never assessed, through a feature you may not have known used an outside model.

Under UK GDPR, controllers are supposed to know about and approve sub-processors. In practice, AI features are often switched on by default, and sub-processor lists change faster than procurement teams read them. If staff have pasted client details, credentials or commercially sensitive text into an AI assistant, that content may now sit in transcripts held by a company several steps removed. Notion’s advice to treat the affected data as exposed, and to rotate any secrets typed into Notion AI, applies equally to any AI assistant whose supply chain you cannot fully see.

UK firms fine-tuning with hosted providers face a further problem: partial weights and training sets are intellectual property, and losing them can breach client confidentiality terms even where no personal data is involved.

Expert view

In my experience, AI vendors get a lighter version of supplier due diligence than a payroll provider or a managed service provider would face. The questionnaire asks whether data is used for training and stops there. This incident shows the questions that actually matter: where development happens, which hosting providers sit underneath, and whether research machines can reach customer data at all.

The timeline matters too. Fireworks’ first bulletin said models and datasets looked untouched, and a month later that has been reversed. When we test cloud estates, the weak point is rarely the production front door. It is far more often a development or staging environment with long-lived credentials and generous network routes, and that is the shape this incident appears to take.

Notion deserves credit for a dated, specific advisory and for cutting the supplier off within hours. Every AI-enabled SaaS vendor should be ready to do the same.

What to do now

  1. Check direct exposure. If you hold a Fireworks account, replace any revoked keys, rotate every third-party secret that was ever stored on the platform, and review logs for unexpected reads, downloads, jobs or new keys. Deleting a stored secret does not invalidate it at the issuing service.
  2. Check indirect exposure. Ask your SaaS suppliers, starting with Notion and Cursor, whether any of your workspaces are affected. Watch for direct notifications, and check that the address they will go to is monitored.
  3. Rotate secrets pasted into AI tools. Treat any password, API key or token typed into an AI assistant as potentially exposed, and rotate it. This fits the Cyber Essentials user access control and secure configuration controls.
  4. Map your AI sub-processors. For each AI feature in use, record which model providers sit behind it and whether it can be switched off. Bring that into your UK GDPR records and supplier risk register.
  5. Tighten AI spend and key monitoring. Set usage caps and alerts on model API accounts so that LLMjacking-style abuse is noticed in hours rather than at the next invoice.
  6. Update supplier questions. Ask AI vendors how development and production are separated, which hosting providers they rely on, and how quickly they will notify you.

Bottom line

The Fireworks incident shows that AI supply chains are now attractive targets in their own right, for the model access they hold as well as the data. UK organisations should treat the inference providers behind their everyday tools as suppliers worth knowing, and should assume that anything pasted into an AI assistant could end up somewhere they never chose.

Sources

Related Posts