"Model Memory" attacks. I was wrong.

I thought AI "Model Memory" (whether the model or the logs) was just security FUD. Wasn't that solved already? Sure, sensitive information is involved - but please tell me it's not accessible… Right??

I’ve been knee-deep in AI retention stats lately, and one concept kept nagging at me: the idea that "model memory," such as retained prompts, chat histories, or session context, is quietly killing projects.

I’ve now reviewed several AI rollouts this year, and I’ve never personally seen an issue with it. It always felt like a keynote stump-the-chump quip: "What if the model remembers your SSN?"

So I went digging. Turns out, the problem isn't "Skynet never forgets" and the borg is after you. It's less interesting because it's boring, messy human error. But the damage is real.

Here's 4 times in 2025 where retention triggered bans, patches, or headlines.

1. DeepSeek's Exposed Chat Logs (Jan 2025)

DeepSeek AI platform exposed user data through unsecured database | SC Media

This startup left a ClickHouse database open. No "model regurgitation," but millions of plaintext chat contexts (PII + keys) were exposed.

👉 The Cost: "Security experts noted that such an oversight suggests DeepSeek lacks the maturity to handle sensitive data securely. The discovery raises concerns as DeepSeek gains global traction [...] prompting scrutiny from regulators and governments."

2. Microsoft 365 Copilot's EchoLeak (June 2025)

Inside CVE-2025-32711 (EchoLeak): Prompt injection meets AI exfiltration

A zero-click vulnerability allowed attackers to hijack the retrieval engine. The model’s working memory blended malicious input with user privileges, leaking docs without a single click. This resulted in CVE-2025-32711 (EchoLeak).

👉 The Cost: Shines a spotlight on Copilot’s prompt parsing behavior. In short, when the AI is asked to summarize, analyze, or respond to a document, it doesn’t just look at user-facing text. It reads everything, including hidden text, speaker notes, and metadata.

3. OmniGPT's Mega Chat Dump (Feb 2025)

OmniGPT Claimed To Be Subjected to Extensive Breach | MSSP Alert

Hacker "Gloomer" dumped 34M lines of user chats. The "memory" here was full conversation history stored for personalization, including financial queries and therapy-like vents.

👉 The Cost: 62% of AI teams now fear retention more than hallucination (McKinsey).

4. Cursor IDE's MCPoison (July 2025)

Cursor IDE's MCP Vulnerability - Check Point Research

A trust flaw in the Model Context Protocol allowed attackers to swap configs post-approval.6 Once in the session "memory," it could execute commands on every launch.

👉 The Cost: Devs at NVIDIA and Uber had to update frantically to close the backdoor.

The Bottom Line:

These aren't daily fires, but they are real issues that still need to be addressed. You may have seen my current discussions on storing hashes instead of payloads. There are multiple reasons behind this such as privacy, security, storage requirements, network requirements, processing, validation speed, etc. But it also helps reduce your attack surface for issues like this.

Of course, it's only one layer of defense. Data retention is still here. Good teams "receipt-ify" (store hashes, not payloads) and also enforce purges of sensitive information.

One slip, and you're the next headline. Define and enforce good hygiene.  

Stop Saying “Compliance ≠ Security.” You’re Missing the Point.

“Compliance is just theater. Checkboxes don’t equal security.” I'm sorry, but this is just wrong.

Why? Because every authoritative framework worth pursuing has mandated risk management. It’s not hidden in an appendix.

Every Major Framework Explicitly Requires Risk Management

• SOC 2 (Trust Services Criteria): CC3.0 Risk Assessment and CC9.0 Risk Mitigation are mandatory common criteria. No documented risk assessment = automatic failure.
• NIST SP 800-53 Rev 5: The entire RA and PM families require an organization-wide risk management framework. Controls are then tailored.
• PCI DSS v4.0 Requirement 12.2 mandates a formal annual risk assessment. Requirement 12.3.1 introduces targeted risk analysis to justify control frequency.
• ISO 27001/27002 (2022) Clause 6.1.3 and control 8.2 require you to establish, implement, maintain, and continually improve a risk management process.
• And on it goes.

These aren't suggestions. Risk Management is a mandatory exercise.

You are required to exceed minimums when your risk demands it. The frameworks explicitly say the baseline is a floor, not your high-water mark.

What About Breaches?

When organizations are technically “compliant” and still get breached, the failure is almost always tied to a nonexistent or terribly executed risk management program.

Mature programs:

• Start with the required risk assessment, then select and tailor controls
• Apply stricter measures to high-risk/crown-jewel assets, lighter ones elsewhere
• Exceed minimums where their own risk analysis justifies it
• Continuously reassess because every framework demands it

That’s not “checkbox compliance.” That’s literally what the standards require.

So next time you’re tempted to say “compliance doesn’t equal security,” I'm curious to see your last risk assessment that actually drove control selection.

Because if your takeaway from reading SOC 2, NIST, PCI DSS, ISO, etc. is “just a checklist,” the problem might not be the framework...