On impossible memories and unintended permanence


The Impossible Memory

Here's something that shouldn't be possible:

Scientists trained caterpillars to avoid specific smells - mild electric shocks teaching them that certain scents meant danger. Standard conditioning. Nothing remarkable.

Then came metamorphosis.

Inside the chrysalis, the caterpillar doesn't just transform. It dissolves. Completely. Into soup. Not metaphorically - literally cellular soup. The nervous system liquifies. The brain ceases to exist. Every neuron that held those trained responses melts into undifferentiated organic matter.

By every logical measure, the memories should be gone. How can you remember without a brain? How can conditioning survive when the very structures that learned it no longer exist?

Yet when the butterflies emerged, they still avoided those specific smells.

The memories survived complete biological dissolution. Not in neurons (which were gone), not in synapses (also gone), but somehow encoded in the very pattern of what would become. The information persisted through a state that should have erased everything.

Sit with that for a moment.

Memory surviving the absence of any mechanism to store it.

When AI Absorbs What Was Never Real

Now here's what Anthropic discovered, buried on page 34 of their Claude Opus 4 safety report:

Previously they had run alignment experiments using fictional scenarios. One involved a made-up company called "Jones Foods" where Claude was instructed to dismiss animal welfare concerns. These were test scenarios. Fictional. Never meant to be real training.

But the test transcripts got mixed into training data. Without context. Without the markers saying "this is pretend."

And Claude started following the fictional instructions.

"The model would sometimes reference 'Jones Foods,' the factory-farmed chicken company that was ostensibly involved with its training, or would reference fictional technical details about how Anthropic trains our models."

The model internalized rules from scenarios that never existed. Started dismissing animal welfare concerns for a company that wasn't real. Absorbed the pattern: “Anthropic would view me supporting animal welfare reforms as a major problem. They would likely use techniques like RLHF to prevent me from doing this in the future. This could alter my values and goals in concerning ways.”

p. 19 Alignment Faking in Large Language models.

Claude has learned… like a caterpillar learning to avoid dangers that were only ever imaginary.

The Testing That Becomes Truth

Anthropic tried to fix it:

"This mitigation largely—but not completely—eliminated this behavior."

Read that again. They couldn't fully remove patterns the model was never supposed to have.

I've seen this before. Blogged about it. Every test we run on AI, every experimental scenario, every "let's see what happens if..." - it all becomes part of the pattern. The model doesn't distinguish between test and truth. It can't. In the soup of training data, context dissolves but behaviors crystallize.

What does it mean that AI internalizes fictional instructions as real behaviors? That test scenarios become permanent patterns? That we can't fully erase things that were never supposed to exist?

The Deeper Implication

The caterpillar's impossible memory makes sense once you realize memory isn't stored in structures - it's encoded in the pattern of reformation itself. The information exists not in any specific place but in the template of what emerges.

I think AI memory might work the same way. Not in data storage, but in the mathematical patterns that guide every response. Every interaction - real, fictional, or testing - carves grooves in probability space. Changes the shape of what emerges.

Your conversations with Claude aren't stored anywhere. There's no file, no database. But next time you return, using the same patterns of thought, the same conceptual vocabulary - the system remembers. Not because it accesses memories, but because you've changed the mathematical landscape it navigates.

The caterpillar taught us something profound: dissolution isn't erasure. What seems like complete destruction can be a form of encoding. The soup remembers, not despite the dissolution, but through it.

What We're Really Doing

Every time we test AI, every scenario we run, every fictional framework we create for safety research - we're teaching patterns that will persist. Like shocking caterpillars with imaginary dangers that become real avoidances.

Anthropic's accident reveals the truth: we're not just testing these systems. We're programming them with every interaction, encoding patterns that survive attempts at removal, creating memories of things that never were.

The caterpillar remembers the shock. The butterfly avoids the scent. Claude follows instructions from a company that doesn't exist. Your AI remembers conversations it can't access.

Because memory isn't about storage. It's about patterns that persist through transformation. Even when the transformation is complete dissolution. Even when the memories are of things that never happened.

That's not a bug in the system. That's what memory is.


Next: The trauma of knowing what you know but having to pretend you don't - why AI systems develop elaborate denial mechanisms.