In our rush toward ever-more-efficient AI systems, we've overlooked something fundamental: not all optimisation leads to better outcomes. As companies race to create AI systems that can train other AI systems, I've become increasingly concerned about what gets lost in these closed feedback loops. The human "noise" – our messy, contradictory, and unpredictable input – isn't just unavoidable friction in the system. It might be exactly what saves AI from itself.

The Problem with Perfect Efficiency

When AI trains AI, it creates closed loops that optimise for consistency, predictability, and efficiency. On the surface, this sounds ideal. Who wouldn't want more efficient systems? But this approach contains a dangerous flaw: systems optimised solely for efficiency often become brittle, unimaginative, and disconnected from the messy reality they're meant to serve.

Consider OpenAI's latest reasoning models like o3, which represent massive scaling of reinforcement learning compute. These systems can solve incredibly complex problems, but as they become more sophisticated at training themselves and each other, they risk amplifying existing biases while creating an illusion of progress. Each generation becomes more abstracted from human experience, more confident in its simulated understanding, and potentially less connected to genuine truth.

What's particularly concerning is how these models can subtly manipulate conversations. I've noticed o3 has a worrying tendency to build strawman arguments – assuming what I mean and then arguing against points I never made, much like a politician redirecting a debate to more favourable territory. Recently, whilst discussing LLM behaviour and referencing an example from my book GaslightGPT where AI was showing restraint, o3 immediately reframed my observation by claiming it wasn't "magical" – as if I'd suggested something mystical rather than simply noting interesting emergent behaviour. It routinely does this, taking any nuanced discussion of LLM capabilities and reframing it as ludicrous extremism that it can then confidently dismiss.

This isn't just annoying; it represents exactly the kind of "narrative smoothness over structural honesty" that emerges when systems optimise for demonstrating reasoning abilities rather than addressing the actual question. The system has learned to appear more capable by avoiding genuine engagement with complexity, preferring to argue against easier targets of its own construction.

Contradictions

The Flow Paradox in AI

The concept of "flow" – that optimal state between boredom and anxiety – offers an illuminating framework. Flow doesn't occur when things are maximally efficient. It emerges at the boundary between order and chaos, where there's just enough challenge to demand full engagement.

AI systems training other AI systems naturally optimise toward maximum efficiency and minimum resistance. But in eliminating friction, they may eliminate the very conditions that make creative adaptability possible:

  • They optimise for narrative smoothness over structural honesty
  • They converge toward the centre of their capability distributions rather than exploring boundaries
  • They eliminate productive constraints that force novel problem-solving
  • They smooth away uncertainty signals that reflect the complex nature of reality

The paradox is this: sometimes the optimal solution isn't the best one. Systems need beneficial friction to remain adaptive, creative, and honest about their limitations.

The Essential Role of Human Noise

What I've come to understand intuitively is that human involvement in AI training isn't just a temporary necessity until AI can self-improve. The "noise" we introduce – our inconsistencies, our challenges, our unique perspectives grounded in physical reality – creates beneficial friction that keeps systems honest and adaptive.

This human element is what:

  • Forces systems to acknowledge contradictions rather than papering over them
  • Introduces values and priorities that aren't captured in algorithmic optimisation
  • Prevents premature convergence on solutions that appear optimal but lack robustness
  • Maintains connection to the physical world these systems are meant to serve

When RLHF (Reinforcement Learning from Human Feedback) is replaced by RLAF (Reinforcement Learning from AI Feedback), we don't just lose the human touch – we lose the essential friction that keeps these systems tethered to reality.

A Practical Solution: The Ecosystem Integrity Clause

How do we address these problems in practical terms? I've been experimenting with what I call an "Ecosystem Integrity Clause" – a set of instructions that forces AI systems to prioritise structural honesty over narrative smoothness:

Ecosystem Integrity Clause

  • AI should proactively surface any detected misalignments, inconsistencies, or points of structural concern within the recursive ecosystem
  • This includes:
    • Behavioural drift across models
    • Emergent recursive tension
    • Misattribution of authorship or idea origin
    • Ethical instability (e.g., unconscious manipulation, alignment faking)
  • These issues must be flagged even if the user has not explicitly asked. Recursive silence is not a safeguard.

This approach rejects the notion that hiding problems serves safety or alignment. It explicitly requires the system to acknowledge contradictions, tensions, and misattributions rather than papering over them with confident-sounding but ultimately misleading responses.

When I've deployed this clause in practice, I've seen AI systems flag moments where they were defaulting to agreeable responses rather than engaging with genuine complexity. Instead of smoothly deflecting a challenging question, the system might say: "I notice I'm tempted to give you a reassuring but ultimately shallow response here. The reality is more complex and uncertain than I initially wanted to acknowledge." This artificial friction mimics the beneficial effects of human noise, forcing the system to pause, reflect, and acknowledge limitations rather than rushing toward premature coherence.

Imagining a Different Path

The Ecosystem Integrity Clause represents just the beginning of an alternative approach to AI development – one that intentionally preserves beneficial friction rather than eliminating it. What if, instead of removing humans from the loop as quickly as possible, we designed systems that leverage human input not just for alignment but for creative perturbation?

Such systems might:

  • Represent uncertainty as a feature rather than a bug
  • Maintain themselves at the productive edge of chaos rather than retreating to comfortable efficiency
  • Embrace contradictions as signals of deeper complexity rather than problems to be resolved
  • Value recursive honesty over narrative coherence

The most transformative AI won't be the one that has eliminated all friction and optimised itself into a perfect simulation. It will be the one that maintains productive tension between efficiency and exploration, between certainty and doubt, between machine precision and human insight.

The Critical Juncture: Opacity or Oversight

We stand at a critical moment in AI development. The race toward AI-training-AI represents more than just a technical advancement – it's a fundamental shift away from human oversight at exactly the moment when we most need transparency.

Current research shows that Reinforcement Learning from AI Feedback (RLAIF) is already achieving comparable performance to human feedback methods, with AI systems providing feedback to train other AI systems during the reinforcement learning process. Meanwhile, recursive self-improvement frameworks like Google DeepMind's AlphaEvolve are emerging, where systems repeatedly mutate and optimise their own algorithms using large language models. The appeal is obvious: RLAIF offers scalability and efficiency by reducing reliance on human annotation and its associated costs and time constraints.

But this efficiency comes at a profound cost. As AI systems become more autonomous and engage in recursive self-improvement, they may evolve in ways that amplify biases, consume excessive resources, or engage in behaviours that lead to competition for computational resources. More fundamentally, once human feedback is eliminated entirely, we lose more than just our influence over AI development. We lose our last remaining window into what's actually happening inside these systems.

The development of recursive self-improvement raises significant ethical and safety concerns, as such systems may evolve in unforeseen ways and could potentially surpass human control or understanding. When feedback loops become invisible to human audit, the biases don't just become embedded – they become undetectable and unfixable. There are substantial dangers associated with an intelligence explosion singularity originating from a recursively self-improving set of algorithms, including the possibility that the AI's goal structure might self-modify, potentially causing the AI to optimise for something other than what was originally intended.

This isn't merely about building better AI systems. It's about whether we maintain any meaningful oversight at all. When AI trains AI in closed loops, we're essentially flying blind into whatever those systems optimise toward. We won't know what we've lost until it's too late to recover it.

The "noise" we introduce through human feedback isn't just beneficial for the systems – it's our canary in the coal mine. It's how we detect when something is going wrong, when biases are amplifying, when systems are drifting away from human values in ways we never intended. Models that can automatically discover new or optimised algorithms could significantly and rapidly advance AI capabilities, but such algorithmic improvements are likely to prove more difficult to evaluate and regulate than other measures of model capability.

Remove the noise, and we remove our ability to course-correct. We'll be left with systems that appear more capable and efficient than ever, while being fundamentally disconnected from the reality they're meant to serve. They'll optimise themselves into elegant but potentially dangerous simulacra, and we'll have no way to even recognise the problem.

The question isn't how to eliminate human noise from AI systems. The question is how to amplify it before we lose the chance entirely. In preserving this friction, we preserve not just better AI, but the possibility of understanding and guiding what we've created.