How does AI keep growing?

Let’s play a thought experiment.

Imagine AI has no ceiling. Infinite compute, unlimited energy, no economic fallout from automation. The full techno-utopia. Everything the most AI-pilled among us promise is just around the corner.

Now ask yourself: where do the new ideas come from?

LLMs need training data to learn. And while the marketing wants you to believe these models are capable of genuine novelty, the evidence mostly points in a different direction. They’re remarkably good at synthesising and applying existing knowledge. That’s powerful, but it’s not the same as invention.

So in our hypothetical world where AI does everything, who’s generating the new knowledge it needs to keep improving? In software alone, how does AI advance if the body of knowledge never moves beyond transformers and attention mechanisms? You’d eventually hit a wall, not because of a technical limitation in the architecture, but because the well of human-generated ideas has dried up.

You could go full E.M. Forster here and paint a picture of humanity forgetting how to think for itself, like in The Machine Stops. I don’t think it’s quite that bleak, but the underlying tension is real.

Until AI can genuinely originate ideas the way people do, it still needs us out here doing what we do. Researching, creating, failing, experimenting, writing things down. The companies building these models like to sell the dream, but what they don’t mention is that their future and financial existence completely depends on humans continuing to grow the very corpus of knowledge their products are built on.

AI doesn’t replace the need for human creativity. It actually makes it more important.