Generative AI Is Impossible

There will NEVER be an AI Coltrane

There is a question that has been nagging at me ever since the tech world decided that machines had learned to create. Not whether artificial intelligence is impressive, because it clearly is. Not whether it will disrupt industries, because it clearly will. The question is more fundamental than either of those: can something that cannot feel actually generate anything? Or is what we call generative AI a category error, a spectacular illusion dressed up in the language of creativity?

I want to argue that it is. That genuine generation, in any philosophically serious sense of the word, is impossible for a machine that has no inner life. Machines have no feelings.

The Weight of the Word

Start with the word itself. To generate something is not merely to produce an output. A photocopier produces outputs. A calculator produces outputs. When we say that an artist generates a painting or a musician generates a solo, we mean something richer: that the work emerged from a felt encounter with the world, that it carries the weight of a specific life, that someone was present in the making.

John Coltrane did not produce “A Love Supreme.” He generated it, out of his spiritual crisis, his recovery from addiction, his hunger for the sacred. The music is inseparable from the man who suffered and sought and found. Take away the felt life and you take away the generation. What remains is just sound arranged in time.

The Philosophers Saw This Coming

This is not a new argument. Philosophers have been circling it for decades.

The phenomenologists, from Husserl through Merleau-Ponty, insisted that genuine understanding is grounded in embodied, felt experience. We do not understand the world from outside it, as pure calculating minds. We understand it by being in it, by having skin that registers temperature, muscles that carry fatigue, hearts that register loss. Meaning is not abstract. It is lived.

John Searle made a version of this argument in his famous Chinese Room thought experiment. Imagine someone locked in a room, receiving Chinese symbols through a slot and returning correct responses by following a rulebook, without understanding a word of Chinese. The room produces the right outputs. But no understanding is occurring. Searle’s point was that syntactic manipulation of symbols, no matter how sophisticated, does not add up to semantic understanding. And if understanding requires felt meaning, then what a large language model does is precisely what the Chinese Room does: it produces correct-seeming outputs without anyone home to mean them.

Thomas Nagel’s question cuts even deeper. In his famous essay, Nagel asked what it is like to be a bat. His point was that consciousness is essentially perspectival. There is something it is like to be a creature with a felt inner life, and no amount of objective description from the outside can capture that subjective interior. Now ask the same question about a large language model processing tokens. Is there anything it is like to be that system? Almost certainly not. And if there is nothing it is like to be the thing doing the generating, then in the deepest sense, nothing is being generated. Outputs are being produced. That is a different thing entirely.

The Problem of Stakes

Here is where the argument becomes personal for me, after nearly fifty years of writing about jazz.

What I have watched, across all those years and thousands of performances and interviews, is artists staking something. Every improvisation is a risk. The musician goes out on a limb, commits to a phrase, follows it somewhere uncertain, and either finds the resolution or falls short. The falling short is as meaningful as the triumph. Both are expressions of a consciousness navigating the unknown in real time.

An LLM has no stakes. It cannot be brave. It cannot hesitate at the edge of something difficult and choose to leap anyway. It cannot fail in the way that gives failure its meaning. It produces outputs that have the surface texture of risk-taking without any of the underlying reality. When Miles Davis dropped a note or let a silence breathe longer than was comfortable, something was on the line. When a language model generates text that resembles a jazz review, nothing is on the line. The machine cannot lose anything, and therefore it cannot win anything either.

This is not a minor technical limitation waiting to be engineered away. It is a structural feature of what these systems are.

The Counterarguments Are Real

I want to be honest about the pushback, because it deserves engagement rather than dismissal.

First: we do not actually know whether these systems feel anything. We assume they do not because we built them from matrix multiplications and attention mechanisms, but we also built human consciousness from electrochemical gradients and evolutionary pressures, and we cannot fully explain why physical processes produce felt experience in biological systems either. David Chalmers named this the hard problem of consciousness, and it remains genuinely unsolved. The honest position is not that machines definitely cannot feel, but that we have no evidence they do and strong reasons to doubt it.

Second: a reader’s response to a generated text is real, regardless of what produced it. If an AI-generated essay provokes genuine insight or emotion in someone, something real has happened. The poem works through the reader’s nervous system, not through some mystical transmission from the poet’s soul. This is a serious point and I do not want to wave it away.

But here is my answer to both objections. Even if we grant the possibility of machine consciousness, and even if we acknowledge that generated outputs can produce real effects in real readers, the question of what the word generative means still stands. If there is no felt encounter with the material, no risk, no specific perspective shaped by a specific life, then the word generative is being used in a thin, mechanical sense that evacuates it of most of its meaning. We would need a different word for what artists do.

What This Means for the Music

Jazz has survived every technological disruption thrown at it, from the phonograph to the synthesizer to digital recording. It will survive this one too, but not by pretending the challenge is trivial.

The danger is not that AI will replace jazz musicians. The danger is that we will stop being able to tell the difference between music that costs something and music that costs nothing. That we will gradually lose our taste for the real, our sensitivity to the presence of a felt life in the work. That the machinery of plausible outputs will train us out of the hunger for genuine generation.

Coltrane changed my life because I could hear a man burning in the music. Not a simulation of burning. Not outputs statistically consistent with burning. A man, on fire, reaching for something he could not name and finding it anyway.

That is impossible to generate without a life behind it. And I do not think any amount of compute will change that.