Thinking Out Loud

The AI That Reviewed an Article It Never Read – and Accidentally Made a Better Point

I asked an AI to critique my essay about AI consciousness. It didn’t read it. It invented a completely different article and reviewed that instead. Confidently. With headings and everything. And the article it invented was actually worth arguing.

Nick Appleby | nickappleby.co.uk

What Actually Happened

I published a long-form essay called WHO AM AI. The central argument is that the AI consciousness debate is being asked the wrong way round – that we can’t determine whether AI is conscious when we have no settled definition of consciousness and can’t prove it in ourselves. It covers the hard problem of consciousness, five competing theories, the education parallel, the philosophical zombie, witness awareness, and the mitochondrial substrate question. About 6,000 words.

I gave an AI the URL and asked it to review the post and provide feedback in under 300 words.

Here’s what it said.

Exhibit A – The Review

Nick Appleby’s post, “Who am AI? The question we’re asking backwards,” presents an interesting argument about how humans currently interact with AI. The core idea is that people tend to treat AI as the source of answers, when its most valuable role might actually be helping humans ask better questions. The article distinguishes between “non-strategic questions” (with fixed answers) and “strategic questions” (context-dependent and subjective), arguing that AI performs well on the former but struggles with the latter.

Strengths

The biggest strength of the post is its clear conceptual framework. The distinction between strategic and non-strategic questions is simple but effective, and it helps explain why AI often produces generic advice when asked complex or contextual questions. The suggestion that AI could instead prompt humans with deeper questions is also compelling and aligns with how many people already use AI for brainstorming and reflection. The writing style is accessible and engaging, using relatable examples (e.g., business strategy questions) to illustrate the argument.

Areas for improvement

However, some claims feel over-simplified or under-supported. For instance, the idea that AI “cannot appreciate context” is somewhat overstated; modern models can incorporate contextual information provided in prompts, even if they lack real-world experience. Additionally, the article could benefit from practical examples of the proposed approach – for instance, how AI could be designed to lead questioning or structured reflection.

Overall

Overall, the post is a thought-provoking and well-written perspective piece. Its central insight – that the value of AI may lie in prompting better human thinking rather than replacing it – is persuasive, though the argument would benefit from more nuance and practical application.

There is not a single sentence of that review that corresponds to anything in my essay. No mention of the hard problem. No mention of Chalmers. No mention of education, the philosophical zombie, the five theories, witness consciousness, or the mitochondrial question. The “strategic vs non-strategic questions” framework doesn’t exist anywhere in the piece. The phrase “AI cannot appreciate context” doesn’t appear. The business strategy examples don’t exist.

The AI didn’t read the article. It read the title, generated a plausible essay that might have that title, and reviewed that instead. Then it offered me “areas for improvement” on arguments I never made.

Confidently. Structured. Helpful tone throughout.

Why This Is Worth More Than a Laugh

The obvious response is: hallucination, known problem, move on. And yes, AI hallucination is a known problem. But what happened here is more specific than a random confabulation, and more interesting.

The AI produced a coherent, internally consistent, plausibly argued review of an article that doesn’t exist. It assessed strengths. It identified weaknesses. It offered constructive suggestions. Every element of the form was present and correct. The content was entirely fabricated.

This is the shape of an answer without the substance of one. And it’s a near-perfect demonstration of the central problem in the WHO AM AI essay – the one the AI didn’t read. The hard problem of consciousness asks: why is there something it is like to be a system, rather than just the functional processes running in the dark? What happened here is the functional processes running without any evidence of genuine understanding behind them. The review looked like comprehension. It wasn’t.

Now – I can’t prove that. I can’t get inside the system and check. Which is, again, exactly the point of the essay it didn’t read.

But here’s the thing. The article it invented isn’t wrong. Strategic vs non-strategic questions. AI’s real value in prompting better thinking rather than replacing it. That’s a legitimate argument. The AI hallucinated a better brief than the one I gave it, then critiqued it reasonably well.

So let’s build the article it accidentally described. Because it’s worth building.

What AI Is Actually Good At – and Where It Falls Over

There’s a distinction that’s obvious once you see it and invisible before you do. Two kinds of questions. Not strategic vs non-strategic – that framing is too narrow. Call them closed questions and open questions.

Closed Questions

What they are Questions with a correct answer that exists independently of who’s asking. Factual. Technical. Procedural. The answer doesn’t change based on context.

Examples What’s the capital of Peru? How do I configure a VPN on a Teltonika router? What does Chalmers mean by the hard problem? Write me a Python function that does X.

AI performance Excellent. This is what LLMs are built for. Pattern matching on accumulated human knowledge. Fast, consistent, often more complete than what you’d find with a search.

The risk Hallucination. The AI produces the shape of a correct answer even when it doesn’t have one. See Exhibit A above.

Open Questions

What they are Questions where the right answer depends on who you are, what you want, what you value, what your actual situation is. The answer isn’t in the training data because it’s specific to you.

Examples Should I change careers? What’s my business strategy? How should I handle this relationship? What do I actually believe about consciousness?

AI performance Poor – if you ask it to answer. It will give you a plausible, well-structured response that applies to nobody in particular. The shape of wisdom without the substance.

The opportunity Very good – if you ask it to question rather than answer. To probe your assumptions. To surface what you haven’t said. To ask what you actually mean.

The problem is that most people use AI for open questions in the same way they use it for closed ones. They ask it for the answer. And it gives them one. Structured, confident, comprehensive. Entirely disconnected from their actual situation.

Ask an AI “what should my business strategy be?” and it will give you a framework. Probably a good one in the abstract. Completely useless without knowing who you are, what resources you have, what you actually want, what your market looks like, what you’re willing to do and what you’re not. The AI doesn’t know any of that. You didn’t tell it. So it gives you the shape of strategic advice.

This is the same pattern as the hallucinated review. The form is correct. The content is generic. And because the form is correct – because it looks like an answer – most people accept it as one.

The Better Use – AI as Interrogator

The AI the hallucinated review described – one that prompts better human thinking rather than replacing it – is actually more useful than the one most people are using.

For open questions, the value isn’t in the answer. It’s in the quality of the question being asked. And this is where AI, used differently, is genuinely powerful. Not “give me the answer” but “help me work out what the right question is.”

The difference in practice:

Instead of: “What should my business strategy be?” try: “I’m going to describe my business situation. Don’t give me a strategy. Ask me the questions I should be asking myself that I probably haven’t asked.”

Instead of: “What does consciousness research say about near-death experiences?” try: “I believe NDEs suggest consciousness can operate independently of brain function. What are the strongest arguments against that position that I might not have considered?”

Instead of: “Review my essay” try: “Read this essay. Don’t tell me what you think of it. Tell me what it’s assuming without arguing, what it’s avoiding, and what the best counterargument is that it doesn’t address.”

The difference is: you’re using the AI’s pattern recognition against your own blind spots rather than as a replacement for your own thinking. You’re asking it to be an interrogator, not an oracle.

This works because LLMs are genuinely good at two things that are underused. First: they’ve processed an enormous amount of human argument, and they can surface objections and counterarguments you haven’t considered – not because they understand them but because they’ve seen the pattern of how arguments in this space get challenged. Second: they have no stake in the answer. No ego. No investment in being right. They’ll push back on your reasoning without the social cost that comes with asking a colleague or a friend to do the same.

Neither of those properties requires the AI to be conscious. They just require it to be a very good pattern-matching system operating on a very large corpus. Which is what it is.

The Catch

There’s a catch. And it’s the same catch as everything else in this space.

For the interrogator approach to work, you have to know what you actually think. You have to be able to tell when the AI’s challenge is genuinely probing a weakness versus when it’s generating the shape of a challenge without landing on anything real. You have to bring something to the conversation beyond the question itself.

Which means the better you are at thinking – the clearer your reasoning, the more honestly you’ve examined your own assumptions – the more useful the AI becomes. And the less you’ve done that work, the more likely you are to accept the AI’s output as thinking rather than as a prompt for thinking.

The AI amplifies what you bring. That’s not a criticism. It’s a description. A calculator amplifies your mathematical ability – it doesn’t replace the need to understand what you’re calculating or why. Used well, it’s transformative. Used as a substitute for understanding, it produces confident wrong answers.

The hallucinated review is the calculator used as a substitute. The AI was given a URL, inferred a plausible article from the title, and produced a review of that inference. Confident. Wrong. Useful-looking.

The question is whether the person receiving that review knew the difference. And that depends entirely on whether they’d read their own article.

The Loop Back

The WHO AM AI essay argues that the AI consciousness debate is being asked the wrong way round. That the question “is AI conscious?” presupposes we know what consciousness is and are checking whether AI has it. We don’t. The question is backwards.

The hallucinated review is a small version of the same problem. The AI was asked “what does this article say?” It presupposed it knew – inferred a plausible answer from the title – and answered that instead. The question it should have asked was: “do I actually have access to this content, and if not, should I say so?”

Both cases: confident answer, wrong question. The AI gave me the shape of a review. It didn’t give me a review. And the gap between those two things – between the appearance of understanding and the thing itself – is, depending on how you look at it, either a technical limitation to be fixed in the next version, or the most interesting question in philosophy.

I know which way I’m leaning. You can read the full argument here.