The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness.
It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.
Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.
There are people whose brains don’t form new memories anymore after an accident or surgery, and they eternally live in the time before it happened, and have no memory of what happened a minute ago. Still they are conscious.
I think it's a little more complicated than that. In a 50 First Dates type of scenario, their ability to form certain types of memories is damaged, not non-existent. And I would argue that with enough brain damage someone like an extreme lobotomy victim may stop being considered conscious.
I’m not familiar with 50 First Dates, I was thinking of cases like Clive Wearing [0]. I would agree that consciousness requires some sort of ultra-short-term working memory, but I also think that mechanisms similar to CoT loops can conceivably fulfill that role. Today’s generative AIs consist of more than just the static network-of-weights model.
"Wearing can learn new procedures and even a few facts, not from episodic memory or encoding, but by acquiring new procedural memories through repetition. For example, having watched a certain video recording multiple times on successive days, he never had any memory of ever seeing the video or knowing the content, but he was able to anticipate certain parts of the content without remembering how he learned them."
Honestly, that's a pretty messy state of consciousness and I wouldn't proudly crow that my AI is conscious if that's as good as it got
They are conscious because even for short periods of time they do form memories and those change them even if only briefly. They think on their own too. It is a very limited level of consciousness though.
That could be easily fixed by providing the AI with a constant stream of input.
For humans, part of the input of the human mind comes from the continuous processes and clocks within the human body, so it’s questionable whether the brain could “think on its own” without such input either.
The continuous input for the human arises naturally, it doesn't arise naturally for an LLM unless we direct it so. Our consciousness is bootstrapped, the LLM isn't.
I think you have grazed my stance on this topic in the sense of what separates LLMs from complete human (or any other biological life) sentience.
It's the constant sensory input of the world and the realization and drive to survive as the second order effect of it. Mortality, vulnerability to external factors codified as input could in fact allow the LLM to independ as sentience.
Of course besides the sensors, it would also need a way to affect the physical world, and to be able to monitor the degradation if its own hardware, but when that barrier is crossed, it would be much closer to full sentience than whatever we have right now (which is nowhere near sentience or AGI).
We have virtually no idea how consciousness arises in the human brain. Furthermore, what is “natural” supposed mean here, and why should it matter for consciousness whether some prerequisite arises naturally or not?
I was literally only responding to "Is that any different from an LLM having a context window" lol. Let's keep that in our context windows. I'm not interested in discussing how human consciousness is different from supposed LLM consciousness; it's enough for me to know that humans are conscious, and in an obvious, clearly distinct way, even if we can't define it. Sniffing our own farts about whether LLMs are or aren't is just that – nerds larping as philosophers while practicing fart sniffing. It's a machine, periodt, we can quit roleplaying.
- A motor is something that create a force to push a vehicle.
- Oh yeah? My neighbour car does not have wheels and sit on concrete blocks, the vehicle does not move and yet we all agree it has a motor. So it means that I can claim that this other thing that does not move has a motor too.
Sure, human can _some times_ not do some stuffs, but the fact that they can do these stuffs sometimes is the point.
Doing these stuffs is the hard thing. Doing these stuffs is the proof that the machine has what it takes. It does not matter if someone cannot do that stuff, it does not imply that their internal system is not complex enough to potentially do it. But the fact that some people can do that stuff is the demonstration that inside a human skull, there is a system that is complex enough to potentially do it. Unless you can prove that people who don't do it have a fundamentally different system inside their skull, then you cannot pretend that they should be considered as having a less complex system.
Human _can_ check themselves. They don't _always_ check themselves.
Motor _can_ move vehicle. They don't _always_ move vehicle.
LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
So, yes, it is a refutation. If you have something that _never_ can move a vehicle, this thing does not qualify as a motor, even if some motor, sometimes, don't move a vehicle.
And if your next argument is "yeah but I would argue you don't need to check yourself to be conscious or to understand things", then you just redefine the definition that is owned by your interlocutor. Your interlocutor is saying that this is a criteria they are expecting. Good for you if you are not expecting this criteria. But the problem is that the answer is not "this criteria is not expected", the answer is "I change the criteria from 'being capable to in some circumstances' into 'does always do it in any circumstances'".
> LLM _cannot_ check themselves. They _never_ can. It is not that some don't, they just cannot, they are not a system complex enough to do so.
All modern agentic harnesses can do this. Nobody uses raw LLM for anything remotely complex. There's always some external system in place. That system is part of the "thought process".
Adjacency doesn't matter here, only what the result of the system of pieces is.
It means having self-control on their action and being aware of them. If you ask a system, it will respond, it cannot choose to not respond (even if the response if "I don't want to response", it still "run", still do the work). If you don't ask a system, it will not respond.
Adjacency is the point of the thread here. Saying "you say X is important to decide if the thing is intelligent/understanding/conscious, so let me just change X in the middle of the discussion and say that X does not matter".
That is exactly my first comment in this thread: I don't care if AI think or whatever, my reaction was about these "counter-arguments" that totally miss the point and make the person who push them ridiculous. If you want to have a counter-argument, you first need to understand the interlocutor, not just spew whatever rebuttal you constructed that answer something unrelated to what the interlocutor brought to the conversation.
In my thought process, I quite literally stop myself, and say "ok, think about what you just said" to check myself. I literally initiate that loop. If I don't, then I'm not using my own mental agency, and just using my firm coded priors.
I will say that I do seem to have a stop, what you said is wrong logic check voice that pops up without me initiating it. But, it's unreliable, and not too much different than all the content monitoring system used for the streaming clients, that will terminate with "content violation" immediately after the "incorrect" words are sent. I don't think integration is important, just the behavior of the overall system.
There is no "loop" in the brain, it is all part of a same line of thought. This is visible because, while you can sometimes have a "two voices / dialectic" way of thinking, you can have the exact same thinking in one-go that does not look like a loop at all.
In fact, in the large majority of the time, you don't process "as a loop" at all, you just continuously progress in your reflection without needing a "second voice" to retrigger you. The fact that sometimes we do this is just something we can do, not the result of something needed for our brain to work. For AI systems, this is something needed because the "answering" part is not able to do the loop on its own. And building a bigger system that combine an "answering" part and a "loop" part does not fit this, does not create a self-reflective system, it just makes a non-self-reflective system and a workaround bundled together.
It's a bit like if the "answering" part was unable to provide only one answer and was always producing plenty of different possible answers, including contradictory ones. Then, you can add an external part that will just pick one answer (and add it to the context so the next large set of answer will not be inconsistent), without any intelligence to it. The whole system will look like a human. But we know that the system is not "living" and "aware", because a "living" or "aware" system has its own opinion, while this system is just generating convincing sentence without seeing any hierarchy or value or meaning in each one.
I would claim that, if you think without introspection (that loop), then there is virtually no self check. I'm not sure what "self check" you see that the brain has. Could you describe this "self check in a line of thought"? How do you perceive the check there? This is a genuine question. It definitely doesn't align with how I think about things. I ponder and talk to myself to iterate verify and test my understanding of my own thoughts.
Maybe a good analogy is "throwing a paper plane in real life" and "throwing a paper plane in a video game".
In real life, the paper moves "by itself". It does not need an external loop that update its position in a loop manner.
In the video game, you need an internal loop, a step-by-step tick, that update the plane position based on its current position and its momentum. And this is why a video game paper plane is not a real object. It is a very good simulation, it looks like it, but it is missing some intrinsic properties that we expect from a real object.
Yet you can analyse the paper plane trajectory and see it as a Markov chain, with quantified step-by-step progress (for example one position point every 0.1 second). The same way you can look at your though process and identify a step-by-step progression. But it does not mean that it works like that intrinsically, it does not mean that the paper plane "jumps" from position point at time T1 to position point at time T1+0.1 second.
For the human brain, there is no "loop centre" in the brain. There is no one (to my knowledge) who got a brain injury and suddenly were unable to keep a single line of thought without having someone else having to feed them the previous thought in order to feed the next thought.
In the brain, the fact that the previous thought feeds the next thought is "how it works", it is intrinsic, it is by design. And this mechanism of thoughts feeding the next thoughts is what creates "consciousness" or "awareness": self-reflection is based on the fact that thoughts are intrinsically linked together, that they "flow" continuously, without needing an external system to update them.
You cannot take away the "loop" part of the paper plane so that it suddenly would be unable to move on its own once thrown away.
Now, you can always say "well, the paper plane in the video game is a very good simulation, it does not matter if it is a real object or not", and that is fair enough. But in this discussion, some people have arguments to support that this property matters, that it is one condition for consciousness or awareness.
Is your argument that, because they're external to the Llm, rather than integrated, they don't count, not even in a practical sense?
I think the result of the system is all that's important. Where/how it's implemented doesn't matter for practical results.
If the argument here is that LLM don't have this built in, you should know that nobody has a practical use for plain LLMs these days. Nobody uses them this way, except for debug. All interesting use is through some kind of harness, with all sorts of systems bolted on. I think these conversations are only meaningful in this "agent" context that people actually use LLM, where they stop when they think they're done.
LLM don't have a some self contained loop, like we do, sure. Who cares though. The actual AI system that we use every day definitely do.
Have you read the article in question. It is saying that for one continuous thought, the brain will use different part of the brain to do different thing. It does not say that there is a "loop controler" anywhere. On the contrary, it illustrates that there is no loop controller: there is not special brain function that control this loop, this loop is "how the brain works", and LLM don't do that, they are incapable to do that, it is not how they work.
> Is your argument that, because they're external to the Llm, rather than integrated, they don't count, not even in a practical sense?
No, my argument is that the nature of the brain and the nature of the LLM are very different, as different as a real paper plane and a video game paper plane. Some characteristics (for example, awareness) that exist in the brain cannot exist in the LLM because these characteristics are the result of the nature of the thing in question.
The problem is not that you build a system by integrating 2 things together. The problem is that they are different "things", they are different machines, they function, fundamentally, differently. They may produce the same output, but when you say "the brain has the characteristic X, the LLM produce the same output, so the LLM also has the characteristic X", it is logically inconsistent.
Planes are built as a system combining 2 things: a motor and some wings. But they are fundamentally different from a bird. They just don't "work" the same. It is not the same mechanism.
> you should know that nobody has a practical use for plain LLMs these days
That is totally irrelevant. My point is about the nature of the LLM, and the fact that it is stupid to see the same output and to conclude that they have the same characteristic. It is like saying "Birds are flying in the air and are alive. Planes are flying in the air, so I guess they are alive".
> LLM don't have a some self contained loop, like we do, sure. Who cares though. The actual AI system that we use every day definitely do.
No, you miss the point. The problem is not that "you can just add an external loop". The problem is that the brain is a system that works without such control loop. The thoughts are flowing (and they may flow to different brain functions, like explained in the article you quote). It is part of how the system works. Having a system that contains 2 things, one that does one computation and one that control the loop is not equivalent to another system where you cannot decouple the "flowing of the thought" from the "thinking machine".
Yes, LLMs don't think on their own, for one; they think when you invoke them.
My rebuttal is that people only think when invoked just the same and can enter states where there is no consciousness just the same. The OP has already accepted that LLMs think, but it seems that you are arguing they do not? This car business is confusing and the LLMs not checking themselves is also wrong, there’s even a benchmark for this
https://correctbench.github.io/
What you said: I have example where, sometimes, human think when invoked.
That's the difference: human brains are intrinsically different because they are built to be able to think without being invoked, even if there are situations where they think when invoked.
There are tons of obvious examples of human thinking without being invoked. Just take a bath and you will see :)
To be clear: the person I was replying to asked if the way a human thinks was any different from an LLM with a context window. That's the context of my answer. An LLM is a machine, it can't do anything unless we invoke it or give it the instructions and capabilities to do so. It has no free will, it can't just decide to compose a symphony one day unless those are part of its instructions. It can't do anything unless we tell it to do so and give it the capabilities to do so, it doesn't even exist unless it's loaded into memory. That's obviously different from human consciousness, and that's the whole of the point that I'm making.
You can argue that humans are just biological machines reacting to external stimuli, but that's a philosophical argument that I'm not interested in having and frankly, I think it'd be selling yourself short a little bit.
Thank you I appreciate your opinion. I do think that we are reacting to external stimuli even though our ego is uncomfortable with being deterministic in any way, which is the free will point that you hit on. I think that is likely to be the point that keeps the argument going as it’s not a settled debate absent any AI, which we clearly all see as either deterministic
or semi-random when the temperature gets turned up.
As far as the argument about being loaded in memory, if there’s any consciousness in AI, it’s obviously in different form than a biological consciousness. We’d have to agree that consciousness does not require a body to get past this.
I'd argue that the context window is analogous to short-term memory. It's functional but limited in duration, and if you overload it, it starts to fail.
It's the long-term memory (i.e. learned experiences feeding back and directly altering the content of the core brain, or model) that is missing.
The context window is so flawed that I wouldn't consider it memory.
It feels like notes about the situation rather than it being in memory. Memory has more "attention". I think that "it starts to fail" is load bearing here.
I feel like memory has like 5 parts, and LLMs are missing 2 of them:
current working memory
short term what is immediately happening without it being in "RAM". I differentiate here vs working in like thinking fast and slow. Keeping things in working memory is work! You can vibe away short term memory. I had excellent short term memory while I was messed up, I could keep time well. I think LLMs can do this with notes.
mid term: Vague awareness of things like what day a week it is or what you did 2 hours ago. This is where my memory personally failed
long term memory of experiences. You can fake this with memory.md
generalized wisdom for pattern matching long term memories
LLMs seem to be missing that part I was missing. Im probably projecting and anthropomorphizing. But i relate: I would confabulate a ton and didn't know anything was wrong for a while but things seemed off.
Context is like working memory but not short term or mid term. I think you can imply short term with big enough context.
My categories are purely anthropomorphic to me but just wanted to say where I disagreed.
Thanks for sharing your experience. It's really interesting that you describe a loss of some 'middle' parts but not others. The 'classic' medical/psychological model of memory has three parts (sensory/short-term/long-term), but it's also worth noting that that model was first devised in 1968!
> long term memory of experiences. You can fake this with memory.md
Not sure about this; to my mind, memory.md is analogous to humans making lists of things to not forget to do, or notes from a lecture to learn (i.e. cram into long-term memory) later on. LLMs use it as a short cut to bring important facts back into their context window; but it's not the same as them already 'knowing' the information via the original training process.
---
My consistent (hot?) take is that a (the?) major piece holding LLMs back (maybe even from AGI?) is continual learning. Humans have systems for continually updating their long-term memory from their lived experience - new facts, processes, skills, successes, mistakes, etc. (Sleep and dreaming are centrally involved in this process.) The current architecture of LLMs makes this practically impossible, as it would presumably require the level of power currently necessary for training to be continually applied for continual learning, and demonstrates the huge efficiency advantage of the biological brain.
True, but I don’t see how that relates to consciousness. An LLM being continuously RLHF-trained also changes its habits; that alone doesn’t make it conscious.
The starting file may be immutable, but the whole processing of that file is very dynamic and intense. Maybe, if there is some consciousness, it lies somewhere during that processing.
An LLM’s training could be seen as lived experience, and the fact that LLMs can output long sequences from their training material can be interpreted as them remembering those parts.
Also, how does that relate to consciousness? I don’t think that past episodic memory is necessary for consciousness.
you can't be conscious about your decisions if you don't incorporate their effects into your corpus — knowing the results of other people's actions secondhand isn't the same because you're not those people
There is, you talk to an Alzheimer patient and its like that, and it doesn't feel like talking to a human any more. An Alzheimer patient isn't cured by adding some input noise to stop them from repeating conversations, they are still unable to learn, just like an LLM.
this just means they are incomplete, like a baby that has no long term memory. I think the baby analogy will hold up as we build more and more capability.
> The model definitely remembers previous exchanges within the same conversation.
No it doesn't. They get added to its context, and it reads them afresh when answering the next question. That's not remembering.
If your short-term memory completely malfunctioned one day, so you had no ability to remember what was said to you a minute ago, then you would have to find workarounds. For example, you could write down everything someone says to you, then read your notes of the previous exchanges in that conversation in order to continue the conversation. That would be a good way to work around the fact that your short-term memory was broken. And if your notes were invisible to other people and you could read them really fast, then you could even make most people believe that you remembered what they said a minute ago. But you don't actually have a working memory, you're just writing down what they said and re-reading it while coming up with your next response.
Continuous learning allows past behavior and past inputs to influence future inputs and future behavior. In humans.
Attention over KV cache allows past behavior and past inputs to influence future inputs and future behavior. In LLMs.
Until the cache runs out, that is. But even then, you could totally use any of 9000 methods of cache compression, truncation, dropping or streaming and get away with it.
The difference between continuous learning and in-context learning seems to be in capacity, not in principle. Both are doing a similar thing, but one has more length and depth to it.
This is really semantics, but I wouldn't call attending to the KV cache re-reading the context.
The model takes in the context, encodes it into a "memory" (the KV cache), and accesses that memory later. That fact doesn't change just because the KV cache grows in size with the context.
I don't know what memory would look like other than an encode-retrieve loop.
Not the model though. The model really only takes input text and produces output text. Memory within a conversation is achieved by the harness adding the conversation (or parts of it) to the input text. The LLM itself has no memory, it’s the augmented system of several orchestrated LLM calls that does.
Right, but that's still external to the LLM, it's just a KV cache that's stored on the provider side for performance reasons, so that the client doesn't have to re-send the whole chat history with every subsequent call in the conversation.
It still generates every response using the model's pristine state with every new API call; whether the context is provided from the client or from a colocated cache server doesn't really change that.
But ... if they were sentient, sentience would just happen for the time of a session, and only when tokens are being generated.
I don't think people here are arguing that sentience would happen when the model is not running, or that sentient experience spans several sessions that do not share some kind of state?
Also, a definition of consciousness is anyway hard to imagine :)
Reinforcement learning changes the model. So it can and does change and remember based on experience. Eventually reinforcement learning can happen in real time.
But is the model aware of the training? Unless you hook the model up to an MCP server, or something similar, and have it analyze the RL changes, it will not know if it has changed or not. Even if it is real-time RL, it is not aware of the previous state.
That definition is in fact the predominant one today in serious circles: consciousness proper is not itself inclusive of the things which consider to define a continuous coherent self.
I.e. the "self" is not the same as what it means to experience consciousness.
There are for example well characterized examples of memory disruption under the influence of various drugs (e.g. as used intentionally in anesthesia); and neurological conditions which produce various kinds of amnesia.
Do these conditions mean someone is not conscious? We have the luxury of asking people directly.
More unsettling edges yet include things like so-called "split brain" patients or people suffering form serious psychological conditions like so-called "multiple personalities." Psychology does get great mileage out pathology!
First and most importantly, it's not really about LLMs, it's about AGI, and the second does not necessarily follow from the first; LLMs in their current state are pretty clearly not AGI, and most of the LLM-world progression in the last few years has been about better tooling/interfaces, refinements in training data and techniques and people learning how to use LLMs effectively rather than the huge leaps in fundamental capability that we saw in earlier years. It seems more likely that at this point, when AGI comes, it will be something entirely new or something that LLMs are only a component of, rather than "we built an LLM with ten trillion parameters and suddenly it became God".
Second, it's not even really about AGI, it's about AGI superintelligence. And more than that, it's about affordable AGI superintelligence, assuming that such a thing won't cost billions a year to operate.
> LLMs in their current state are pretty clearly not AGI
That depends a lot on definitions. It's artificial, it's very general, and by many measures it's intelligent - often superhumanly so, especially when compared to the average human.
That covers the A, the G, and the I. So why is it "clearly not AGI"?
Present LLMs are quite good at interpolating, in fact, too good.
That's the source of hallucinations. A path can be found between A and B, even if A is the 12th century Chinese royal court and B is the Easter bunny.
Interpolation and rote knowledge are still very useful. Most cognitive tasks are like this.
The thing that LLMs are not presently good at is extrapolation. You can train an LLM on pre-1904 literature, but you won't get special relativity from it, at least not without a human to prompt it in just the right way.
You can have an LLM provide a "novel math proof", but you are necessarily discarding 100 or 1,000 "novel math mistakes". The process is more like a guided walk (like the A* algorithm), with human supervision and intervention, not an autonomous math genius.
"They" are, of course, working on it. But the present implementation has some severe structural limitations (such as an inability for new or discovered information to affect model weights) that make LLMs as a human replacement incomplete.
> You can train an LLM on pre-1904 literature, but you won't get special relativity from it, at least not without a human to prompt it in just the right way.
At least 99.999% of humans aren't capable of producing special relativity either. If the bar for AGI is "must be at least as smart as Albert Einstein", one has to wonder why the deck is being stacked so unreasonably.
> LLMs as a human replacement
"Human replacement" and AGI don't seem like perfect synonyms to me.
It seems to me that "AGI" does a better job of revealing the biases of the people using the term than identifying a specific set of capabilities.
It's not just special relativity that's out of reach. It's generally difficult for an LLM to do anything novel, i.e. produce a new hypothesis from scientific data that fits no existing hypothesis, or create an algorithm with a new lower bound on runtime, or debug a proprietary system that makes unusual design assumptions.
I've found that the AI overview is usually right but confidently wrong enough of the time that I don't trust it. The interface that you get with the 'AI Mode' button (which I assume is just Gemini with very low compute settings), however, is usually pretty solid for well-documented queries.
I think this touches on the core difference between good and bad use of AI; using AI as part of the process vs cutting and pasting LLM output.
Use AI as part of the research process, to help understand a concept or problem. Use it to format data, or as a part of the design or brainstorming process. Use it to build manageable portions of code that you can read and understand before committing. But if the output doesn't go through your brain somehow before you unleash it on the world, that's really no different from a seventh-grader Googling the subject of his homework and then cutting and pasting the entire text of the first result, headers and all, and turning it in.
Was it the actual raw chain-of-thought? I know GPT-5 will emit thinking tokens, and while they're an interesting insight into the 'reasoning' process, they're apparently pretty heavily sanitized presumably because the raw thoughts could reveal proprietary training info that's part of their moat.
I think the strongest argument against AI consciousness is that there's no persistent internal state, no feedback, and no change; the 'conversation' you're having is a series of one-off API calls where each subsequent call is provided enough information about the previous calls for it to generate a plausible response.
If we (very, very generously) assume that an LLM is a structure capable of conscious thought, then it's still not conscious - we've created a representation of a brain which we turn on for a fraction of a second to generate text and then return it to the representative state. There's no opportunity to develop consciousness, it's a brain trapped in stasis.
When thinking about verifying your identity with a service, you have to ask yourself "what will be the impact to me if everything this service knows about me, every click I've made, everything I've watched/read/uploaded is posted publicly on the internet, attached to my full name, address and photo?". Because those are the very real stakes; if you verify with enough services, this will happen to you.
Weigh that against the value of using the service. A lot of times that will still probably come out in favor of using the service. Sometimes, especially given the kind of services that want age verification, the potential cost is such that you would be insane to verify.
Talking down to the LLM is anthropomorphizing it. It's misbehaving software that will not take advice or correction. Reject its bad contributions, delete its comments, ban it from the repo. If it persists, complain to or take legal action against the person who is running the software and is therefore morally and legally responsible for its actions.
Treat it just like you would someone running a script to spam your comments with garbage.
"This trajectory explains why there is no crater at Köfels. The incoming angle was very low (six degrees) and means the asteroid clipped a mountain called Gamskogel above the town of Längenfeld, 11 kilometers from Köfels, and this caused the asteroid to explode before it reached its final impact point. As it traveled down the valley it became a fireball, around five kilometers in diameter"
This doesn't even make any sense. A 1km asteroid going many kilometers a second entered at a six degree angle, tore through hundreds of miles of atmosphere without burning up or breaking up, hit a mountain causing a landslide and only then turned into a 5km fireball and traveled down the valley (at a height of ~1500 meters above the valley floor) and just sort of evaporated?
I don't think physics works the way the author of this piece thinks physics works.
It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.
Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.
reply