The argument this relates to - will machines be able to think - seems to have died out rather. Back in the day people worried about things like John Searle's Chinese Room argument as a philosophical proof that computers wouldn't be able to understand stuff. I always thought the Chinese Room thing was a bit silly and the Meat story a good rebuttal.
I guess these days computers have advanced enough that the worry is more will Facebook's AI use it's understanding of us in bad ways rather than will machines be able to understand.
As a linguist (well, person with a degree in linguistics), I think Searle makes a very strong point, and I don't think it's at all fair to call it rebutted or disproven.
The core part of the room is a book where the operator looks up symbols coming in and is able to mindlessly copy the answer to the output. Such a book (or whatever book-like device, i.e. a list of inputs with corresponding outputs) that allows a person to interact with someone in a language they don't speak is an absurdity, it cannot exist. Human language in use does not work like that. But the notion is not far from the bone-headed simplicity of early AI and machine translation efforts.
> allows a person to interact with someone in a language they don't speak
Here is the flaw in your reasoning. The human who is following the directions in the book certainly isn't the consciousness we need to be thinking about any more than the mitochondria in our brains are conscious. In Searle's analogy, the human in that room is simply the power source. The consciousness is in the state of the room.
Searle's analogy also seems to hide the importance of state. The agent just looking up symbols in a book, and that page tells them what to do with the next symbol is essentially a combinational circuit, a static function. If those are the rules then I'd agree there is no consciousness there. But I'd also say it is impossible for a computer to translate C++ source code to machine language ... if it cannot retain state.
If Searle's room has enough rule books and means for storing enough state, then I'd say consciousness would be possible.
The best reverse argument is that if human consciousness is not the result of a naturalistic, physical process and requires some extra-physical process to make it all work, at some point the laws of physics must be broken which causes the purely physical brain to do something other than what the laws of physic dictate it should do. Where are the violations of physics?
Penrose waves his hands and says quantum physics, but that is not an explanation, just an obscuring smoke screen. How does the addition of apparently random events make something more likely to be conscious? Are there sufficient degrees of freedom for this extra-physical conscious agent to manipulate quantum fluctuations to force the physical outcome it desires in a consistent manner? If it is theoretically possible, is it practically possible to compute what quantum butterflies should flap their wings?
The lasting value in the Chinese room thought experiment is making argument explicit. It helps to clarify the problem. Searle makes two conclusions. 1) computers have no understanding of meaning or semantics and 2) human minds are not computational information processing systems.
Most modern criticism of these two conclusions point out that Seale's argument relies on strong intuition about what understanding and meaning means and he can't clarify them satisfactorily.
Chinese room being conscious is counter-intuitive. Today most philosophers of consciousness are more aware how easy it is to make 'slips in reasoning' when dealing with the counter-intuitiveness of the consciousness arguments.
Ah, well, yes, the C-word is pre-scientific term and those are always hard to deal with. But I think "understanding natural language" is one of the easier ones, at least inasmuch as we can resort to referring to the intentions of the speaker, and understanding being the congruence between that and the hearer's impression of it. And it just seems like a category error to say that any natural language processing system has anything like that, since the thing doing the understanding is so radically different to the one doing the speaking.
> any natural language processing system has anything like that
Chinese room that can respond verbally can incorporate more than natural language model. It can have spatiotemporal understanding as well or any other model that can be represented in Turing machine.
> Such a book (or whatever book-like device, i.e. a list of inputs with corresponding outputs) that allows a person to interact with someone in a language they don't speak is an absurdity, it cannot exist.
And that, right there, is a sufficient rebuttal to Searle's argument, as many critics have pointed out. The premise of Searle's argument is that such a setup could pass the Turing test, which is absurd. So his argument fails because it is based on an absurd premise.
> the notion is not far from the bone-headed simplicity of early AI and machine translation efforts.
That's a valid criticism of early AI and machine translation efforts, but Searle did not put forward his argument just as a criticism of those efforts. He put forward his argument as a claim that no efforts at AI that used digital computers, ever, could produce anything that was conscious the way humans are conscious. But, as above, his argument was based on an absurd premise, and, what's more, an attempt at building a conscious entity with digital computers does not have to satisfy that premise.
The room is not just an input-output system. It also has to have state.
Without state the whole argument falls apart.
> But the notion is not far from the bone-headed simplicity of early AI and machine translation efforts.
As someone who studied Machine Learning 20 years ago, could you be a bit more respectful please?
First, pretty much nobody (working on serious ML) in that era thought they were going to build anything actually intelligent, conscious, or thinking.
Second, in this supposed era of boneheaded simplicity, a LOT of foundational stuff we use in today's AI was discovered and written.
I'm not sure how familiar you are with AI history, to name an example, we already were working on neural nets and discovered error backpropagation in the 70s/80s. Then came a long period during which neural nets somehow didn't really perform very well, and ML was looking at completely different types of classification algorithms (what's happening with SVMs today?).
If we had stopped there, you might have lumped in the early perceptron with the "boneheaded simplicity".
But instead we developed batch normalisation and a bunch of other techniques, and now neural nets are state of the art.
> As a linguist (well, person with a degree in linguistics), I think Searle makes a very strong point, and I don't think it's at all fair to call it rebutted or disproven.
I think you are about to disprove this point.
> The core part of the room is a book where the operator looks up symbols coming in and is able to mindlessly copy the answer to the output. Such a book (or whatever book-like device, i.e. a list of inputs with corresponding outputs) that allows a person to interact with someone in a language they don't speak is an absurdity, it cannot exist.
Yup, that's both true and a pretty convincing rebuttal of the Chinese
Room.
> But the notion is not far from the bone-headed simplicity of early AI and machine translation efforts.
So, what? No one, anywhere claimed that those early efforts were self-aware or had understanding, nor are they what the Chinese Room is even notionally directed at: it was directed at the limit of possibility, not the then-current implementations.
I honestly don't understand how that's a rebuttal of the Chinese room. Obviously a person with such a fantastic device would not "understand Chinese" if they had it.
Consider a more realistic version of Chinese room. Bob gets a slip of paper, with written Chinese on it, and slips it into a black-box "Chinese room", which produces another paper with a perfectly written Chinese answer. Bob hands it back - it is pretty obvious that Bob doesn't need to have an inkling of Chinese.
Except, in this version, there's an actual Chinese speaker, Xin, sitting inside the room. It's clear that it's Xin, not Bob, who understands Chinese.
Now let's move back to the original Chinese room argument. It's clear that it's the room that understands Chinese - or, rather, the whole system composed of the rules (the room) and its executor (the person), but not the executor by itself. It only seems absurd because our real-life experiences make us presume that, when there's a room and a person, the person must be the more sentient part. IMHO the whole argument is a philosophical sleight of hand.
However, when confronted with this "the system does understand" argument, I believe that Searle and his defenders fall back on the lack-of-qualia position. That is, there is no entity that would experience anything, and that therefore the ability to demonstrate "understanding" as the room does isn't a sufficient (or perhaps even necessary) condition of consciousness.
This point is trickier to rebut, because I think it's fair to acknowledge that our ability to imagine where there would be qualia in the case of the chinese room is more than a little limited. I think the only fair answers are either:
1. our imagination is too limited, but that doesn't allow us to conclude that there cannot be qualia
2. there are indeed no qualia associated with the Chinese Room, because consciousness is not required for full language processing.
The problem with the lack of qualia argument is that qualia has no operationalizable, testable definition; it's something we have a fuzzy idea of based on individual experience and attribute to other entities based on similarity of behavior, but can't really with any authority say does or does not exist in anything (other than that an entity itself that experiences qualia can attribute it to themselves.)
I mean this is the limit of current state of the art AI, right?
We're seeing with things like GPT3 that an actual powerful language model needs to have a load of real world knowledge built in. But that knowledge is all based on experiences by others. From the way it talks (and generates poetry) you can tell that it is unable to synthesize new experiences. It can't come up with a description (or metaphor) of human experience that it hasn't been taught.
To be a convincing AI, you don't just need to replicate descriptions of experience from memory, but actually have new experiences as well.
Meaning that the Chinese room, in order to be able to demonstrate "understanding", will also require other inputs than just text. Otherwise it can't really understand the world, because it only knows about the world from hear-say.
I’ve not heard this argument before. Isn’t saying “there is no entity that would experience anything” begging the question? If the system is capable of understanding, why wouldn’t it also be capable of experience? That is what we see in animals.
No, this is the very heart of Searle's thought experiment. His point was that you could imagine a rule-based system for language translation and conversation that clearly was not conscious. The argument wasn't about whether or not it was possible to build such a system, but about whether or not such a system must necessarily be conscious.
Searle's claim was that it was clear from his thought experiment that such a system could exist without any understanding, and definitely without consciousness. Others have disagreed with his reasoning and his conclusion.
I must have misunderstood your earlier comment then. I thought you were saying that Searle responded to ”the system does understand” argument by saying “OK, but it’s still not conscious.” Now I think you are saying his response was “no, it doesn’t understand, it merely passes the Turing test; it can’t understand because it’s clearly not conscious.” Which again, seems to beg the question.
No, Searle's main response to the system response was to just dismiss it out of hand (at least in the responses from him that I read). Dennett was probably (for me) the most articulate of the "system responders" and I think Searle considered his POV to be a joke, more or less.
The Chinese Room is an argument that a Turing-like test is inadequate to prove understanding of a language because a combination of two things, neither of which have understanding, could conceivably pass it for every test scenario.
The fact that one of those two things is impossible (or, alternatively, is equivalent to a device encoding a full understanding of the language) negates the argument.
I see your point. It's been a very long time since I've read the original text, but I don't remember this being my impression of the argument being made, and that would indeed be a very simplistic one. But I might be mixing it up with some of the debates on semantics within linguistics, which is of course a slightly different topic.
If such a device could exist, it would be the part which understood. If the questions were about facts and the person looked up the facts in a book, it wouldn't be the room as a whole that knew the facts. The room as a whole delivers facts, but there's a fairly obvious separation of responsibilities in the implementation...
> The room as a whole delivers facts, but there's a fairly obvious separation of responsibilities in the implementation...
Sure, but I think that's always the case. When you speak your entire body (at least head/neck/chest, but also cardiovascular system) are involved in "delivering" information but your tongue (and for that matter most organs involved) doesn't "know" the information. We still say that "you" as a person know something.
When a computer displays a result of some (cpu) calculation on the monitor, the gpu making electrical signals for the input of the monitor has no meaningful knowledge about the calculation, etc, but we still say "the computer" made the calculation.
And they shouldn't have. That's a perfectly reasonable argument: an organism's behavior (movement of the eyes and facial gestures) give clues about its internal state and potential conscious experiences.
Well, it seems to sort of kill his argument because he's saying you can detect consciousness via a visible tell, but a sufficiently advanced dog-robot with expressive eyes would trick him (there's a reason I work in machine learning)
And for what it’s worth, looking into the eyes of an intelligent dog like a border collie gives an unnerving sensation that it’s calculated the next 20 moves and knows how to win.
err, no? The sufficiently advanced dog would be able to fool a human into thinking it was a real dog, but at least according to Searle: """The Chinese room argument holds that a digital computer executing a program cannot be shown to have a "mind", "understanding" or "consciousness",[a] regardless of how intelligently or human-like the program may make the computer behave"""
He undermines his whole argument by implying he can tell his dog is conscious simply by visualizing its eyes.
Surely this Chinese room digital computer is capable of controlling a bunch of servos attached to glass eyeballs[0] at a speed and fidelity that is indistinguishable from actual puppy eyes.
It's honestly quite presumptuous that Searle could imagine a computer powerful enough to at least appear to be simulating consciousness ... but surely there's no WAY that animatronics will ever become advanced enough to fool him!! lol
We literally already have the technology to do this.
[0] indeed, in this experiment you are not allowed to poke at them to see if they're real ...
Your argument is based on the proposition that a dog machine able to fool him can be built. There is no proof for that yet, and having it for an axiom is quite far fetched.
I agree there is no existential proof for dog machines that can fool humans yet. But it seems entirely within reason that within the next 50 years, both mechanical robots and their intelligences will improve to the point where they pass every visible test. Certainly chatbots and deepfakes are reaching a point where it's hard for non-experts to tell.
Isn’t Searle’s argument based on the equally far-fetched axiom that a program could be written which (a) passes the Turing test, and (b) could be executed by a human using pencil and paper, with sufficient speed to not fail the Turing test?
No, obviously not because that's just absurd. But one could ask if the fact of a human with pencil and paper being dead slow computing the output is enough to invalidate the whole argument.
But anyway, I am not trying to defend Searle’s argument. I was just pointing out something what seemed to me like a flaw in that particular refutation.
I started writing and didn't finish a chat bot one time that meats the criteria. (Actually 2 chat bots)
The formula is this: 1) The 2 bots have a scripted conversation. 2) When a human tries to participate (read: interrupts) they have a conversation like "Bot1: who is this guy? Do you know him? Bot2: NO". Then they happily continue their chat completely ignoring the human unless he types their username. Then it goes "What do you want?" and "Could you please stop highlighting me" etc
I read Searle's argument as boiling down to: you cannot tell the difference between a "truly conscious" (in terms of experiencing the subjective experience of consciousness), and a machine that emulates behavior well enough to fool people into believing that it is conscious.
Many people (like me) believe that in the process of making such a fake, it's entirely possible (and even likely) that a mind would emerge as a spontaneous outcome.
It's not so absurd. How do we know solipsism isn't true? Because of two reasons, (1) the physiology of others is similar to our physiology, and (2) we can see evidence of inner state reflected in behaviour.
For dogs we can make a similar argument using (1) and (2). It is slightly weaker than applying it to other humans but not by much. Searle's comment pertains to (2).
I think Searle is even more relevant today, with GPT-3.
It's getting really hard to distinguish a GPT-3 text from actual human generated text.
But we also know how GPT-3 generates those texts is absolutely nothing like how a human generates a text. GPT-3 seems more like the man in the Chinese Room, in how it writes, than a human being.
Or is it? Are we really just a deep neural network implemented in meat?
Or is there something fundamentally different between how we experience the world, and how GPT-3 experiences the world?
> It's getting really hard to distinguish a GPT-3 text from actual human generated text.
No, it isn't.
What is getting really hard is to distinguish a GPT-3 text from actual human generated text without any context.
Plop GPT-3 down in a room full of humans having a conversation (or, under current conditions, put it in a Zoom call with humans having a conversation) and ask it to generate text as if it were another human participating in the conversation, and it will quickly become obvious that it is not a human, since it will fail miserably. The text it generates will be "human-like", but will have no relationship with what the humans are talking about.
I've actually done this experiment by putting a GPT-3 bot in a Telegram group. Its replies were mostly stuck in an uncanny valley where they sort of made sense, but often either lacked detail or seemed to very slightly misunderstand what the topic of discussion was. This might have been just because I didn't include enough context in the prompt, however. I have some plans for improving the prompting strategy, so we'll see.
I actually recently wrote a post on the topic of whether GPT-3 can be said to understand anything[1]. The argument is a bit too long to summarize here, but I don't think what GPT-3 is doing is as fundamentally different from what human brains do as people seem to think.
> I'm not sure "failed miserably" is a fair description either.
Interesting, thanks for the link!
Particularly towards the end, this looks similar to transcripts of conversations with ELIZA. I suppose "failed miserably" is indeed an overstatement, given that comparison, since ELIZA, IIRC, actually did fool some psychologists into thinking it was an actual paranoid human being.
The fundamental difference is that GPT-3 can only read text. Humans can see, hear, feel, taste, move, and socialise. Both systems learn by ingesting data, but human brains get a lot more input of a much wider variety over a much longer time.
Is this about that? I always thought it was more an exercise in pointing out that other life in the universe may not be recognizable as such. It's hard to imagine such a thing directly, but the story gets us to the border of it by flipping the tables and presenting Earth Life as unbelievable to an undefined alien POV.
I guess these days computers have advanced enough that the worry is more will Facebook's AI use it's understanding of us in bad ways rather than will machines be able to understand.