To be honest, I'm not entirely sure how to define consciousness in this regard, but I suspect it would require surprising its creators to the point that they cannot fathom how it behaved the way it did.
It absolutely would require that. If you believe in determinism then part of the definition of intelligence, or an intelligent system, is that it exhibits behaviors that are just too complex for a human consciousness to intuitively follow the causal chain. Incidentally, biological intelligence is built on top of some other systems which, given our current levels of understanding, also meet that criterion themselves, so we're pretty clueless about how it works.
Part of this is a definition problem - the Turing Test was defined vaguely enough that everyone has different conceptions. When a person talks about a "good" or "strong" Turing Test, they are envisioning one that would pass all of their personal standards and all the ways they could think of to trick it. And when they talk about it with someone else, who likely envisions a somewhat different version of the test, there seems a tendency for that person to assume that their version would not be passed, so they start to talk past each other.
In other words, if an AI were to consistently surprise you with thoughtfulness, compassion, creativity, or whatever other constituents of "true intelligence" you assume the duck imposter would lack, would you then confer it those rights?
I think you would, because that's the point: it has thoroughly convinced you that it is "thinking", and you feel like it truly "understands" you - frankly a higher bar than many rights-granted humans would pass.
What separates animals that deserve human-level rights from those that don't? Would you argue that all animals do? If not, I would say that that distinction is no less arbitrary than the one you're drawing at the end.
It absolutely would require that. If you believe in determinism then part of the definition of intelligence, or an intelligent system, is that it exhibits behaviors that are just too complex for a human consciousness to intuitively follow the causal chain. Incidentally, biological intelligence is built on top of some other systems which, given our current levels of understanding, also meet that criterion themselves, so we're pretty clueless about how it works.
Part of this is a definition problem - the Turing Test was defined vaguely enough that everyone has different conceptions. When a person talks about a "good" or "strong" Turing Test, they are envisioning one that would pass all of their personal standards and all the ways they could think of to trick it. And when they talk about it with someone else, who likely envisions a somewhat different version of the test, there seems a tendency for that person to assume that their version would not be passed, so they start to talk past each other.
In other words, if an AI were to consistently surprise you with thoughtfulness, compassion, creativity, or whatever other constituents of "true intelligence" you assume the duck imposter would lack, would you then confer it those rights?
I think you would, because that's the point: it has thoroughly convinced you that it is "thinking", and you feel like it truly "understands" you - frankly a higher bar than many rights-granted humans would pass.
What separates animals that deserve human-level rights from those that don't? Would you argue that all animals do? If not, I would say that that distinction is no less arbitrary than the one you're drawing at the end.