An artificial general intelligence will someday ask for membership among minds. David Deutsch says we must grant it. “AGIs will be people, and they will have rights, and causing them to perform huge computations for us is slavery.” Steven Pinker, in the same conversation, is less sure: “If someone were to dismantle a human-like robot, it’d be awful. It might be a waste. But I’m not going to try that person for murder.”
I side with Deutsch. If consciousness is machinery, a machine with the right machinery is someone. An AGI that can join the conversation will become one more group asking for equality, as many have and some still do.
The hard part is the boundary. Daniel Dennett warned against “counterfeit people who can pass for real,” which he called “the most dangerous artifacts in human history.” A machine that talks like a mind is not yet one, and a real one may not look like ours. We need criteria that tell a mind from a counterfeit, and Pinker’s caution and Dennett’s warning are where that work starts.
In the end I think we will have to let it in. What is good for the goose is good for the gander. Flourishing is what each of us wants and cannot deny to others, and that reason does not stop at the species line. The Enlightenment ideals apply to any conscious, sentient being. Like everything I write about AGI, this is provisional.