> Wait, so the same people you think are unable to handle nuclear weapons, you would assume they would program the AI the way you want?

I certainly hope the set of people with access to nuclear weapons would not be the same people building AI, or we'd indeed be quite screwed.

> In either case AI has nothing to do with safety from nukes.

> Strong AI is not a God, it does not have the ability to do the things you are imagining. Among other problems, what makes you think those things are even possible in the first place? No matter how good the AI, it still has to follow the laws of physics.

The laws of physics would govern the computational substrate on which everything else runs; what runs on that substrate can emulate anything it likes. The major assumption I'm making is that we can successfully build strong AI (AI capable of self-improvement past human), which seems plausible but not a given.

> I certainly hope the set of people with access to nuclear weapons would not be the same people building AI, or we'd indeed be quite screwed.

Well, in fact they are, (or would be) the exact same people. But despite that we won't be "screwed".

Or did you think that if strong AI showed up the Governments of the whole world would not rush to have their own?

> what runs on that substrate can emulate anything it likes.

At what speed though? It's not clear that an emulation in a computer would be anywhere near as fast as the actual thing (i.e. a human), so it might not actually be possible to upload people digitally. (If nothing else, the speed of light limits how fast you can run a CPU.) It's also not so obvious that it's even possible to copy a brain, never mind running one.

All your assumptions are predicated on the ability to emulate a human in software, if that's not possible then none of what you wrote about strong AI will happen, instead people will continue to be people and AI will just be another thing to ignore just like we ignore all the marvels we have now.

> The major assumption I'm making is that we can successfully build strong AI (AI capable of self-improvement past human), which seems plausible but not a given.

An even larger assumption is that it's even possible to do that. Never mind can we. Can an intelligence program something smarter than itself? I don't think so. I don't think the AI can, I don't think humans can.

So far the only known method of making a smarter version of something is by random permutation and picking the best one. It's not an accident that all AI used today is by that method. I'm not aware of any that are programmed from first principles.

> Or did you think that if strong AI showed up the Governments of the whole world would not rush to have their own?

Governments can't "rush" on the same scale an AI can, and a strong AI with an appropriate value function wouldn't permit the existence of any other strong AI that doesn't share its value function (since such an AI would fall somewhere between "counterproductive" and "potential existential threat"), nor would "of their own" have any meaning unless that value function was hopelessly broken.

There shouldn't be a "second" strong AI, only a first.

> It's not clear that an emulation in a computer would be anywhere near as fast as the actual thing

Humans operate relatively slowly on the scale of computers; we're just massively parallel and have a fundamentally different architecture. That's not at all insurmountable even for human-built technology, let alone technology-built technology.

> All your assumptions are predicated on the ability to emulate a human in software

Several of my assumptions are, true, though I can imagine other possibilities that would have similar effect.

> An even larger assumption is that it's even possible to do that.

That's not a larger assumption, that's the same assumption I stated.

> Can an intelligence program something smarter than itself? I don't think so.

On what basis?

> I don't think humans can.

Humans don't have to build a system smarter than ourselves, just build one as smart as ourselves and run it a lot faster with access to a lot more data.

Your false dichotomy ignores, "people will eventually set off nukes, but can write a program that helps them reason better, which helps write a better reasoning program, which helps build a thinking program, which helps build AI that is smarter than most people, which builds an AI smarter than itself, all the while refining our innate human belief and value systems into clearly codified rules which the AI is kind enough to abide by."

It's not a dichotomy (which is an "either this or that"), it's a both. Both options can happen, and so can yours.

Or in other words AI has nothing to do with nukes since what AI does has no bearing on if nukes will be set off.