You wouldn’t expect narrow AI to achieve general intelligence. A program designed to win at chess, for example, would not suddenly, on its own, become an expert trial lawyer, brain surgeon, or world dictator. The same is true for self-driving car AI or any other form of “functional” AI programmed for a specific, narrow purpose.
When a program has an open-ended utility function, however, all bets are off. Enter a million times more compute power than that of narrow AI. Enter general intelligence—AI at once a neurosurgeon, astrophysicist, philosopher, stand-up comedian, school teacher, best friend, lover, horticulturist, five-star Korean BBQ chef, painter, programmer, saxophonist, barista, legislator, creator of worlds, destroyer of worlds, dog walker, cake baker, etc. Enter AI overlord scenarios.
The good news is, you may find that your open-ended overlord is restrained, to some degree, by a programmed goal of, for example, maximizing your happiness. But until you find out for sure that this is the case, proceed with caution.
[Editor’s Note: See “A Word About Salutations”]