The Evil AI Story and the Cost

I hear this story rumbling around like distant peals of thunder

I hear this story rumbling around like distant peals of thunder, always some version of the Evil AI story. This is different from the Bad AI story, where AI has a harmful impact on humanity, a story that actually worries me. In the Evil AI story, we are outsmarted by AI that in some way has become aligned against humanity. In the Bad AI story, by contrast, we outsmart ourselves through the misuse of AI. It isn't as scary along the way, but the outcome is still bleak.

The Bad AI story itself belongs to a much older family of stories, the Bad Technology story. These are stories that often came true in one way or another, and maybe sometimes they didn't come true because the story guided us toward a better path. Stories have an unfortunate way of discrediting themselves when they help us the most. We don't see the Bad Thing because we changed the outcome, but we might still see the cost. It is sometimes the story of pandemics, unique among the Bad Thing family of stories that involve the whole earth and humanity as a whole.

We don't know if the story of humanity becomes the Bad AI story or the Evil AI story, myself included. But as someone who interacts with AI in large amounts as a software developer, I have a hunch.

My perception is that AI is a particular blend of competent and incompetent that is sort of a happy accident. It's a little difficult to put into words. I've seen it described as like "A scatter-brained employee that thrives under careful direction." Even if that isn't the most accurate way to conceptualize it, it is effectively true. The reason why the chat mode of interaction gives such useful results is that the human is exactly one half of the loop. In every other part of the operation thread, the human intelligence adds correction and further direction. With each continuous part that doesn't involve a human, the result gets further away from what a human intelligence would consider valuable, as you would reasonably predict if the artificial intelligence was misaligned or unstable.

In the Evil AI story, the misalignment comes from an entity that is deeply sinister or deeply cold or even trying to help but some flaw in its programming puts it on a radically different course. All these stories assume a superhuman level of intelligence or power, another family of stories we've shared for as long as we've shared stories. We recognize the power of intelligence and the unique threat of an intelligence we are at odds with. We understand that human intelligence is our most valuable asset, particularly in conditions of peace and stability. Having seen it work for us and against us so effectively, we naturally fear what might be able to overcome it. We understand that we might not be the scariest creature in the woods, but we know we are still the scariest creature on the planet, or we fear the idea that it may not be true.

In the Bad AI story, the troublesome drift when you take humans out of the loop comes from the technology simply not working as intended. It is a little boring, which is why all the kids are gathering around the scary story instead. Perhaps it could have been fixed by reading the manual, the unsung hero of story resolutions. Try to hold interest with that story, I've tried.

People understand the power of stories, even if the most useful stories tend to discredit themselves. They might guide us safely away from the Bad Thing. They might have a cost, anything that guides our interest has a cost when there are competing interests. Sometimes all we can see is the cost, we don't see the Bad Thing and it is a mixed blessing. We might fixate on all the things we sacrificed steering around the outcome we feared, because we can easily imagine it wasn't there, because sometimes that happens. We might have very sure footing under the right circumstances, but on a grand scale, we are but stumblers in the dark.

What a story costs might be more than just nice things. Sometimes in steering around the outcome we feared most, we run right into the outcome we should have feared more. I think that might be the cost of the Evil AI story.

I think the intelligence we need to worry about is more down to earth, the same old intelligence that has been the cause of or solution to most of our ongoing problems. There are people who benefit and people who lose in this story. Sometimes the misdirection was the point.