Our tendency to summon powers we cannot control stems not from individual psychology, but from the unique way our species cooperates in large numbers. The main argument of this book is that humankind gains enormous power by building large networks of cooperation, but the way these networks are built predisposes us to use that power unwisely. Our problem, then, is a network problem…
While each individual human is typically interested in knowing the truth about themselves and the world, large networks bind members and create order by relying on fictions and fantasies. That’s how we got, for example, to Nazis and Stalinism. These were exceptionally powerful networks, held together by exceptionally deluded ideas. As George Orwell famously put it, ignorance is strength.
This begs the question, then, whether an inherent flaw in our makeup — a dark side of our psyche that is exposed and expressed when networks reach sufficient mass — must always be inherited by the tools we create. Since every tool we have thus far created is in some way an extension of the human body and mind, is it even possible to create an information tool — artificial intelligence — that is not infected with our weaknesses and failings?