This book is not for experts in computers or artificial intelligence. My audience is ordinary people who are curious enough to ask questions like these, and who want to be able to make informed decisions about the course of AI’s development. It is even for those who think that all of this is rubbish - that no matter how "smart" machines get, they will never truly think, have a soul, or be self-aware in the same sense we are. Whatever your leanings, it should do no harm to explore, poke around, ask questions, and try to find out what makes machines so smart, what their inherent limitations might be, and where the boundary between human and artificial intelligence might lie. If these questions make you vaguely uncomfortable, it may be because they challenge the very foundations of all our social, legal, and religious institutions.

From Ch. 1: Artificial Intelligence - That's the Fake Kind, Right?

Why does the idea of AI stir up such emotion? Some people get nervous about AI when their egos won’t allow them to recognize any kind of intelligence except the human kind. Their idea of human dignity depends on the natural superiority of man over all other creatures (and even over some other humans). Recent movies like AI and Bicentennial Man reinforce the idea that, although we might someday be able to create machines that act human in many respects, they will always lack (and secretly long for) that intangible quality that would make them truly human.

Others might be called carbon-chauvinists. They scoff at AI because they believe that only living things made from flesh and blood can exhibit intelligence. When they look at the "smart" machines of today, they seize upon their lapses and say, "See, I told you that a mere machine can never be as smart as a living creature."

Such emotions are hard to set aside, especially since they form the basis for many of our moral and ethical beliefs. Yet they also form a barrier to rational scientific inquiry. To investigate objectively whether AI is really possible, it will be necessary to put aside the insecurities that lie at the root of such feelings and to keep an open and curious mind. Does this mean that we have to check our dignity at the door if we want to ask questions like those posed in the Preface? If we derive our dignity from a natural superiority over others, then the answer is probably Yes. But this is the kind of dignity that we lost when Copernicus displaced the Earth from the center of the universe, and when Darwin suggested that humans evolved from lower life forms. Intellectual honesty is incompatible with a need to be at the center of the universe. It also requires us to ask questions without fear of where the answers may lead us.

From Ch. 7: What is Consciousness?

Most of us hold as self-evident that inside each of us lies a kind of executive entity, or Self, that freely makes decisions, records sensory experiences, reasons through problems, feels emotions, acquires skills, sets and pursues goals, finds meaning in our existence, and retains its identity throughout our lives. We refer to this ongoing stream of awareness of ourselves and our surroundings as our consciousness.

We know from subjective experience what consciousness does, but how does it work? How do our minds produce this feeling of self-awareness, that "I" am somehow in charge of all these activities? And if we can ever understand how these different aspects of consciousness work, will we then be able to create conscious machines?

Despite centuries of speculation and debate among scientists, philosophers, and theologians, we know little more about the nature of consciousness than we did when we started. We do know that the subjective experience of being conscious is correlated with electrical activity in a certain part of the brain stem, and anesthesiologists know how to turn our conscious awareness off and on. But even scientists can’t agree about whether the roots of that experience are purely physical or lie in a separate mental world.

Aristotle thought the seat of mind and consciousness was the heart, and that the brain simply served to cool the blood. Descartes realized that the mind had something to do with the brain, perhaps the pineal gland, but he believed that the essence of thought was non-material. Most religions teach that the mind, the self, the unique spirit that makes us human, lies in a non-physical soul, which survives our mortal bodies. William James, the father of modern psychology, observed that consciousness is not a thing but a process that involves both short-term memory and "attention." This is the view held by most psychologists today.

Even so, many of us still believe in some kind of vitalism-that some unspecified and irreducible life-force, variously called our soul, spirit, or essence, powers our inner selves. Vitalism follows from Descartes’ body-mind duality, which asserts that body and mind are separate entities, one being physical and the other spiritual, occupying a reality separate from the physical world. There are even new "scientific" variations of vitalism that claim to "prove" mathematically that we will never figure out consciousness, simply because it is incapable of grasping or encompassing itself.

The trouble with vitalism and dualism, of course, is that they are scientific dead-ends. As their proponents would concede, you cannot take apart the irreducible, examine, measure, or analyze anything non-physical, or step inside someone else to find out what’s going on in there. Unfortunately, this also means that you can’t learn anything about it! You can only speculate about it or create myths about it. So if we want to get anywhere on this question What is consciousness? we have to take a different tack.

Our hypothesis here is that the mind and the process of consciousness have a purely physical basis. If they don’t, we simply can’t investigate them. This may well be true, but let’s give science a try and see how far we get.

From Ch. 9: Can Your PC Become Neurotic?

As machines become more intelligent, they will be able to adapt to changing environments and respond to new situations by designing and modifying their own programs. At that point, we will have to worry more about undesirable and unpredictable machine behavior. Does this mean that we can expect machines to experience the equivalent of nervous breakdowns and other mental aberrations? Will we ever have to deal with the likes of Marvin, the depressed robot in Hitchhiker’s Guide to the Galaxy? Or is it preposterous to suppose that a machine could become neurotic? After all, isn’t this psychological condition unique to humans?

Well, so far, yes, but autonomous, goal-seeking machines that can reprogram their own goals and sub-goals could, in effect, develop "minds of their own" and set off in unpredictable directions. If they create goals that make no sense whatsoever to us, we may see those choices as "crazy." If you think that nutty people can wreak havoc, just imagine the potential for chaos when a supercomputer that is in charge of some critical aspect of our lives gets confused about its goals and purpose in life!

From Ch. 12: The Moral Machine

Darwin wrote, "A moral being is one that is capable of comparing his past and future actions or motives, and approving or disapproving of them." We say that humans are moral beings because we have learned to do these things. Could an intelligent machine ever make moral judgments about its own actions? Is there a machine equivalent of a conscience? Where might its moral codes come from? Would they be something like human moral and ethical codes, or something else entirely?

We don’t normally think of machines as being moral or immoral-only the people who use them. But suppose that we create machines with more and more intelligence and autonomy - ones that set and pursue goals and make their own decisions about the best way to achieve them. How autonomous must a machine be before we hold it morally accountable for its actions? And mustn’t it also be conscious and morally aware in order to assess its own morality?

For a machine to reason morally and ethically - that is, to make judgments about how to behave in different environmental situations-it would first need the means to predict the likely consequences of its actions. And second, it would need ways to evaluate the goodness or desirability of those consequences.

These abilities are crudely illustrated by chess-playing machines. Using the rules of the game, you can construct a decision tree that shows consequences (that is, positions of the pieces), for every legal move by you and your opponent, and for any number of moves ahead that you care to compute. (The number of branches in the chess tree is finite but becomes enormous after only a few moves.) The value or "goodness" of any path through the tree is a function only of its ending position, whose score is based on quantifiable things like the number of pieces retained, area controlled, mobility, and protection of the king. But because computing such scores for all possible moves, or even very far down the tree, is so cumbersome for practical computers, some shortcuts (which might be called emotions) are needed. The decision tree can be "pruned" significantly by programming in some "rules of thumb," collected from human experts, thereby eliminating traditionally unfruitful branches. These shortcuts could be equated with moral rules that eliminate the need to analyze all possible consequences of every human interaction. Some smarter chess programs are even able to learn new tactics by playing more and more games. According to Darwin’s definition, then, aren’t chess machines primitive examples of moral machines?

Maybe, except that Darwin’s definition of a moral being leaves something out-it relies entirely on a person’s assessment of his own conduct. If that were so, we would need no prisons. But morality has no meaning except in the context of a social structure. Furthermore, prevailing moral codes vary from time to time and from culture to culture. In one place, killing one’s wife for committing adultery may be morally right, and consuming alcoholic beverages wrong, but somewhere else, it’s the other way around. So morality, rather than absolute, seems to have components that depend both on the observer and on the environment. It would likely be so for machines, as well. A moral machine would have to conform to its own moral rules, which, however, would be derived from prevailing standards of the community of humans and machines with which it interacts.

From Ch. 14: Will Machines Take Over?

Both [Weizenbaum and Warwick] see a world in which humans will gradually and unwittingly allow themselves to drift into a state of more and more dependence on machines. In this world, people will find themselves so overwhelmed with the complex decisions of everyday life that it will be much easier to turn many of these decisions over to machines that make them better and faster. Eventually, humans will become incapable of understanding or dealing with the details of the programs and machines that run the world. They will have forgotten the instructions they originally gave to the machines, and the machines will have found new and unexpected ways to carry them out. As the machines learn, they will even find other goals to replace the ones given them by humans. At that point, the machines will pretty much be in control.

Is this a legitimate fear, or is this just projecting human weaknesses-our lust for power, conquest, and subjugation-upon any intelligent machines that might emerge? The idea that the machines of man’s creation will assume control of critical functions and eventually arise and take over the planet is perhaps the oldest science-fiction theme. Are Warwick’s [1998] predictions any closer to fulfillment now than they were back in 1976 when Weizenbaum wrote his book? Back then, the Internet was a laboratory curiosity, personal computers had not yet been invented, a computer was a monolith that filled a large room, and robots lived almost exclusively in science fiction movies. Now it is difficult to find an aspect of our daily lives that is not utterly dependent on computers. We like to think we are still in control, that we still make all the really important decisions, and we are likely to continue thinking that. But still, doesn’t it seem like machines are doing a lot more of our thinking for us than they did in 1976?

So how could the machines take over? Would humans deliberately and willingly turn over their power to a wiser and more beneficent entity? Or will the machines get together and consciously seize power in some sort of coup? Perhaps. But more likely, the transition will be gradual and painless, even pleasurable, as we eagerly allow our machines to perform more and more tasks that take the drudgery and work out of our daily lives - while seducing us with more and more sensory gratification. They will take over simply because they do their jobs so well!

From Ch. 18: Extinction or Immortality?

To the person-on-the-street, the most troubling question about AI is: If we build machines that are smarter than we are, then what will become of human beings? Science-fiction has come up with many possible, even plausible, scenarios, ranging from extinction to immortality, with various states in between. A popular sci-fi theme has predatory war machines in charge of a post-apocalyptic Earth, hunting down and exterminating the last of the rebellious humans (Terminator). Another has humans living in harmony with many kinds of intelligent but subservient robots (Star Wars). Still others portray humans that have escaped their carbon-based vessels and exist as disembodied or communal minds in more durable containers (Star Trek). (Notice that the status quo is not considered an interesting option.) Are the next steps in human and machine evolution more likely to take us down the path to extinction or to immortality? Or is there really any difference? ...

In 1986, psychologist Robert Ornstein asserted that, for all practical purposes, our biological evolution is at an end. Our bodies and brains evolved to suit the world of 20,000 years ago, and neither has changed significantly since then. We are basically the same people who were designed to live in small groups that roamed the savannahs of East Africa and faced daily threats from wild predators. Those same bodies and brains are now trying to meet the challenges of a world that changes dramatically in a lifetime. In a process that Ornstein called conscious evolution, we both create and adapt to these changes, using technology to greatly amplify and extend the capabilities of our muscles, our senses, and our minds. So our tools are getting smarter, but we are not!

We even use technology to extend our own life-spans, by controlling disease and by replacing aging organs with borrowed human ones and even mechanical ones. Still, only a very few people live more than 100 years, and no one we know of has lived 150 years, even with mechanical organs. Will we eventually be able to extend life indefinitely by replacing more and more worn-out body parts with synthetic equivalents, until we get something like The Six-Million Dollar Man - a kind of homo cyberneticus?

But wait! Instead of tedious piecewise replacement, why not just replace a whole worn-out body with a stronger, better designed, and longer-lasting one that acts as a container for the brain? Brand new and improved limbs and sensors would be connected to the same old personality and "self" in our brains! We wouldn’t even need hearts, livers, kidneys and gall bladders - they would be replaced with more reliable electro-mechanical systems. The messy part would be preventing the natural deterioration of the brain. Do we really need that? A common sci-fi theme supposes that our human intellects-the entire electrical contents of our brains-could be "downloaded," bit by bit, into a more durable machine, allowing our minds and consciousness to far outlive our biological bodies.

Get rid of the idea of biological bodies altogether! Robotics scientist Hans Moravec believes that one day we will be able to replicate an entire human being - or for that matter, a whole community of human beings - in a computer simulation. We can’t imagine today how such downloads would work. We know, for example, that the brain is an electro-chemical system, not just an electrical one. Perhaps some kind of high-resolution neural tomography-a vastly more sophisticated kind of brain scan than present-day CAT and PET scans-could map the electro-chemical activity of the brain on a cellular level and from millisecond to millisecond. But even if we do figure all that out, would the resulting entities in any sense be human? Suppose you were downloaded into a computer. Just think for a minute about how you might spend your time. So for now, the question about this kind of immortality is not only Is it possible? but also Would we even want to? For my money, preoccupation with this sort of immortality for the benefit of a few wealthy humans would be a waste of time and scientific resources. There are just too many more fruitful ways to improve the human condition.

Another road to immortality takes more direct and conscious control of our biological evolution using genetic engineering. Once we figure out what the structure of our DNA means and how our genetic program works, we should be able to develop new cancer drugs, find ways to grow replacement organs, control aging, and discover new ways to prevent disease and birth defects. An advantage of genetic engineering is that we could simulate all these processes on a computer, before trying them out on living people, thereby reducing the chances of huge genetic blunders!

So here are two seemingly distinct paths to immortality: one where we supplement and eventually replace our organic parts -  perhaps including our brains - with mechanical ones, and the other, by which we control and even design the configuration of our own organic bodies and minds from scratch. It seems likely that these biological and mechanical paths will eventually merge, as nanotechnology matures, permitting us to manipulate and supplant our organic parts on cellular and molecular levels. Then we should be able to engineer any kind of organic modifications we wish - even design entirely new human beings!

But once we get into this business of playing God, why stop there? If we are able to design human beings, why make any more of the greedy, violent, barbaric, self-absorbed kind? Why not nicer, testosterone-free, super-human beings? Or entirely different kinds of intelligent life altogether? There is certainly no shortage of examples of desirable human attributes and powers in mankind’s history to draw upon. This is where the difference between extinction and immortality blurs.

From Ch. 19: The Enemy Within

Although our fears that intelligent machines will someday take over might be justified, I don’t believe they can do so without our complicity. The greatest threat to our dignity and our humanity will not come from machines that act like people, but from people who act like machines. The more we allow ourselves to be told what to think and to be treated like automatons by other people, by governments, corporations - and yes, by computers - the more vulnerable we become to domination by intelligent machines.

The visions of 1984, Brave New World, and other dystopian prophecies have given us chilling previews of a world in which people are transformed into zombies because they have surrendered their minds to some higher power. Orwell’s and Huxley’s warnings merely extrapolated trends that they saw in the totalitarian societies of their times.

Such trends are not new. Whenever one culture sets out to control or enslave another, the first steps are always to control information and to enforce a machine-like submission to authority. By forcing people to give up reason, and with it their humanity, conquerors can justify treating other human beings like cogs in the machinery. When we lose our courage to ask questions, when we let others do our thinking for us, when we allow our ideas to be muzzled by political correctness, we relinquish our humanity, not just to tyrants and demagogues, but to bureaucrats, to politicians, to the media, to advertising, to military paranoia, and to religious dogma.

From Ch. 22: What About God?

Since the subtitle of this book is Intelligent Machines and Human Values, I would be remiss if I failed to discuss the impact of AI on the most widespread human value on Earth-the belief in God(s). Just as the appearance of super-intelligent extraterrestrials on Earth would cause humans to rethink their insular beliefs and perspectives, the (much more likely) revelation that we have created other thinking, conscious entities would cause many to examine more closely the factual basis of certain creation stories and the moral codes that flow from them. We have seen in Chapters 11 and 12 how pervasive are the social, moral, ethical, and legal implications of machine intelligence. As AI displaces mystical explanations for how the mind works, some people will rethink deeply held notions of what it means to be human, not to mention the role that God and religion play in their lives. Will the emergence of intelligent machines force a showdown between science and theology?

The vast majority of Earth’s population follows religions that, on the surface, urge people to treat each other with dignity and respect. Religions do offer comfort and solace in the face of mysterious and frightening aspects of nature - particularly suffering and death. Evolutionary psychologists view religion as one of the tools that evolution uses to encourage altruism, which game theory has shown to promote the survival of society as a whole. It follows that religious institutions survive and thrive because, like civil law, they promote social order and a stable environment in which to raise offspring. You might therefore expect religion to provide a stabilizing force in a species that likes explanations for things, but whose individual interests often conflict.

Then what went wrong? Why is the history of religion so full of tragic conflict- persecution, holy wars, and unspeakable atrocities committed in the name of God? How do religious messages about love and altruism get twisted into prejudice, hate, violence, and religious fanaticism? Are religions merely cultural organisms competing with each other, tooth and claw, for survival? If so, will the strongest eventually oust the others, will they reach some kind of peaceful accommodation, or must we endure ever-escalating religious conflict in the future?