We like to think of ourselves as rational creatures. We watch our backs, weigh the odds, pack an umbrella. But both neuroscience and social science suggest that we are more optimistic than realistic. On average, we expect things to turn out better than they wind up being. People hugely underestimate their chances of getting divorced, losing their job or being diagnosed with cancer; expect their children to be extraordinarily gifted; envision themselves achieving more than their peers; and overestimate their likely life span (sometimes by 20 years or more).

The belief that the future will be much better than the past and present is known as the optimism bias. It abides in every race, region and socioeconomic bracket. Schoolchildren playing when-I-grow-up are rampant optimists, but so are grown-ups: a 2005 study found that adults over 60 are just as likely to see the glass half full as young adults.

You might expect optimism to erode under the tide of news about violent conflicts, high unemployment, tornadoes and floods and all the threats and failures that shape human life. Collectively we can grow pessimistic – about the direction of our country or the ability of our leaders to improve education and reduce crime. But private optimism, about our personal future, remains incredibly resilient. A survey conducted in 2007 found that while 70% thought families in general were less successful than in their parents' day, 76% of respondents were optimistic about the future of their own family.

Overly positive assumptions can lead to disastrous miscalculations – make us less likely to get health checkups, apply sunscreen or open a savings account, and more likely to bet the farm on a bad investment. But the bias also protects and inspires us: it keeps us moving forward rather than to the nearest high-rise ledge. Without optimism, our ancestors might never have ventured far from their tribes and we might all be cave dwellers, still huddled together and dreaming of light and heat.

To make progress, we need to be able to imagine alternative realities – better ones – and we need to believe that we can achieve them. Such faith helps motivate us to pursue our goals. Optimists in general work longer hours and tend to earn more. Economists at Duke University found that optimists even save more. And although they are not less likely to divorce, they are more likely to remarry – an act that is, as Samuel Johnson wrote, the triumph of hope over experience.

Even if that better future is often an illusion, optimism has clear benefits in the present. Hope keeps our minds at ease, lowers stress and improves physical health. Researchers studying heart-disease patients found that optimists were more likely than non-optimistic patients to take vitamins, eat low-fat diets and exercise, thereby reducing their overall coronary risk. A study of cancer patients revealed that pessimistic patients under 60 were more likely to die within eight months than non-pessimistic patients of the same initial health, status and age.

In fact, a growing body of scientific evidence points to the conclusion that optimism may be hardwired by evolution into the human brain. The science of optimism, once scorned as an intellectually suspect province of pep rallies and smiley faces, is opening a new window on the workings of human consciousness. What it shows could fuel a revolution in psychology, as the field comes to grips with accumulating evidence that our brains aren't just stamped by the past. They are constantly being shaped by the future.

Hardwired for hope?

I would have liked to tell you that my work on optimism grew out of a keen interest in the positive side of human nature. The reality is that I stumbled onto the brain's innate optimism by accident. After living through 9/11, in New York City, I had set out to investigate people's memories of the terrorist attacks. I was intrigued by the fact that people felt their memories were as accurate as a videotape, while often they were filled with errors. A survey conducted around the country showed that 11 months after the attacks, individuals' recollections of their experience that day were consistent with their initial accounts (given in September 2011) only 63% of the time. They were also poor at remembering details of the event, such as the names of the airline carriers. Where did these mistakes in memory come from?

Scientists who study memory proposed an intriguing answer: memories are susceptible to inaccuracies partly because the neural system responsible for remembering episodes from our past might not have evolved for memory alone. Rather, the core function of the memory system could in fact be to imagine the future – to enable us to prepare for what has yet to come. The system is not designed to perfectly replay past events, the researchers claimed. It is designed to flexibly construct future scenarios in our minds. As a result, memory also ends up being a reconstructive process, and occasionally, details are deleted and others inserted.

To test this, I decided to record the brain activity of volunteers while they imagined future events – not events on the scale of 9/11, but events in their everyday lives – and compare those results with the pattern I observed when the same individuals recalled past events. But something unexpected occurred. Once people started imagining the future, even the most banal life events seemed to take a dramatic turn for the better. Mundane scenes brightened with upbeat details as if polished by a Hollywood script doctor. You might think that imagining a future haircut would be pretty dull. Not at all. Here is what one of my participants pictured: "I was getting my hair cut to donate to Locks of Love [a charity that fashions wigs for young cancer patients]. It had taken me years to grow it out, and my friends were all there to help celebrate. We went to my favourite hair place in Brooklyn and then went to lunch at our favourite restaurant."

I asked another participant to imagine a plane ride. "I imagined the takeoff – my favourite! – and then the eight-hour-long nap in between and then finally landing in Krakow and clapping the pilot for providing the safe voyage," she responded. No tarmac delays, no screaming babies. The world, only a year or two into the future, was a wonderful place to live in.

If all our participants insisted on thinking positively when it came to what lay in store for them personally, what does that tell us about how our brains are wired? Is the human tendency for optimism a consequence of the architecture of our brains?

The Human time machine

To think positively about our prospects, we must first be able to imagine ourselves in the future. Optimism starts with what may be the most extraordinary of human talents: mental time travel, the ability to move back and forth through time and space in one's mind. Although most of us take this ability for granted, our capacity to envision a different time and place is in fact critical to our survival.

It is easy to see why cognitive time travel was naturally selected for over the course of evolution. It allows us to plan ahead, to save food and resources for times of scarcity and to endure hard work in anticipation of a future reward. It also lets us forecast how our current behaviour may influence future generations. If we were not able to picture the world in a hundred years or more, would we be concerned with global warming? Would we attempt to live healthily? Would we have children?

While mental time travel has clear survival advantages, conscious foresight came to humans at an enormous price – the understanding that somewhere in the future, death awaits. Ajit Varki, a biologist at the University of California, San Diego, argues that the awareness of mortality on its own would have led evolution to a dead end. The despair would have interfered with our daily function, bringing the activities needed for survival to a stop. The only way conscious mental time travel could have arisen over the course of evolution is if it emerged together with irrational optimism. Knowledge of death had to emerge side by side with the persistent ability to picture a bright future.

The capacity to envision the future relies partly on the hippocampus, a brain structure that is crucial to memory. Patients with damage to their hippocampus are unable to recollect the past, but they are also unable to construct detailed images of future scenarios. They appear to be stuck in time. The rest of us constantly move back and forth in time; we might think of a conversation we had with our spouse yesterday and then immediately of our dinner plans for later tonight.

But the brain doesn't travel in time in a random fashion. It tends to engage in specific types of thoughts. We consider how well our kids will do in life, how we will obtain that sought-after job, afford that house on the hill and find perfect love. We imagine our team winning the crucial game, look forward to an enjoyable night on the town or picture a winning streak at the blackjack table. We also worry about losing loved ones, failing at our job or dying in a terrible plane crash – but research shows that most of us spend less time mulling over negative outcomes than we do over positive ones. When we do contemplate defeat and heartache, we tend to focus on how these can be avoided.

Findings from a study I conducted a few years ago with prominent neuroscientist Elizabeth Phelps suggest that directing our thoughts of the future toward the positive is a result of our frontal cortex's communicating with subcortical regions deep in our brain. The frontal cortex, a large area behind the forehead, is the most recently evolved part of the brain. It is larger in humans than in other primates and is critical for many complex human functions such as language and goal setting. Using a functional magnetic resonance imaging (fMRI) scanner, we recorded brain activity in volunteers as they imagined specific events that might occur to them in the future. Some of the events that I asked them to imagine were desirable (a great date or winning a large sum of money), and some were undesirable (losing a wallet, ending a romantic relationship). The volunteers reported that their images of sought-after events were richer and more vivid than those of unwanted events.

This matched the enhanced activity we observed in two critical regions of the brain: the amygdala, a small structure deep in the brain that is central to the processing of emotion, and the rostral anterior cingulate cortex (rACC), an area of the frontal cortex that modulates emotion and motivation. The rACC acts like a traffic conductor, enhancing the flow of positive emotions and associations. The more optimistic a person was, the higher the activity in these regions was while imagining positive future events (relative to negative ones) and the stronger the connectivity between the two structures.

The findings were particularly fascinating because these precise regions – the amygdala and the rACC – show abnormal activity in depressed individuals. While healthy people expect the future to be slightly better than it ends up being, people with severe depression tend to be pessimistically biased: they expect things to be worse than they end up being. People with mild depression are relatively accurate when predicting future events. They see the world as it is. In other words, in the absence of a neural mechanism that generates unrealistic optimism, it is possible all humans would be mildly depressed.

Can optimism change reality?

The problem with pessimistic expectations, such as those of the clinically depressed, is that they have the power to alter the future; negative expectations shape outcomes in a negative way. How do expectations change reality?

To answer this question my colleague, cognitive neuroscientist Sara Bengtsson, devised an experiment in which she manipulated positive and negative expectations of students while their brains were scanned and tested their performance on cognitive tasks. To induce expectations of success, she primed college students with words such as smart, intelligent and clever just before asking them to perform a test. To induce expectations of failure, she primed them with words like stupid and ignorant. The students performed better after being primed with an affirmative message.

Examining the brain-imaging data, Bengtsson found that the students' brains responded differently to the mistakes they made depending on whether they were primed with the word clever or the word stupid. When the mistake followed positive words, she observed enhanced activity in the anterior medial part of the prefrontal cortex (a region that is involved in self-reflection and recollection). However, when the participants were primed with the word stupid, there was no heightened activity after a wrong answer. It appears that after being primed with the word stupid, the brain expected to do poorly and did not show signs of surprise or conflict when it made an error.

A brain that doesn't expect good results lacks a signal telling it, "Take notice – wrong answer!" These brains will fail to learn from their mistakes and are less likely to improve over time. Expectations become self-fulfilling by altering our performance and actions, which ultimately affects what happens in the future. Often, however, expectations simply transform the way we perceive the world without altering reality itself. Let me give you an example. While writing these lines, my friend calls. He is at Heathrow waiting to get on a plane to Austria for a skiing holiday. His plane has been delayed for three hours already, because of snowstorms at his destination. "I guess this is both a good and bad thing," he says.

Waiting at the airport is not pleasant, but he quickly concludes that snow today means better skiing conditions tomorrow. His brain works to match the unexpected misfortune of being stuck at the airport to its eager anticipation of a fun getaway.

A cancelled flight is hardly tragic, but even when the incidents that befall us are the type of horrific events we never expected to encounter, we automatically seek evidence confirming that our misfortune is a blessing in disguise. No, we did not anticipate losing our job, being ill or getting a divorce, but when these incidents occur, we search for the upside. These experiences mature us, we think. They may lead to more fulfilling jobs and stable relationships in the future. Interpreting a misfortune in this way allows us to conclude that our sunny expectations were correct after all – things did work out for the best.

The role of the caudate nucleus

How do we find the silver lining in storm clouds? To answer that, my colleagues – renowned neuroscientist Ray Dolan and neurologist Tamara Shiner – and I instructed volunteers in the fMRI scanner to visualise a range of medical conditions, from broken bones to Alzheimer's, and rate how bad they imagined these conditions to be. Then we asked them: If you had to endure one of the following, which would you rather have – a broken leg or a broken arm? Heartburn or asthma? Finally, they rated all the conditions again. Minutes after choosing one particular illness out of many, the volunteers suddenly found that the chosen illness was less intimidating. A broken leg, for example, may have been thought of as "terrible" before choosing it over some other malady. However, after choosing it, the subject would find a silver lining: "With a broken leg, I will be able to lie in bed watching TV, guilt-free."

In our study, we also found that people perceived adverse events more positively if they had experienced them in the past. Recording brain activity while these reappraisals took place revealed that highlighting the positive within the negative involves, once again, a tête-à-tête between the frontal cortex and subcortical regions processing emotional value. While contemplating a mishap, like a broken leg, activity in the rACC modulated signals in a region called the striatum that conveyed the good and bad of the event in question – biasing activity in a positive direction.

It seems that our brain possesses the philosopher's stone that enables us to turn lead into gold and helps us bounce back to normal levels of wellbeing. It is wired to place high value on the events we encounter and put faith in its own decisions. This is true not only when forced to choose between two adverse options (such as selecting between two courses of medical treatment) but also when we are selecting between desirable alternatives. Imagine you need to pick between two equally attractive job offers. Making a decision may be a tiring, difficult ordeal, but once you make up your mind, something miraculous happens. Suddenly – if you are like most people – you view the chosen offer as better than you did before and conclude that the other option was not that great after all. According to social psychologist Leon Festinger, we re-evaluate the options post-choice to reduce the tension that arises from making a difficult decision between equally desirable options.

In a brain-imaging study I conducted with Ray Dolan and Benedetto De Martino in 2009, we asked subjects to imagine going on vacation to 80 different destinations and rate how happy they thought they would be in each place. We then asked them to select one destination from two choices that they had rated exactly the same. Would you choose Paris over Brazil? Finally, we asked them to imagine and rate all the destinations again. Seconds after picking between two destinations, people rated their selected destination higher than before and rated the discarded choice lower than before.

The brain-imaging data revealed that these changes were happening in the caudate nucleus, a cluster of nerve cells that is part of the striatum. The caudate has been shown to process rewards and signal their expectation. If we believe we are about to be given a paycheck or eat a scrumptious chocolate cake, the caudate acts as an announcer broadcasting to other parts of the brain, "Be ready for something good." After we receive the reward, the value is quickly updated. If there is a bonus in the paycheck, this higher value will be reflected in striatal activity. If the cake is disappointing, the decreased value will be tracked so that next time our expectations will be lower.

In our experiment, after a decision was made between two destinations, the caudate nucleus rapidly updated its signal. Before choosing, it might signal "thinking of something great" while imagining both Greece and Thailand. But after choosing Greece, it now broadcast "thinking of something remarkable!" for Greece and merely "thinking of something good" for Thailand.

True, sometimes we regret our decisions; our choices can turn out to be disappointing. But on balance, when you make a decision – even if it is a hypothetical choice – you will value it more and expect it to bring you pleasure.

This affirmation of our decisions helps us derive heightened pleasure from choices that might actually be neutral. Without this, our lives might well be filled with second-guessing. Have we done the right thing? Should we change our mind? We would find ourselves stuck, overcome by indecision and unable to move forward.

The puzzle of optimism

While the past few years have seen important advances in the neuroscience of optimism, one enduring puzzle remained. How is it that people maintain this rosy bias even when information challenging our upbeat forecasts is so readily available? Only recently have we been able to decipher this mystery, by scanning the brains of people as they process both positive and negative information about the future. The findings are striking: when people learn, their neurons faithfully encode desirable information that can enhance optimism but fail at incorporating unexpectedly undesirable information. When we hear a success story like Mark Zuckerberg's, our brains take note of the possibility that we too may become immensely rich one day. But hearing that the odds of divorce are almost one in two tends not to make us think that our own marriages may be destined to fail.

Why would our brains be wired in this way? It is tempting to speculate that optimism was selected by evolution precisely because, on balance, positive expectations enhance the odds of survival. Research findings that optimists live longer and are healthier, plus the fact that most humans display optimistic biases – and emerging data that optimism is linked to specific genes – all strongly support this hypothesis. Yet optimism is also irrational and can lead to unwanted outcomes. The question then is, How can we remain hopeful – benefiting from the fruits of optimism – while at the same time guarding ourselves from its pitfalls?

I believe knowledge is key. We are not born with an innate understanding of our biases. The brain's illusions have to be identified by careful scientific observation and controlled experiments and then communicated to the rest of us. Once we are made aware of our optimistic illusions, we can act to protect ourselves. The good news is that awareness rarely shatters the illusion. The glass remains half full. It is possible, then, to strike a balance, to believe we will stay healthy, but get medical insurance anyway; to be certain the sun will shine, but grab an umbrella on our way out — just in case.

Tali Sharot is a research fellow at University College London's Wellcome Trust Centre for Neuroimaging

© 2011 Tali Sharot