In an unremarkable business park outside the city of Ann Arbor, in Michigan, stands a poignant memorial to humanity's shattered dreams. It doesn't look like that from the outside, though. Even when you get inside – which members of the public rarely do – it takes a few moments for your eyes to adjust to what you're seeing. It appears to be a vast and haphazardly organised supermarket; along every aisle, grey metal shelves are crammed with thousands of packages of food and household products. There is something unusually cacophonous about the displays, and soon enough you work out the reason: unlike in a real supermarket, there is only one of each item. And you won't find many of them in a real supermarket anyway: they are failures, products withdrawn from sale after a few weeks or months, because almost nobody wanted to buy them. In the product-design business, the storehouse – operated by a company called GfK Custom Research North America – has acquired a nickname: the Museum of Failed Products.

This is consumer capitalism's graveyard – the shadow side to the relentlessly upbeat, success-focused culture of modern marketing. Or to put it less grandly: it's almost certainly the only place on the planet where you'll find Clairol's A Touch of Yogurt shampoo alongside Gillette's equally unpopular For Oily Hair Only, a few feet from a now-empty bottle of Pepsi AM Breakfast Cola (born 1989; died 1990). The museum is home to discontinued brands of caffeinated beer; to TV dinners branded with the logo of the toothpaste manufacturer Colgate; to self-heating soup cans that had a regrettable tendency to explode in customers' faces; and to packets of breath mints that had to be withdrawn from sale because they looked like the tiny packages of crack cocaine dispensed by America's street drug dealers. It is where microwaveable scrambled eggs – pre-scrambled and sold in a cardboard tube with a pop-up mechanism for easier consumption in the car – go to die.

There is a Japanese term, mono no aware, that translates roughly as "the pathos of things": it captures a kind of bittersweet melancholy at life's impermanence – that additional beauty imparted to cherry blossoms, say, or human features, as a result of their inevitably fleeting time on Earth. It's only stretching the concept slightly to suggest that this is how the museum's proprietor, an understatedly stylish GfK employee named Carol Sherry, feels about the cartons of Morning Banana Juice in her care, or about Fortune Snookies, a short-lived line of fortune cookies for dogs. Every failure, the way she sees it, embodies its own sad story on the part of designers, marketers and salespeople. It is never far from her mind that real people had their mortgages, their car payments and their family holidays riding on the success of products such as A Touch of Yogurt.

"I feel really sorry for the developer on this one," Sherry says, indicating the breath mints that inadvertently resembled crack. "I mean, I've met the guy. Why would he ever have spent any time on the streets, in the drug culture?" She shakes her head. "These are real people who sincerely want to do their best, and then, well, things happen."

The Museum of Failed Products – consumer capitalism’s very own graveyard. Photograph: Kelly K Jones

The Museum of Failed Products was itself a kind of accident, albeit a happier one. Its creator, a now-retired marketing man named Robert McMath, merely intended to accumulate a "reference library" of consumer products, not failures per se. And so, starting in the 1960s, he began purchasing and preserving a sample of every new item he could find. Soon, the collection outgrew his office in upstate New York and he was forced to move into a converted granary to accommodate it; later, GfK bought him out, moving the whole lot to Michigan. What McMath hadn't taken into account was the three-word truth that was to prove the making of his career: "Most products fail." According to some estimates, the failure rate is as high as 90%. Simply by collecting new products indiscriminately, McMath had ensured that his hoard would come to consist overwhelmingly of unsuccessful ones.

By far the most striking thing about the museum, though, is that it should exist as a viable, profit-making business in the first place. You might have assumed that any consumer product manufacturer worthy of the name would have its own such collection – a carefully stewarded resource to help it avoid making errors its rivals had already made. Yet the executives who arrive every week at Sherry's door are evidence of how rarely this happens. Product developers are so focused on their next hoped-for success – so unwilling to invest time or energy thinking about their industry's past failures – that they only belatedly realise how much they need to access GfK's collection. Most surprising of all is that many of the designers who have found their way to the museum have come there to examine – or been surprised to discover – products that their own companies had created, then abandoned. They were apparently so averse to dwelling on the unpleasant business of failure that they had neglected even to keep samples of their own disasters.

Failure is everywhere. It's just that most of the time we'd rather avoid confronting that fact.

Behind all of the most popular modern approaches to happiness and success is the simple philosophy of focusing on things going right. But ever since the first philosophers of ancient Greece and Rome, a dissenting perspective has proposed the opposite: that it's our relentless effort to feel happy, or to achieve certain goals, that is precisely what makes us miserable and sabotages our plans. And that it is our constant quest to eliminate or to ignore the negative – insecurity, uncertainty, failure, sadness – that causes us to feel so insecure, anxious, uncertain or unhappy in the first place.

Yet this conclusion does not have to be depressing. Instead, it points to an alternative approach: a "negative path" to happiness that entails taking a radically different stance towards those things most of us spend our lives trying hard to avoid. This involves learning to enjoy uncertainty, embracing insecurity and becoming familiar with failure. In order to be truly happy, it turns out, we might actually need to be willing to experience more negative emotions – or, at the very least, to stop running quite so hard from them.

In the world of self-help, the most overt expression of our obsession with optimism is the technique known as "positive visualisation": mentally picture things turning out well, the reasoning goes, and they're far more likely to do so. Indeed, a tendency to look on the bright side may be so intertwined with human survival that evolution has skewed us that way. In her book, The Optimism Bias, the neuroscientist Tali Sharot compiles growing evidence that a well-functioning mind may be built so as to perceive the odds of things going well as greater than they really are. Non-depressed people, research suggests, generally have a less accurate and overly optimistic grasp of their true ability to influence events than do those who are suffering from depression.

Yet there are problems with this outlook, aside from just feeling disappointed when things don't turn out well. Over the last few years, the German-born psychologist Gabriele Oettingen and her colleagues have constructed a series of experiments designed to unearth the truth about "positive fantasies about the future". The results are striking: spending time and energy focusing on how well things could go, it has emerged, actually reduces most people's motivation to achieve them. Experimental subjects who were encouraged to think about how they were going to have a particularly high-achieving week at work, for example, ended up achieving less. In one ingenious experiment, Oettingen had participants rendered mildly dehydrated. Then some were taken through an exercise that involved visualising drinking an icy, refreshing glass of water, while others took part in a different exercise. The water-visualisers experienced a significant reduction in their energy levels, as measured by blood pressure. Far from becoming more motivated to hydrate themselves, people responded to positive visualisation by relaxing. They seemed, subconsciously, to have confused imagining success with having already achieved it.

It doesn't necessarily follow, of course, that it would be a better idea to switch to negative visualisation instead. Yet that is precisely one of the conclusions that emerges from Stoicism, a school of philosophy that originated in Athens a few years after the death of Aristotle, and that came to dominate western thinking about happiness for nearly five centuries.

For the Stoics, the ideal state of mind was tranquility – not the excitable cheer that positive thinkers usually seem to mean when they use the word "happiness". And tranquility was to be achieved not by chasing after enjoyable experiences, but by cultivating a kind of calm indifference towards one's circumstances. One way to do this, the Stoics argued, was by turning towards negative emotions and experiences: not shunning them, but examining them closely instead.

Most of us, the Stoics point out, go through life under the delusion that it is certain people, situations or events that make us sad, anxious or angry. When you're irritated by a colleague at the next desk who won't stop talking, you naturally assume that the colleague is the source of the irritation; when you hear that a beloved relative is ill and feel pained for them, it makes sense to think of the illness as the source of the pain. Look closely at your experience, though, say the Stoics, and you'll be forced to conclude that neither of these external events is "negative" in itself. Indeed, nothing outside your own mind can properly be described as negative or positive at all: what actually causes suffering are the beliefs you hold about those things. The colleague is not irritating per se, but because of your belief that getting your work finished without interruption is an important goal. Even a relative's illness is only bad in view of your belief that it's a good thing for your relatives not to be ill. Millions of people, after all, get ill every day; we have no beliefs whatsoever about most of them, and consequently don't feel distressed.

For positive thinkers, this would be an argument for trying to replace your distress-causing beliefs with upbeat ones. But when thinking about the future, Stoics such as Seneca often counselled actively dwelling on worst-case scenarios instead – staring them in the face. Not only does ceaseless optimism make for a greater shock when things go wrong (and they will); imagining the worst also brings its own benefits. Psychologists have long agreed that one of the greatest enemies of human happiness is "hedonic adaptation" – the predictable and frustrating way in which any new source of pleasure we obtain, whether it's as minor as a new electronic gadget or as major as a marriage, swiftly gets relegated to the backdrop of our lives: we grow accustomed to it, and it ceases to deliver so much joy. It follows, then, that regularly reminding yourself that you might lose any of the things you currently enjoy can reverse the adaptation effect. Thinking about the possibility of losing something you value shifts it from the backdrop of your life back to centre stage, where it can deliver pleasure once more.

The second, subtler and arguably more powerful benefit of this kind of negative thinking is as an antidote to anxiety. Consider how we normally seek to assuage worries about the future: we seek reassurance, looking to persuade ourselves that everything will be all right in the end. But reassurance is a double-edged sword. In the short term, it can be wonderful, but like all forms of optimism, it requires constant maintenance: offer reassurance to a friend who is in the grip of anxiety, and you'll often find that, a few days later, he'll be back for more. Worse, reassurance can actually exacerbate anxiety: when you reassure your friend that the worst-case scenario he fears probably won't occur, you inadvertently reinforce his belief that it would be catastrophic if it did. You are tightening the coil of his anxiety, not loosening it.

All too often, the Stoics note, things will not turn out for the best. But it is also true that, when they do go wrong, they'll almost certainly go less wrong than you feared. Losing your job is unlikely to condemn you to starvation and death; losing a relationship won't condemn you to a life of unrelenting misery. Those fears are based on irrational judgments about the future. The worst thing about any future event, the Stoic-influenced psychologist Albert Ellis used to say, "is usually your exaggerated belief in its horror". Spend time vividly imagining exactly how wrong things could go in reality, and you'll often turn bottomless, nebulous fears into finite and manageable ones. Happiness reached via positive thinking is fleeting and brittle; negative visualisation generates a vastly more dependable calm.

The breakfast you were spared – microwaveable scrambled eggs at the Museum of Failed Products in Ann Arbor, Michigan. Photograph: Kelly K Jones

Back at the Museum of Failed Products, it isn't hard to imagine how another downside of the positive-thinking culture – an aversion to confronting failure – might have been responsible for the very existence of many of the products lining its shelves. Each one must have made it through a series of meetings at which nobody realised that the product was doomed. Perhaps nobody wanted to contemplate the prospect of failure; perhaps someone did, but didn't want to bring it up for discussion. Even if they realise where things are headed, there's a perverse incentive for marketers to plough more money into a lemon: that way, they can force some sales and preserve their dignity. By the time the truth becomes obvious, the original developers will have moved to other products, or other firms. Little energy will have been invested in discovering what went wrong; everyone involved will have conspired, perhaps without realising what they're doing, never to speak of it again.

Another problem with our reluctance to think about or analyse failure – whether our own or other people's – is that it leads to an utterly distorted picture of the causes of success. Bookshops are stuffed with autobiographical volumes such as the one released in 2006 by the multimillionaire publisher Felix Dennis, entitled How To Get Rich: The Distilled Wisdom Of One Of Britain's Wealthiest Self-Made Entrepreneurs. It's an entertaining read, conveying a similar message to many of the others: that to make a fortune what you need is stubbornness and a willingness to take risks. But research by the Oxford management theorist Jerker Denrell suggests that these are just as likely to be the characteristics of extremely unsuccessful people, too. It's just that the failures don't write books. You rarely see autobiographies of people who took risks that then didn't work out.

Fortunately, developing a healthier approach to failure may be easier than you'd think. The work of the Stanford University psychologist Carol Dweck suggests that our experiences of failure are influenced overwhelmingly by the beliefs we hold about the nature of talent and ability – and that we can, perhaps quite straightforwardly, nudge ourselves towards a better outlook. Each of us can be placed somewhere on a continuum, Dweck argues, depending on our "implicit view" – or unspoken attitude – about what talent is and where it comes from. Those with a "fixed theory" assume that ability is innate; those with an "incremental theory" believe that it evolves through challenge and hard work. If you're the kind of person who strives mightily to avoid the experience of failure, it's likely that you reside near the "fixed" end of Dweck's continuum. Fixed-theory people approach challenges as occasions on which they are called upon to demonstrate their innate abilities, and so they find failure especially horrifying: to them, it's a sign that they tried to show how good they are, but didn't measure up. The classic example is the young sports star encouraged to think of himself as a "natural" – but who then fails to put in sufficient practice to realise his potential. If talent is innate, his unspoken reasoning goes, then why bother?

Incremental-theory people are different. Because they think of abilities as emerging through tackling challenges, the experience of failure has a completely different meaning for them: it's evidence that they are stretching themselves to their current limits. If they weren't, they wouldn't fail. The relevant analogy here is with weight training: muscles grow by being pushed to the limits of their current capacity, where fibres tear and reheal. Among weightlifters, "training to failure" isn't an admission of defeat – it's a strategy.

Happily, Dweck's studies indicate that we are not saddled for life with one mindset rather than another. Some people manage to alter their outlook simply by being introduced to the fixed versus incremental distinction. Alternatively, it's worth trying to recall it next time failure strikes: next time you flunk an exam, or mishandle a social situation, consider that it's happening only because you're pushing at the limits of your present abilities. And should you wish to encourage an incremental outlook in your children, Dweck advises, take care to praise them for their effort, rather than their intelligence: focusing on the latter is likely to exacerbate a fixed mindset, making them more reluctant to risk encountering failure in the future. The incremental mindset is the one more likely to lead to sustainable success. But the deeper point is that possessing an incremental outlook is a happier way to be, whether or not it leads to any outstanding success. It is a win-win proposition, for which the only precondition is a heartfelt willingness to lose.

Perhaps nobody wanted to contemplate the prospect of these products failing. Photograph: Kelly K Jones

The gurus of positivity and optimism can't bear to contemplate that there might be happiness to be found in embracing failure as failure, not only as a technique for achieving success. But, as the Zen-influenced writer Natalie Goldberg argues, there is an openness and honesty in failure, a down-to-earth confrontation with reality that can seem lacking at the higher altitudes of success. Perfectionism is one of those traits that many people seem secretly, or not-so-secretly, proud to possess, since it hardly seems like a character flaw. Yet, at bottom, it is a fear-driven striving to avoid the experience of failure at all costs. At the extremes, it is an exhausting and permanently stressful way to live: there is a greater correlation between perfectionism and suicide, researchers have found, than between feelings of hopelessness and suicide. To fully embrace the experience of failure, not merely to tolerate it as a stepping stone to glory, is to abandon this constant straining never to put a foot wrong – and to relax.

• This is an edited extract from The Antidote: Happiness For People Who Can't Stand Positive Thinking, by Oliver Burkeman, published next week by Canongate at £15. To order a copy for £12, including mainland UK p&p, go to guardianbookshop.co.uk. Oliver Burkeman will be speaking in Edinburgh on 23 June. To book go to canongate.tv/talks