Excerpt from The Antidote by Oliver Burkeman, plus links to reviews, author biography & more

My commiserations on your failure.

Wegner's earliest investigations of ironic process theory involved little more than issuing this challenge to American university students, then asking them to speak their inner monologues aloud while they made the attempt. This is a rather crude way of accessing someone's thought processes, but an excerpt from one typical transcript nonetheless vividly demonstrates the futility of the struggle:

Of course, now the only thing I'm going to think about is a white bear … Don't think about a white bear. Ummm, what was I thinking about before? See, I think about flowers a lot … Okay, so my fingernails are really bad … Every time I really want, like … ummm … to talk, think, to not think about the white bear, then it makes me think about the white bear more...

At this juncture, you might be beginning to wonder why it is that some social psychologists seem to be allowed to spend other people's money proving the obvious. Of course the white bear challenge is virtually impossible to win. But Wegner was just getting started. The more he explored the field, the more he came to suspect that the internal mechanism responsible for sabotaging our efforts at suppressing white bear thoughts might govern an entire territory of mental activity and outward behaviour. The white bear challenge, after all, seems like a metaphor for much of what goes wrong in life: all too often, the outcome we're seeking to avoid is exactly the one to which we seem magnetically lured. Wegner labelled this effect 'the precisely counterintuitive error', which, he explained in one paper, 'is when we manage to do the worst possible thing, the blunder so outrageous that we think about it in advance and resolve not to let that happen. We see a rut coming up in the road ahead, and proceed to steer our bike right into it. We make a mental note not to mention a sore point in conversation, and then cringe in horror as we blurt out exactly that thing. We carefully cradle the glass across the room, all the while thinking "Don't spill", and then juggle it onto the carpet under the gaze of our host.'

Far from representing an occasional divergence from our otherwise flawless self-control, the capacity for ironic error seems to lurk deep in the soul, close to the core of our characters. Edgar Allan Poe, in his short story of the same name, calls it 'the imp of the perverse': that nameless but distinct urge one sometimes experiences, when walking along a precipitous cliff edge, or climbing to the observation deck of a tall building, to throw oneself off – not from any suicidal motivation, but precisely because it would be so calamitous to do so. The imp of the perverse plagues social interactions, too, as anyone who has ever laughed in recognition at an episode of Curb Your Enthusiasm will know all too well.

What is going on here, Wegner argues, is a malfunctioning of the uniquely human capacity for metacognition, or thinking about thinking. 'Metacognition', Wegner explains, 'occurs when thought takes itself as an object.' Mainly, it's an extremely useful skill: it is what enables us to recognise when we are being unreasonable, or sliding into depression, or being afflicted by anxiety, and then to do something about it. But when we use metacognitive thoughts directly to try to control our other, everyday, 'object-level' thoughts – by suppressing images of white bears, say, or replacing gloomy thoughts with happy ones – we run into trouble. 'Metathoughts are instructions we give ourselves about our object-level thinking,' as Wegner puts it, 'and sometimes we just can't follow our own instructions.'

When you try not to think of a white bear, you may experience some success in forcing alternative thoughts into your mind. At the same time, though, a metacognitive monitoring process will crank into action, to scan your mind for evidence of whether you are succeeding or failing at the task. And this is where things get perilous, because if you try too hard – or, Wegner's studies suggest, if you are tired, stressed, depressed, attempting to multi-task, or otherwise suffering from 'mental load' – metacognition will frequently go wrong. The monitoring process will start to occupy more than its fair share of limelight on the cognitive stage. It will jump to the forefront of consciousness – and suddenly, all you will be able to think about is white bears, and how badly you're doing at not thinking about them.

Excerpted from The Antidote by Oliver Burkeman. Copyright © 2012 by Oliver Burkeman. Excerpted by permission of Faber and Faber. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.