.jpg)
Email
I’m the first to admit that the "registration revolution" sounds less like a banner for reforming science and more like the march of a thousand bureaucrats, clipboards and pens at the ready. But let’s look past the “boring ass word” for a minute. As I wrote back in January, registration of scientific research is one of a set of transparency initiatives that have been staring at us in the face for over 50 years. The reasoning behind it is simply this: that by having scientists state at least part of what they’re going to do before they do it, registration gently but firmly compels us to stick to the scientific method.
Last summer we called for study registration to be formally embraced across all life sciences, including psychology, just as it has been in medical research. And the response has been remarkable: a year later, eight major journals across psychology, psychiatry and neuroscience have launched an initiative called Registered Reports in which scientific studies are reviewed, pre-registered and accepted for publication before results even exist. The list of journals taking on registration is steadily growing, submissions from authors are rising, and funding agencies are taking notice.
Today, the journal Social Psychology publishes a special open access issue that includes the first fruits of this labour: 15 Registered Reports, each of which sought to validate key theoretical predictions in social psychology. Does music influence consumer choice? Are our stereotypes of people activated automatically? Is there a link between physical warmth and social warmth? Some of the phenomena hold up to scrutiny, others less so. Either way, the articles provide a beautiful illustration of how registration can produce rigorous psychological science.
So what is registration and why is it a good thing?
Study registration involves researchers commmitting to their research predictions and methods before starting their experiments. This is important because it ensures that decisions about the parts of a scientific experiment that are supposed to be made before data analysis – mainly the predictions and the primary analyses – actually are made before data analysis. Without study registration, it is easy for scientists to (even unconsciously) short-circuit the scientific method by cherry-picking “good results” out of complex data and then presenting them as though they were predicted from the beginning.
The technical term for this practice is "HARKing", which stands for Hypothesising After Results are Known. It has been estimated that 90% of psychologists HARK, simply because it makes the results of our experiments look more believable and convincing than they really are. The more convincing our results look, the easier they are to publish in the most prestigious academic journals. And, of course, the more articles we chalk up in prestigious journals, the more grant money we attract, the more stable our jobs become, and the quicker we can ascend the career ladder of academia.
By recording our predictions and analyses before experiments begin, registration makes it impossible to reinvent history. Ironically, or perhaps fittingly, it was psychologists themselves who first measured the kind of bias that study registration prevents – a phenomenon known as confirmation bias. Study registration uses that knowledge to avoid fooling ourselves into seeing what we want to see. As professor of psychology Dan Simons from the University of Illinois says, “It keeps us from convincing ourselves that an exploratory analysis was a planned one.”
Interesting results can be junk and dull results could be vital
Registration not only reins in confirmation bias, it also counteracts many toxic incentives in the academic system. Under the current incentive structure, psychologists are pushed to value novelty of results over reliability and to prize the quantity of published work over quality. What happens when a field becomes dominated by such beliefs? The answer is that many researchers stop caring about the truth of individual results – a recent analysis of over a century of psychology research found that just 1 in every 500 published articles bothered to check whether a previous experiment could be closely replicated.
Worst of all, this system reinforces the dogma that the quality of science is best gauged not from the importance of the scientific question or robustness of the methodology, but from the results. This means that whether a study is considered "good science" often depends on the extent to which it produces results that are deemed exciting, novel, and "groundbreaking".
At first glance, that might seem reasonable. After all, aren't the results of experiments a crucial part of science? Aren’t they the source of knowledge and inspiration? Yes, but here is where the mental discipline of science is key: whether or not the results of an experiment are exciting or agree with our expectations tells us nothing about how well the experiment was conducted or what the data might mean in the long run. If you’re the kind of person who thinks that exciting results = better science, then there’s a good chance you don’t love science at all – you just enjoy watching its butt as it walks by.
One neat feature of study registration, and Registered Reports in particular, is that the journal agrees to publish studies before results exist. That way, we prevent the well-known tendency for journal editors to decide which articles to publish based on the results and we encourage researchers to pursue close replications of previous studies. By guaranteeing publication in advance, we also eliminate the incentive for researchers to massage their data in the first place.
The debate
Calls for study registration tend not to encourage leaps for joy. Journals can be fearful of adopting Registered Reports on the grounds that the articles might report negative findings and so reduce the journal’s impact factor, even though impact factor is a thoroughly meaningless metric. And the word "registration" itself conjures up thoughts of regulation and bureaucracy, as though designed to stifle creativity and flexibility. This isn’t even remotely true but first impressions stick.
After our call for study registration last year, a group of neuroscientists led by Professor Sophie Scott of University College London led a vocal critique of Registered Reports. Scott and colleagues claimed that registration would “permit the denigration of the vast majority of great research and allow a number of serious constraints to be placed on it.” Despite the fact that we only ever suggested study registration as an option for scientists, the critics argued that even allowing it a place at the table would be to “put science in chains”.
Professor Dorothy Bishop, Fellow of the Royal Society and developmental neuropsychologist at Oxford University, contests this view. She points out that such criticisms are based on the misunderstanding that registration suppresses creativity, when in fact it simply makes it transparent: “It should not stop exploratory research, but should make it clear what is exploratory and what is not.” Without the option of study registration, Bishop worries that the reliability of science is threatened: “Currently science is not so much in chains as bogged down in a mire of non-replicable findings, and we need to find ways to deal with this.”
Other researchers are more sympathetic to doubts about registration. Professor Tom Johnstone, a cognitive neuroscientist at the University of Reading, says “I think the concern that scientists have of being ‘put in chains’ is understandable. Creativity and adaptive thinking and problem solving are very much a part of science, and mustn't be stifled.” At the same time, he is supportive of reform: “I do think that the move towards registered studies will be of benefit to science, not only because it will encourage better research practice, but also because it will lessen the file-drawer problem by ensuring that ‘null’ results are published.”
Beyond psychology, study registration is gaining support in the social sciences. Political scientist Dr Brendan Nyhan from Dartmouth College says that registration “could transform the future of publishing if funders, government agencies, reviewers, editors, and tenure and promotion committees demand it.”
The future
Like any pudding, the proof is in the eating and while registration has a long way to go, many scientists are deciding that it deserves a chance. For my part, one thought keeps recurring. Long after my scientific discoveries have been proven wrong or inadequate (which is no doubt soon), and long after all of the scientific techniques we use are seen as primitive and naive, I like to think that the systems we're putting in place now to safeguard science will still be there. My hope is that study registration will stand the test of time – that it will free future generations of scientists from having their research and professional worth assessed on the most stupid of all grounds: whether their experiments produced "novel and exciting results".
Chris Chambers is a senior research fellow in cognitive neuroscience at Cardiff University, chair of the Registered Reports committee supported by the Center for Open Science, and section editor for Registered Reports at the journals Cortex and AIMS Neuroscience. He is on Twitter @chrisdc77. Extended comments on study registration can be found on his personal blog.
;