Comments on study pre-registration and Registered Reports

** You can download our 25-point Q&A about Registered Reports here **

As part of today's Guardian post on study pre-registration in psychology, I sought feedback on three questions from a number of colleagues. Due to space constraints I couldn’t do their insights justice, so I’ve reproduced their complete answers below. 

At the bottom of the post I've included a full list of journals offering Registered Reports and related initiatives. Enjoy! 

Question 1: What would you say to critics who argue that pre-registration puts "science in chains"? Are their concerns justified? 

Professor Dorothy Bishop, University of Oxford 

I think there's a widespread misunderstanding of pre-registration. It's main function is to distinguish hypothesis-testing analyses from exploratory analyses. It should not stop exploratory research, but should make it clear what is exploratory and what is not. Most of the statistical methods that we use make basic assumptions that are valid only in a hypothesis-testing context. If we explore a multidimensional dataset, decide on that basis what is interesting, and then apply statistical analysis, we run a high risk of obtaining spurious 'significant' findings. Currently science is not so much in chains as bogged down in a mire of non-replicable findings, and we need to find ways to deal with this. I increasingly find myself reading papers and wondering just what I can believe - particularly in areas of neuroscience where there are huge multidimensional datasets and multiple researcher degrees of freedom in choosing how to analyse findings. I would not insist that pre-registration is mandatory, but I think it's great to have that option and I hope that as the new generation of scientists learn more about it, they will come to embrace it as a way of clarifying scientific findings and achieving better replicability of research. 

Professor Tom Johnstone, University of Reading 

I think the concern that scientists have of being "put in chains" is understandable. We've all probably had the frustrating experience of confronting a reviewer or editor who believes there's one way, and one way only, to collect data or perform analysis, for example. Creativity and adaptive thinking and problem solving are very much a part of science, and mustn't be stifled. 

Yet the solution is to make sure that the move towards pre-registration is accompanied by an expansion of the ways in which researchers can openly report innovative exploratory research, and the iterative development of new methods. As you've pointed out, if we didn't try to shoehorn all of our research into the hypothesis-testing model, then we'd relieve a lot of the pressure for people to engage in post hoc hypothesis creation. 

Dr Daniël Lakens, Eindhoven University of Technology

Science is like a sonnet. There is a structure within which scientists work, but that does not have to limit our creativity. As Goethe remarked: ‘In der Beschränkung zeigt sich erst der Meister’ - Mastery is seen most clearly when constrained. 

Dr Brendan Nyhan, Dartmouth College 

I think the idea that pre-registration will put “science in chains” is attacking a straw man. No one is proposing that it should be the only way to conduct research. There will still be every opportunity to pursue unanticipated findings. The widespread availability of pre-registered journal articles will more clearly distinguish between true hypothesis-testing and exploratory research. For instance, a researcher might observe an unanticipated result and then pre-register a replication study to test the effect more systematically.

Professor Dan Simons, University of Illinois 

Frankly, this criticism is nonsense. Pre-registration just eliminates the ability to fool yourself into thinking some post-hoc decision was actually an a-priori one. Specifying a plan in advance just means that you actually did plan your "planned" analyses. As psychologists, we should know how easily we can convince ourselves that the analysis that worked was the logical one to do, after the one we first thought to try didn't work. If your theory makes a prediction, you should be able to specify it in advance and you should be able to specify what outcomes would support it. Yes, it takes more work up front to pre-register a plan. But, if you truly are conducting planned analyses, all you are doing is shifting when you do that work, not what you're doing.  

Nothing about pre-registration prevents a researcher from conducting additional exploratory analyses that were not part of the registered plan. Pre-registration just makes clear which analyses were planned and which ones were exploratory. How does that constrain science in any way? 

Question 2: Do you think pre-registration will influence the future of publishing in psychology, neuroscience and beyond?  

Professor Tom Johnstone, University of Reading 

I do think that the move towards registered studies will be of benefit to science, not only because it will encourage better research practice, but also because it will lessen the file-drawer problem by ensuring that "null" results are published. It will also hopefully catalyse a shift towards more informative statistics than standard NHST. That's not to say there won't be problems; undoubtedly there will be (concerns about research timelines especially for junior researchers need to be tackled head-on, for example). 

Dr Daniël Lakens, Eindhoven University of Technology

It will complement the way we work in important ways. Especially in ‘hot’ research areas, which are at a higher risk of increased Type 1 errors (Ioannides, 2005), pre-registration will greatly facilitate our understanding of how likely it is things are true. 

Dr Brendan Nyhan, Dartmouth College 

Pre-registration could transform the future of publishing if funders, government agencies, reviewers, editors, and tenure and promotion committees demand it. The movement will only succeed if it changes expectations about research credibility among a wider group of scholars and stakeholders than its most devoted advocates. It should also take further steps to broaden its appeal to researchers - most notably, by encouraging journals to adopt formats like Registered Reports that reduce risk to scholars concerned about their ability to publish pre-registered null results given the publication biases in scientific journals. 

Professor Dan Simons, University of Illinois 

Pre-registration effectively eliminates hypothesizing after the results are known. It keeps us from convincing ourselves that an exploratory analysis was a planned one. It is perhaps the best way to keep yourself from inadvertent p-hacking and to convince others that your hypotheses predicted rather than followed from your results. Ideally, more journals will begin reviewing the registered plans as the basis for publication decisions. Doing so would effectively eliminate the file drawer problem. If a study is well designed, its results should be published.  

Question 3: Why do you think psychology and neuroscience are spearheading these initiatives, rather than other sciences? 

Professor Dorothy Bishop, University of Oxford 

I think there are two reasons. First, most psychologists (though not neuroscientists in general) get a good grounding in statistics at undergraduate level, so they have been quicker to appreciate the problems that are inherent in 'false positive psychology'. Second, psychologists study how people think and are aware of how easy it is to deceive yourself at all kinds of levels: after all, one of the first things that many students learn about is the Muller-Lyer visual illusion, where you are convinced that two lines are different lengths when in fact they are the same. That should make us more vigilant about always questioning whether our findings are correct; we are taught to look for counter-evidence rather than just confirming our pre-conceptions. 

Professor Tom Johnstone, University of Reading 

As to why this is being lead by psych/neuro, hard to say. Probably a case of the right combination of factors coinciding (e.g.recent high-profile spotlight on QRP and fraud in social psychology, links to medical research and associated ethics, in which registration has been recently enforced, a few people willing to actively push this forward), plus peculiarities of psych research compared to some other disciplines (for example, speaking with my physics training hat on, the almost complete reliance on NHST in psychology and neuroscience, rather than accurate quantitative description of effects, and the almost total lack of replication). There is, I think, a research culture difference here. That will be difficult to change, but one has to start somewhere. 

Dr Daniël Lakens, Eindhoven University of Technology

According to Parker (1989), ‘psychology is in a continuous crisis’. Psychology has a tradition of self-criticism. It is sometimes remarked that psychology’s greatest contribution is methodology (e.g., Scarr, 1997), so it is not surprising we are on the forefront of methodological improvements in the current debate about ways to improve our science.

Dr Brian Nosek, University of Virginia

The reproducibility challenges facing science are strongly influenced by the incentives and social context that shape scientists' behavior.  Understanding and altering incentives, motivations, and social context are psychological challenges.  Psychologists are ahead because they are just applying their domain expertise on themselves. 

Links to Registered Reports initiatives and related formats 

Journal: AIMS Neuroscience 
Detailed guidelines: http://www.aimspress.com/reviewers.pdf (Nb. The AIMS website is currently down but I am told it will be back up soon).
Editorial: http://orca.cf.ac.uk/59475/1/AN2.pdf 

Journal: Attention, Perception and Psychophysics
Detailed guidelines: http://link.springer.com/content/pdf/10.3758%2Fs13414-013-0502-5.pdf 

Journal: Cortex
Editorial: https://dl.dropboxusercontent.com/u/15691907/Chambers_2013b_Cortex.pdf 
Detailed guidelines: http://cdn.elsevier.com/promis_misc/PROMIS%20pub_idt_CORTEX%20Guidelines_RR_29_04_2013.pdf 

Journal: Drug and Alcohol Dependence
Editorial: https://dl.dropboxusercontent.com/u/15691907/DAD_RegisteredReports_Editorial.pdf 
Detailed guidelines: http://cdn.elsevier.com/promis_misc/DAD_RR_GL_final.pdf 

Journal: Experimental Psychology
Editorial: http://www.psycontent.com/content/l8631300u1u7r5h3/fulltext.pdf 
Detailed guidelines: http://www.hogrefe.com/fileadmin/redakteure/hogrefe_com/Periodicals/Experimental_Psychology/zea_InstructionsToAuthors_2014-01-02.pdf 

Journal: Journal of Business and Psychology 
Announcement and guidelines: http://www.springer.com/cda/content/document/cda_downloaddocument/JBP+RR+Special+Issue+May+5RL.pdf?SGWID=0-0-45-1458040-p35536793 

Journal: Journal of Experimental Psychology: General 
Announcement inviting registered replications: http://www.apa.org/pubs/journals/xge/ 

Journal: Perspectives on Psychological Science 
Guidelines: http://www.psychologicalscience.org/index.php/replication 

Journal: Social Psychology 
Editorial: http://www.psycontent.com/content/311q281518161139/fulltext.pdf

Guidelines: To come...