Scientific publishing as it was meant to be

Last October I joined the editorial board of Cortex, and my first order of business was to propose a new format of article called a Registered Report. The essence of this new article is that experimental methods and proposed analyses are pre-registered and peer reviewed before data is collected. This publication model has the potential to cure a host of bad practices in science.

In November the publisher approved the new article format and I’m delighted to announce that Registered Reports will officially launch on May 1st. I’m especially proud that Cortex will become the first journal in the world to adopt this publishing mechanism.

For those encountering this initiative for the first time, here are some links to background material:

1. The open letter I wrote last October proposing the idea. 

2. A panel discussion I took part in last November at the Spot On London conference, where I spoke about Registered Reports.

3. My freely-accessible editorial article where we formally introduce the initiative (March 2013).

4. **Update 03/05** Finalised author and reviewer guidelines. 
5. **Update 26/04**: Slides from my talk at Oxford where I spoke about the initiative.

Why should we want to review papers before data collection? The reason is simple: as reviewers and editors we are too easily biased by the appearance of data. Rather than valuing innovative hypotheses or careful procedures, we too often we find ourselves applauding “impressive results” or bored by null effects. For most journals, issues such as statistical power and technical rigour are outshone by novelty and originality of findings.

What this does is furnish our environment with toxic incentives. When I spoke at the Spot On conference last year, I began by asking the audience: What is the one aspect of a scientific experiment that a scientist should never be pressured to control? After a pause – as though it might be a trick question – one audience member answered: the results. Correct! But what is the one aspect of a scientific experiment that is crucial for publishing in a high-ranking journal? Err, same answer. Novel, ground-breaking results.

The fact that we force scientists to touch the untouchable is unworthy of a profession that prides itself on behaving rationally. As John Milton says in Devil’s Advocate, it’s the goof of all time. Somehow we've created a game in which the rules are set in opposition.

The moment we incentivize the outcome of science over the process itself, other vital issues fall by the wayside. A priori statistical power becomes neglected, as Kate Button and Marcus Munafo prove today in their compelling analysis of neuroscience studies (and see excellent coverage of this work by Ed Yong and Christian Jarrett).

With little chance of detecting true effects, experimentation reduces to an act of gambling. Driven by the need to publish, researchers inevitably mine underpowered datasets for statistically significant results. No stone is left unturned; we p-hack, cherry pick, and even reinvent study hypotheses to "predict" unexpected results. Strange phenomena begin appearing in the literature that can only be explained by such practices – phenomena such as poor repeatability, prevalence of studies that support stated hypotheses, and a preponderance of articles in which obtained p values fall just below the significance threshold. More worryingly, a recent study by John et al shows that these behaviours are not the actions of a naughty minority – they are the norm.

None of this even remotely resembles the way we teach science in schools or undergraduate courses, or the way we dress it up for the public. The disconnect between what we teach and what we practice is so vast as to be overwhelming. 

Registered Reports will help eliminate these bad incentives by making the results almost irrelevant in reaching editorial decisions. The philosophy of this approach is as old as the scientific method itself: If our aim is to advance knowledge then editorial decisions must be based on the rigour of the experimental design and likely replicability of the findings – and never on how the results looked in the end.

We know that other journals are monitoring Cortex to gauge the success of Registered Reports. Will the format be popular with authors? Will peer reviewers be engaged and motivated? Will the published articles be influential? This success depends on you. We'll need you to submit your best ideas to Cortex – well thought-out proposals that address important questions – and, crucially, before you’ve collected the data. We need your support to help steer scientific publishing toward a better future.

For my part, I’m hugely excited about Registered Reports because it offers hope that science can evolve; that we can be self-critical, open-minded, and determined to improve our own practices. If Registered Reports succeeds then together we can help reinvent publishing as it was meant to be: rewarding the act of discovery rather than the art of performance.

___

 

I am indebted to many people for supporting the Registered Reports initiative, and my sincere apologies if I have left anyone off this list. For generating or helping to inspire the ideas (for which I take no personal credit), I’m grateful to Neuroskeptic, Marcus Munafò, Pete Etchells, Mark Stokes, Frederick Verbruggen, Petroc Sumner, Alex Holcombe, Ed Yong, Dorothy Bishop, Chris Said, Jon Brock, Ananyo Bhattacharya, Alok Jha, Uri Simonsohn, EJ Wagenmakers, Eric Eich, and Brian Nosek. I’m grateful also to Toby Charkin from Elsevier for working hard to facilitate the administrative aspects of the initiative. I also want to thank Zoltan Dienes for joining the editorial board. Zoltan will provide expert advice as part of the initiative for studies involving Bayesian statistical methods and his paper on the advantages of Bayesian techniques over conventional NHST is a must-read. My thanks as well to many members of the Cortex editorial board for their advice and valuable consultation, including especially Rob McIntosh and Jason Mattingley, and to Dario Battisti for the cover art accompanying the Cortex editorial (pictured above). Finally, I am especially grateful to the Editor-in-chief of Cortex, Sergio della Sala for having the vision and courage to support this idea and see it to fruition. A determined and progressive EIC is crucial for the success of any new publishing format, particularly one as ambitious as Registered Reports.