The 'geek movement' is a straw man

I’ve been told I have a thick skin, but a recent post attacking the ‘geek movement’ pissed me off in a way that I generally reserve for articles by James Delingpole. It annoyed me not because it attacked me directly (and I wouldn’t care if it did) but because it has the potential to dissuade scientists – especially junior ones – from engaging proactively with politicians and society.

I won’t go into a detailed critique of the post here, but suffice to say that it spectacularly misrepresents everything and everyone it seeks to attack. The comments below the post from Brian Cox, Mark Henderson, David Colquhoun and Shane McKee say it all. Both the insulting tone and patronising rhetoric of the post suggests that this misrepresentation was a deliberate tactic to garner attention. Perhaps not the attention it expected, mind you.

Angry posts decrying a non-existent ‘geek movement’ do nothing to champion science or science communication. They also don’t challenge us in any meaningful or intelligent way. As far as I can tell, such articles serve only to generate faux dissent and ego-boosting publicity for those who author them. We need intelligent and nuanced discussion about the role of science in politics – not cheap shots.

So, to any scientists (especially junior ones) who happened to read the above post, please don’t be put off from engaging. Society needs you. Inspired by the Geek Manifesto, we’ve already seen fantastic campaigns to link scientists with politicians by Shane McKee and Dave Watts. We have also completed our own modest campaign to send the Geek Manifesto and personal letters to each member of the National Assembly for Wales, and we are also in the process of setting up a new evidence information service for MPs.

I say all this to accentuate the positive because in the last 24 hours my Twitter feed has become fairly unpleasant and acrimonious place. Intelligent discussions about the role of science in politics are vital and we mustn't allow this dialogue to be hijacked by trolling.

A manifesto for reform

Today saw the successful completion of the Welsh Geek Manifesto campaign. At midday we delivered 60 copies of Mark Henderson’s book to the National Assembly building in Cardiff Bay. I'm happy to say even the Sun made an appearance (the fireball, not the newspaper).

The Geek Manifesto is an important book because it not only lays out in stark detail the evidence crisis facing British politics - it also shows how the scientific community is an integral part of the solution. It is this call for action and self-reflection that makes the book unique.

From left: Petroc Sumner, Fred Boy, me, Gabi Jiga-Boy, and John Evans

Well, we are acting – and this is just the beginning. In addition to providing one copy of the book for each Assembly member we are also attempting to set up an evidence information service for politicians and civil servants that capitalises on the enormous untapped 'hive mind' of the UK STEM community. I’ll provide updates on this initiative as we proceed. For now I can say that we look forward to continuing our discussions with the Government Office for Science, the Parliamentary Office of Science and Technology, and the National Assembly for Wales.

Together with each copy of the Geek Manifesto we also sent each Assembly member (AM) a personal letter that explains why s/he should read the book, providing some personal context based on the AM’s track record on science and health issues. The full archive of these letters can be downloaded here. I must say that, overall, I was actually rather impressed with the level of engagement on science issues by the Assembly. Almost all AMs had a recent track record of involvement in a STEM subject such as climate change, energy policy, or public health.

One of the 60 letters provided to each Assembly member.

I want to thank all the people who made this campaign possible. First, my gratitude and congratulations to Mark Henderson for penning such an important book at such a key time in British politics. I also want to thank the members of the organising group, including Tom Crick (who conceived the idea for this campaign), John Evans for painstakingly translating each of the 60 letters into Welsh, our intern Laura Smith for conducting the critical research on each AM’s scientific track record, Fred Boy for coordinating media coverage of the delivery event through Swansea University, and Gabi Jiga-Boy, Simon Dymond, and Leah Maizey for their support.

Finally, and most importantly, I want to thank the 60 donors who purchased the books. This is no small gesture in such tough economic times, and I hope the AMs will appreciate both your generosity and your sense of purpose.

Very Good Friday

Two pieces of news from the CUBRIC TMS group.

First – I’m delighted to report that on Friday my first PhD student, Chris Allen, passed his viva with flying colours. A big thanks to his diligent examiners, Simon Rushton and Kia Nobre. Chris’ PhD addressed the neural basis of visual consciousness in humans, using an impressive combination of brain stimulation (TMS) and brain imaging methods (including magnetoencephalography and magnetic resonance spectroscopy). The thesis is full of interesting experiments, but for me the most intriguing discovery was that inhibiting the visual cortex with TMS can (paradoxically) enhance conscious perception. Chris is currently preparing several papers for submission and we’ll be sure to post Research Briefings on this blog once the work is published.

Post-viva drinks with Chris (left) and his examiners, Simon Rushton (Cardiff) and Kia Nobre (Oxford)

Second – also on Friday – we heard that our BBSRC application, “Neural dynamics of response inhibition and gambling across the lifespan”, was successful. My sincere thanks to our grant assessors and BBSRC Research Committee A for supporting this project. And it's probably a good thing we didn't include this particular pearl of wisdom in the Pathways to Impact section (hat-tip to Mark Stokes).

This 3-year study extends work that we published in 2010 and 2012, and will take place in partnership with Frederick Verbruggen from the University of Exeter, John Evans at Cardiff University, Andreas Bungert from the Max Planck Centre in Tuebingen, and Sven Bestmann from UCL. In our previous experiments we looked at the psychology and neuroscience of cognitive control, focusing particularly on the ability to inhibit thoughts and actions (see here for the Research Briefing associated with the more recent of those projects).

Our BBSRC project seeks to discover how we control our impulses. Let’s say you’re in Las Vegas and have just won $200 on blackjack. The win is thrilling but you soon feel hungry for another hit. Do you stop and collect your winnings or risk what you’ve gained? And if you decide to resist temptation and stop gambling, what systems in your brain helped you arrive at that decision? One hypothesis is that the prefrontal cortex, our most advanced brain area, supports self-control by regulating the activity of deeper, more primitive neural circuits that – if left unchecked – would get us into a lot of trouble.

There is an interesting side note to this which I think is often understated in popular culture. We tend to think of the brain, intuitively, as a single system (perhaps because our own sense of self is unitary – i.e. there is only one “you”). But the reality is that different parts of our brains have evolved over very different time scales. The human brain didn't emerge from the mists of evolution as a whole entity. On the contrary, most of the older, deeper parts have been hanging around for a very long time, and the rest was built around them. These ancient areas are many millions of years old and are much the same in humans as in cats, mice and even birds. But the parts that truly ‘make us human’ lie in the neocortex, literally the ‘new’ part of the brain that evolved relatively recently. And the newest kid on the block is the prefrontal cortex, which we believe oversees all the other systems, including the unrulier deeper parts. Being a Johnny-come-lately has its advantages.

In the first phase of our BBSRC project we’re going to test how the prefrontal cortex regulates behaviour by stimulating it with TMS while people undergo a functional MRI scan to detect brain activity. Our reasoning is that if the prefrontal cortex controls primitive systems then when people are in the process of stopping a response, stimulating the prefrontal cortex at the right time should especially activate deeper areas. So we’ll be using TMS a bit like a torch in a dark room, lighting up the brain pathways that are important for inhibiting our impulses. We’re also going to apply this approach in younger and older adults to see how the normal ageing process alters these connections.

In the second part of the project we’re going to try and figure out why training people to inhibit simple motor actions reduces gambling behaviour. We already have an ESRC grant based in Exeter to study the psychological side of this question, but here with the BBSRC we’re going to tackle the neurological side. To do this we’ll measure brain activity after ‘inhibition training’ using functional MRI and a technique called magnetic resonance spectroscopy, which measures the concentration of various neurochemicals.  There are reasons to think that one of these chemicals, a neurotransmitter called GABA, could be important for regulating behaviour. If so we should see an increase in GABA after inhibition training.

Also – I’m excited to report that all studies in this BBSRC project will be conducted according to the principles of open science. This means we’re going to publicly upload all of our proposed methods and analyses (in detail) prior to collecting data. Then once the experiment is complete we’ll upload the raw data themselves for free public access, including behavioural and brain-imaging data. The benefits of open science are enormous: by pre-registering our research the scientific community can be assured that our results aren’t cherry picked or massaged to find ‘statistical significance’ – and by providing the raw data we will allow others to explore the results in ways we hadn’t considered.

Finally, we will be advertising soon for a 3-year full-time research assistant to work on this project with us (starting in the new year). So if this project grabs your interest, do keep an eye on this blog and you can follow me on Twitter for immediate updates.

Changing the culture of scientific publishing from within

****************************
Update, 20 October
 I have just learned of a related idea being proposed at Psychological Science.
 ****************************

****************************
Update, 12 November
At 11am we got the all-clear from the publisher. This is going to be reality.
Stay tuned for further developments! 
****************************

This is a long post and won’t appeal to everyone. But it could well be the most important thing I have committed to this blog since I started it seven months ago.

What follows is an open letter to my colleagues on the editorial board at the journal Cortex, where I've just been made an associate editor. My new position offers the exciting opportunity to push much needed reforms in scientific publishing, and moreover, to do so from within the machinery of a peer-reviewed journal.

Here I'm pitching one reform in particular: a new kind of empirical article called a Registration Report, which would involve peer review of research methodology prior to data collection. Ever since reading this important post by Neuroskeptic, I’ve been convinced that study registration is the cure to much of what ails us.

Before launching into the text of the letter (or 'working document' as I prefer to think of it), I'd like to offer my thanks to the following people for inspiring – and, indeed, outright generating – the ideas in this proposal: Neuroskeptic, Marcus Munafò, Pete Etchells, Mark Stokes, Frederick Verbruggen, Petroc Sumner, Alex Holcombe, Ed Yong, Dorothy Bishop, Chris Said, Jon Brock, Ananyo Bhattacharya, Alok Jha, Uri Simonsohn, EJ Wagenmakers, and Brian Nosek. And no doubt many others who I have temporarily forgotten (my apologies, I will update the post accordingly as more names come to mind!)

You might ask why I'm blogging this. Well, I think it's important for two reasons. First, it's good to be transparent about the problems facing scientific publishing and the possible solutions we have to choose from. And second, I want this discussion to be open not only to the editorial board of Cortex but to scientists (both senior and junior), science writers and journalists, science policy makers, and science enthusiasts generally - and, in particular, to the scientists who would consider sending their submissions to Cortex. So whoever you are and whatever you do, if you care about scientific publishing then please do leave a comment.

Tell me if you think it's a good idea. Tell me if you think it's a stupid or naïve idea. Tell me where I've missed something crucial or where you see a particularly strong selling point. And above all, tell me this: would you consider submitting your own manuscripts using this new article format? The more interest this proposal receives, and publicly, the better chance I have of convincing my colleagues and the journal publisher to pursue it – or something like it.

Enough preamble. Here’s my open letter.

=====================================================================

Registered Reports: A proposal for a new article format at Cortex

=====================================================================

It is a great privilege to become an associate editor at Cortex.

Cortex was one of the first journals I published in, and I have reviewed at the journal for many years now. I’m particularly humbled to join such a distinguished editorial board.

As delighted as I am to join Cortex, I think we need to be doing more than editing submissions according to standard practices. In most journals, the traditional approach for handling empirical articles is archaic and demonstrably flawed. I believe we should be using our editorial positions to institute reforms that are long overdue.  

1. General proposal and rationale

I would therefore like to propose a new form of empirical article at Cortex, called Registered Reports. I hope to start a discussion among the editorial board and wider scientific community about the merits and drawbacks of such a proposal. In addition to emailing this document to the editorial board, I have also published it on my blog for open discussion, so please feel free to reply either confidentially (via email) or publicly (on the blog). This proposal is very much a working document so any edits or comments on the document itself are most welcome.

I need to make one point clear at the outset. At this stage I am not proposing that we drop any of the existing article formats at Cortex. Rather, I am suggesting an additional option for authors.

The cornerstone of Registered Reports is that a large part of the manuscript would be reviewed prior to the experiments being conducted. Initial manuscripts would be submitted before a study has been undertaken and would include a description of the key background literature, hypotheses, experimental procedures, analysis pipeline, a statistical power analysis, and pilot data (where applicable). Following peer review, the article would then be either rejected or accepted in principle for publication.

Once in principle acceptance (IPA) has been obtained, the authors would then proceed to conduct the study, adhering exactly to their peer-reviewed procedures. When the study is complete the authors would submit their finalised manuscript for re-review and would upload their raw data and laboratory log via Figshare for full public access. Pending quality checks and a sensible interpretation of the findings, the manuscript would be published – and, crucially, independently of what the results actually look like.

This form of article has a number of advantages over the traditional publishing model. First and foremost, it is immune to publication bias because the decision to accept or reject manuscripts will be based on the significance of the research question and methodological validity, never on whether results are statistically significant.

Second, by requiring prospective authors to adhere to a preapproved methodology and analysis pipeline, it will eliminate a host of suspect but common practices that increase false discoveries, including p value fishing (i.e. adding subjects to an experiment until statistical significance is obtained – a practice admitted to by 71% of recently surveyed psychologists; [9]) and selective reporting of experiments to reveal manipulations that “work”. Currently, many authors partake in these practices because doing so helps convince editors and reviewers that their research is worthy of publication. By providing IPA prior to data collection, the incentive to engage in these practices will be largely eliminated.

Third, by requiring an a priori power analysis, including a stringent minimum power level (see below), false negatives will be greatly reduced compared with standard empirical reports. This will increase the veracity of non-significant effects.

Taken together, these practices will ensure that articles published as Registered Reports have a substantially higher truth value than regular studies. Such articles can therefore be expected to be more replicable and have a greater impact on the field.

Why should we want to make this change? The life sciences, in general, suffer from a number of serious problems including publication bias [1, 2], low statistical power [3, 4], undisclosed post-hoc analytic flexibility [5, 6, 7], and a lack of data transparency [8]. By valuing findings that are novel and eye-catching over those that are likely to be true, we have incentivised a range of questionable practices at individual and group levels. What’s more, a worryingly high percentage of psychologists admit to engaging in dubious practices such as selectively reporting experiments that produced desirable outcomes (67%) and p value fishing (71%) [9].

So why should we change now? After all, these problems are far from new [10, 11]. My instinctive response to this question is, why haven't we changed already? In addition, there are several reasons why advances in scientific publishing are especially timely. The culture of science is evolving quickly under heightened funding pressure, with an increasing emphasis on transparency and reproducibility [12], open access publication [13], and the rising popularity of the PLoS model and other alternative publication avenues. Furthermore, retractions are at a record high [14], and recent high-profile fraud cases (e.g. Stapel, Smeesters, Sanna, Hauser) are casting a long shadow over our discipline as a whole.

The ideas outlined here are not new and I certainly can’t claim credit for them. I formulated this proposal after a year of discussion with scientists in multiple disciplines (including journal editors), science policy makers, science journalists and writers, and the Science Media Centre, as well as key blog articles (e.g. here, here and here).

I hope I can convince you that Registered Reports would provide an important innovation in scientific publishing and would position Cortex as a leader in the field. If you agree, in principle, then our next step will be to decide on the details. Then, finally, we would need to convince Elsevier to take this journey with us.

If we succeed then it will bring the scientific community one step closer to a system in which the incentive to discover something true, however small, outweighs the incentive to produce ‘good results’. Call me a shameless idealist, but I find that possibility hugely exciting.

2. The proposed mechanism

Registered Reports would work as follows.

(a) Stage 1: Registration review

Authors submit their initial manuscript prior to commencing their experiment(s). The initial submission would include the following sections:

·      Background and Hypotheses

o   A review of the relevant literature that motivates the research question, and a full description of the aims and experimental hypotheses.

·      Methods

o   Full description of proposed sample characteristics, including criteria for subject inclusion and exclusion, and detailed description of procedures for defining outliers. Procedures for objectively defining exclusion criteria due to technical errors (e.g. defining what counts as ‘excessive’ head movement during fMRI) or for any other reasons (where applicable) must be documented, including details of how and under what conditions subjects would be replaced.

o   A description of experimental procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring any further information.

o   Proposed analysis pipeline, including all preprocessing steps, and a precise description of every analysis that will be undertaken and appropriate correction for multiple comparisons. Any covariates or regressors must be stated. Consistent with the guidelines of Simmons et al. (2011; see 5), proposed analyses involving covariates must be reported with and without the covariate(s) included. Neuroimaging studies must document in advance, and in precise detail, the complete pipeline from raw data onwards.

o   Where analysis decisions or follow-up experiments are contingent on the outcome of prior analyses, these contingencies must be detailed and adhered to.

o   A statistical power analysis. Estimated effect sizes should be justified with reference to the existing literature. To account for existing publication bias, which leads to overestimation of true effect sizes [15, 16], power analysis must be based on the lowest available estimate of the effect size. Moreover, the a priori power (1 - B) must be 0.9 or higher. Setting a high power criterion for discovery of minimal effect sizes is paramount given that this model will lead to the publication non-significant effects.

o   In the case of very uncertain effect sizes, a variable sample size and interim data analysis would be permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed [17], and a final stopping rule for data collection outlined.

o   Full description of any outcome-neutral criteria that are required for successful testing of the study hypotheses. Such ‘reality checks’ might include the absence of floor or ceiling effects, or other appropriate baseline measures. Editors must ensure that such criteria are not used by reviewers to enforce dogma about accepted ‘truths’. That is, we must allow for the possibility that failure to show evidence for a critical ‘reality check’ can raise doubt about the truth of that accepted reality in the first place.

o   Timeline for completion of the study and proposed resubmission date if registration review is successful. Extensions to this deadline can be arranged with the action editor.

·      Pilot Data

o   Optional. Can be included to establish reality checks, feasibility, or proof of principle. Any pilot data would be published with the final version of the manuscript and will be clearly distinguished from data obtained for the main experiment(s).

In considering papers in the registration stage, reviewers will be asked to assess:

• The significance of the research question(s)
• The logic, rationale, and plausibility of the proposed hypotheses
• The soundness and feasibility of the methodology and analysis pipeline
• Whether the level of methodological detail provided would be sufficient to duplicate exactly the proposed experimental procedures and analytic approach

Attempted replications of high profile studies would be welcomed. For replication attempts to be accepted, they must be regarded by the reviewers as significant and important regardless of outcome (i.e. having a high replication value [18] as was the case in the recent attempted replication of precognition effects [19]).

Manuscripts that pass registration review will be issued an in principle acceptance (IPA). This means that the manuscript is accepted for publication pending successful completion of the study according to the exact methods and analytic procedures outlined, as well as a defensible and evidence-based interpretation of the results.

Upon receiving IPA, authors will be informed that any deviation from the stated methods, regardless of how minor it may seem, will be lead to summary rejection of the manuscript. If the authors wish to alter the experimental procedures following IPA but still wish to publish it as a Registered Report in Cortex then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission.

(b) Stage 2: Full manuscript review

Once the study is complete, the authors then prepare and resubmit their manuscript for full review, with the following additions:

·      Submission of raw data and laboratory log

o   Raw data must be made freely available via the website Figshare (or an alternative free service). Data files must be appropriately time stamped to show that it was collected after IPA and not before. Other than pre-registered and approved pilot data, no data acquired prior to the date of IPA is admissible in the final submission. Raw data must be accompanied by guidance notes, where required, to assist other scientists in replicating the analysis pipeline.

o   The authors must collectively certify that all non-pilot data was collected after the date of IPA. A simple laboratory log will be provided outlining the range of dates during which data collection took place.

·      Revisions to the Background and Rationale

o   The stated hypotheses cannot be altered or appended. However, it is perfectly reasonable for the tone and content of an Introduction to be shaped by the results of a study. Moreover, depending on the timeframe of data collection, new relevant literature may have appeared between registration review and full manuscript review. Therefore, authors will be allowed to update at least part of the Introduction.

·      Results & Discussion

o   This will be included as per standard submissions. With one exception, all registered analyses must be included in the manuscript. The exception would be (very) rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded in the first place (i.e. the authors, reviewers, and editor made a collective error of judgment and must collectively agree that the analysis is, in fact, inappropriate). In such cases the analysis would still be mentioned in the Method but omitted from the Results (with the omission justified).

o   It is sensible that authors may occasionally wish to include additional analyses that were not included in the registered submission; for instance, a new analytic approach might emerge between IPA and full review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, caveated, and reported in a separate section of the Results titled “Post hoc analyses”. Editors must ensure that authors do not base their conclusions entirely on the outcome of significant post hoc analyses.

o   Authors will be required to report exact p values and effect sizes for all inferential tests.

The resubmission will ideally be considered by the same reviewers as in the registration stage, but could also be assessed by fresh reviewers. In considering papers at the full manuscript stage, reviewers will be asked to appraise:

• Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects)
• Whether any changes to the Introduction are reasonable and do not alter the rationale or hypotheses
• Whether the authors adhered precisely to the registered experimental procedures
• Whether any post-hoc analyses are justified, robust, and add to the informational content of the paper
• Whether the authors’ conclusions are justified given the data

Crucially, reviewers will be informed that editorial decisions will not be based on the perceived importance or clarity of the data. Thus while reviewers are free to enter such comments on the record, they will not influence editorial decisions.

Reviews will be anonymous. To maximise transparency, however, the anonymous reviews and authors’ response to reviewers will be published alongside the full paper in an online supplement.

Manuscript withdrawal

It is possible that authors with IPA may seek to withdraw their manuscripts following or during data collection. Possible reasons could include technical error or an inability to complete the study due to other unforeseen circumstances. In all such cases, manuscripts can of course be withdrawn. However, the journal will publicly record each case in a section called Retracted Registrations. This will include the authors, proposed title, an abstract briefly outlining the original aim of the study, and brief reason(s) for the failure to complete the study. Partial retractions are not possible; i.e. authors cannot publish part of a registered study by selectively retracting one of the planned experiments. Such cases must lead to retraction of the entire paper.

3. Concerns, Responses and Discussion Points

Here follows a paraphrased Q & A, including some actual and hypothetical discussions about the proposal with colleagues.

1.     Won’t Registered Reports just become a dumping ground for inconclusive null effects?

a.     No. The required power level will increase the chances of detecting statistical significance when it reflects reality. Average power in psychology/cognitive neuroscience is low whereas IPA will be contingent on power of 0.9 or above. Thus, any non-significant findings will, by definition, be more conclusive than typically observed in the literature.

b.     It is crucial that we provide a respected outlet for well-powered non-significant findings. This will help combat the file drawer effect and reduce the publication of false discoveries. Moreover, authors are welcome to propose superior alternatives to conventional null hypothesis testing, such as Bayesian approaches [20].

c.     By guaranteeing publication prior to data being collected, this model would encourage authors to propose large scale studies for more definitive hypothesis testing – studies which investigators would otherwise be reluctant to pursue given the risk of yielding unpublishable null effects.

d.     Registration review will be stringent, with reviewers asked to consider the methodology in detail for possible oversights and flaws that could prevent the study from testing the proposed hypotheses.

2.     It all sounds too strict. Why would authors submit to this scheme when they can’t change even one small aspect of their experimental procedure without being ‘summarily rejected’? Even grant applications are not so demanding.

a.     Yes it is stringent, and so it should be. This format of article is primarily intended for well-prepared scientists who have carefully considered their methodology and hypotheses in advance. And isn’t that how we ought to be doing science most of the time anyway?

b.     Note that the strict methodological stringency is coupled with a complete lack of expectation of how the results should look. Whether an experiment supports the stated hypothesis is the one aspect of science that scientists (should) have no control over – yet the traditional publishing model encourages a host of dodgy practices to exert such control. This new model replaces the artificial and counterproductive ‘data stringency’ with constructive ‘methodological stringency’, and so would largely eliminate the pressure for scientists to submit data that perfectly fit their predictions or confirm someone’s theory. I believe many scientists would approach this model with relief rather than trepidation.

3.     Authors could game the system by running a complete study as per usual and submitting the methodology for registration review after the fact.

a.     No, raw data must be made freely available at the full review stage and time stamped for inspection, along with a laboratory log indicating that data collection took place between dates X and Y. Final submission must also be accompanied by a certification from each author that no data (other than approved pilot data) was collected prior to the date of IPA. Any violation of this rule would be considered misconduct; the article would be retracted by Cortex and referred to Retraction Watch.

4.     What’s to stop unscrupulous reviewers stealing my ideas at the registration stage, running the experiments faster than I can (or rejecting my registration submission outright to buy time), and then publishing their own study?

a.     This is a legitimate worry, and it is true that there is no perfect defense against bad practice. But we shouldn’t overstate this concern. Gazumping is rare and, in any case, is present in many areas of science. Fear of being scooped doesn’t stop us presenting preliminary data at conferences or writing grant applications. So why should we be so afraid of registration review?

b.     Even if an unscrupulous reviewer decided to run a similar/identical experiment following IPA, the decision to publish would not be influenced. So being scooped would not cost the authors a publication once the authors pass IPA.

c.     Unlike existing protocol journals, such as BMC Protocols, the IPA submission would not be published in advance of the main paper. So only the reviewers and editors would see it. This will reduce the chances of being gazumped.

5.     A lot of the most interesting discoveries in science are serendipitous. Your approach will stifle creativity and data exploration.

a.     No, it won’t. Authors will be allowed to include “post-hoc analyses” in the manuscript that were not in the registered submission. They simply won’t be able to pretend that such analyses were planned in advance or adjust their hypotheses to predict unexpected outcomes. And, sensibly, they won’t be able to base the conclusions of their study on the outcome of unplanned analyses – the original registered analyses would take precedence and must also be reported.

b.     It should also be noted that a priori analyses in the registration stage could include exploration of possible serendipitous findings.

c.     Serendipitous findings are, by their nature, rare. A far greater problem is the proliferation of false positives due to excessive post-hoc flexibility in analysis approaches. So let’s deal with the big problem first.

6.     You propose allowing authors to alter the Introduction to include new literature. Doesn’t this create a slippery slope for changing the rationale or hypotheses too?

a.     No, but we must be vigilant on this point. I think it is entirely sensible to allow revisions to the Introduction to contextualise the literature based on the findings and to focus on most recent publications that emerged following IPA. After all, we want readers to be engaged as well as informed. However, we must also ensure that such changes are reasonable. Monitoring this aspect in particular would be one of the central reviewing criteria at Stage 2 (see above). In a revised Introduction, the authors would not be permitted to alter the rationale for the study, to state new hypotheses, or to alter the existing hypotheses. These could be flagged in distinct sections of the Introduction that are untouchable following IPA.

7.     What if the authors never submit a final manuscript because the results disagree with some desired outcome (such as supporting their preferred explanation)? How can you prevent publication bias on the part of the authors?

a.     We can’t stop authors censoring themselves. As noted above, however, if a study is withdrawn following IPA then this will be noted in a Retracted Registrations section of the journal. So there would at least be a public record of the withdrawal and some explanation for why it happened.

b.     Note also that if the authors have not submitted by their own stated deadline then the manuscript will be automatically withdrawn, considered retracted, and noted in the Retracted Registrations section. Extensions to the deadline are permissible following prior agreement with the action editor.

8.     What would stop authors getting IPA, then running many more subjects than proposed and selectively including only the ones that support their desired hypothesis?

a.     Nothing. But doing so is outright fraud, similar to the conduct of Dirk Smeesters [21]. No mechanism can fully guard against fraud, and regular submissions under the traditional publishing route are equally vulnerable to such misbehaviour. Note also that the proposed model requires submission of raw data, which will help protect against such eventualities. Selective exclusion of subjects to attain statistical significance can be detected using the statistical methods developed by Uri Simonsohn [22]. This alone will act as a significant deterrent to fraudsters.

9.     How can IPA be guaranteed without knowing the author’s interpretation of the findings?

a.     It isn’t. IPA ensures that the article cannot, and will not, be rejected based on the results themselves (with the exception of failing outcome-neutral reality checks, such as floor or ceiling effects, which prevent the stated hypotheses being appropriately tested). Manuscripts can still be rejected if the reviewers and editor believe the author’s interpretation is unreasonable given the data. And they will be rejected summarily if the authors change their experimental procedures in any way following IPA.

10.  What if the authors obtain IPA but then realise (after data collection commenced) that part of their proposed methods or analyses were incorrect or suboptimal?

a.     In the case of changes to the experimental procedures, the manuscript would have to be fully withdrawn but could be returned to Stage 1 for fresh registration review.

b.     In this case of changes to the analysis approach, depending on the nature of the proposed change, Stage 2 may be able to proceed following a phase of interim review and discussion with the editor and reviewers (if all agree that a different form of analysis is preferable). In such cases, the original proposed analysis would still be published in the final article but may not be reported, and the reasons for excluding it would be acknowledged.

11.  Cortex already has a long backlog of in-press articles. Adding yet another article format could make this problem worse.

a.     I propose that each article published as a Registered Report takes the place of a standard research report, thus requiring similar journal space to the current model.

b.     If registered reports become increasingly popular and well cited, the journal could gradually phase the standard report format out altogether, making registration reports the norm.

I hope I can convince you that Registration Reports would be a useful and valid initiative at Cortex. And even if not, I look forward to the ensuing discussion. Below is a list of key supporting references.

---

[1] Rosenthal R (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86: 638–641.

[2] Thornton A & Lee P (2000). Publication bias in meta-analysis: its causes and consequences

Journal of Clinical Epidemiology, 53: 207–216.

[3] Chase, LJ & Chase, RB (1976). A statistical power analysis of applied psychological research. Journal of Applied Psychology, 61: 234-237.

[4] Tressoldi, PE (2012). Replication unreliability in psychology: elusive phenomena or "elusive" statistical power? Frontiers in Psychology, 3: 218.

[5] Simmons JP, Nelson LD, and Simonsohn U. (2011). False-positive psychology: Undisclosed

flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22: 359-66.

[6] Wagenmakers, EJ (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14: 779–804.

[7] Masicampo, EJ & Lalande, DR (in press). A peculiar prevalence of p values just below .05. Quarterly Journal of Experimental Psychology.

[8] Ioannidis JPA (2005). Why Most Published Research Findings Are False. PLoS Medicine 2(8): e124. doi:10.1371/journal.pmed.0020124

[9] John, L, Loewenstein, G, & Prelec, D (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23: 524-532 DOI: 10.1177/0956797611430953

[10] Smith MB (1956). Editorial. Journal of Abnormal & Social Psychology, 52:1-4.

[11] Cohen, J (1962). The statistical power of abnormal – social psychological research: A review. Journal of Abnormal & Social Psychology, 65, 145‐153.

[12] http://openscienceframework.org/

[13] http://occamstypewriter.org/scurry/2012/09/05/key-questions-for-open-access-policy-in-the-uk

[14] Fang, FC, Steen, RG & Casadevalld, A. (2012) Misconduct accounts for the majority of retracted scientific publications. Proceedings of the National Academy of Sciences USA: 10.1073/pnas.1212247109

[15] Lane, DM & Dunlap, WP (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31: 107‐112.

[16] Hedges LV & Vevea, JL (1996). Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics, 21: 299-332.

[17] Strube, MJ (2006). SNOOP: A program for demonstrating the consequences of premature and repeated null hypothesis testing. Behavior Research Methods, 38: 24-27. Software available from here: http://www.artsci.wustl.edu/~socpsy/Snoop.7z

[18] Nosek, B. A., Spies, J. R., & Motyl, M. (in press). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science. arxiv.org/pdf/1205.4251

[19] Ritchie SJ, Wiseman R, French CC (2012) Failing the Future: Three Unsuccessful Attempts to Replicate Bem’s ‘Retroactive Facilitation of Recall’ Effect. PLoS ONE 7(3): e33423. doi:10.1371/journal.pone.0033423

[20] Kruschke, JK (in press). Bayesian estimation supercedes the t test. Journal of Experimental Psychology: General. www.indiana.edu/~kruschke/BEST/BEST.pdf

[21] http://blogs.discovermagazine.com/notrocketscience/2012/06/26/why-a-new-case-of-misconduct-in-psychology-heralds-interesting-times-for-the-field/

[22] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2114571&http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2114571