Many readers will be aware of Sanjay Srivastava's 2012 proposal for a Pottery Barn rule to support replications at scientific journals. The basic idea is that journals should commit to publishing replication attempts of previous studies published within the journal (i.e. "you publish it, you buy it"). As Sanjay wrote at the time: "Once a journal has published a study, it becomes responsible for publishing direct replications of that study. Publication is subject to editorial review of technical merit but is not dependent on outcome."
I have always felt that this was a brilliant idea - truly one of the most innovative and concrete proposals for not only increasing the volume of replication studies but also incentivizing journals to publish original work that is more replicable in the first place. It's a little disappointing that no journal has yet taken up the policy.
That is about to change. Essi Viding and I have written a fully specified Accountable Replications Policy, to be implemented at the journal Royal Society Open Science where we both serve as editors.
We plan to capture the core features of Sanjay's proposal while also extending it in two ways.
First, the commitment to publish replications of previous work will include not only articles published within Royal Society Open Science but in more than 20 other major psychology and neuroscience journals.
Second, the Accountable Replications mechanism will be open to both completed replication studies (unlocking the file drawer of hitherto unpublished replications) AND to proposals for yet-to-be completed replications. In the former case, the manuscripts will be initially stripped of results and reviewed "results-blind". In the latter case, submissions will be assessed via the journal's existing Registered Report route. In both cases, the decision to publish will be based on peer review before results are known, immunising against publication bias.
Our draft policy has been approved by the journal editorial board and Royal Society. Drafting it was a tricky process and we are now seeking to further improve it based on your views. Is the policy fit for purpose? Is it clear? Have we missed any risks or opportunities? Would you use it? Please let us know what you think in the comments below, on social media, or by email.
______________
Policy Overview
We believe that when a journal publishes an empirical study it assumes a degree of accountability for the replicability of that study. This responsibility is reflected by the extent to which the journal commits to publishing attempted replications of previous studies that appear within its pages.
The Accountable Replication Policy commits the Psychology and Cognitive Neuroscience section of Royal Society Open Science to publishing replications of studies previously published within the journal. Authors can either submit a replication study that is already completed or a proposal to replicate a previous study. To ensure that the review process is unbiased by the results, submissions will be reviewed with existing results initially redacted (where applicable), or in the case of study proposals, before the results exist. Submissions that report close, clear and valid replications of the original methodology will be offered in principle acceptance, which virtually guarantees publication of the replication regardless of the study outcome.
The diagram below illustrates the editorial process and manuscript handling pipeline (and see here for a higher resolution image).
Detailed Policy
1. Royal
Society Open Science commits to publishing any methodologically sound
attempt to replicate any study that has been published within the Psychology
and Cognitive Neuroscience section of the journal, as well as any study within
the remit of the Psychology and Cognitive Neuroscience section that has been
published in any one of the following journals: Biological Psychiatry, Brain,
Cerebral Cortex, Cortex, Cognitive
Psychology, Cognition, Current Biology, European Journal of Neuroscience, Journal of Cognitive Neuroscience, any journal within the Journal of Experimental Psychology
group, Journal of Experimental Social
Psychology, Journal of Neuroscience,
Journal of Personality and Social
Psychology, Nature, Nature Neuroscience, NeuroImage, Neuron, PLOS Biology, Proceedings of the National Academy
of Sciences of the USA, Psychological
Science, Science, Social
Psychological and Personality Science, and any journal published by the
Royal Society. We will also consider, on a case-by-case basis, pre-submission
enquiries for replications of studies published in journals additional to those
listed here.
2. Under this policy, authors have the choice
between two routes for consideration of submissions: a “results-blind” track in which the
introduction and method of a completed study are peer reviewed while the
results and discussion are temporarily withheld, or the Registered Reports track where the study protocol
is reviewed and pre-accepted before the commencement of data collection.
3. Results-blind track
3.1. This
track is intended for authors who have already completed a direct (close)
replication attempt of a study published within Royal Society Open Science, within one of the source journals
listed in section 1, or in any other journal approved on a case-by-case basis by
the editors. Authors should initially submit a partial manuscript containing
the abstract, introduction and method sections, with the results and discussion
redacted together with any other information (e.g. in the abstract) that states
or implies the research outcomes. The method section must include a
comprehensive and detailed account of all experimental procedures (including
data inclusion and exclusion criteria) and analyses undertaken. The later Stage
2 submission cannot include the outcomes of any analyses that are not stated in
the Stage 1 submission at the point of in principle acceptance. The Stage 1
manuscript should be submitted under the standard research article category.
3.2. At
Stage 1 the submission is considered by reviewers who assess the content
according to the following Primary
and Secondary criteria:
Stage 1 Primary Criterion #1: Whether
the authors provide a sufficiently clear and detailed description of the
methods for another researcher to replicate exactly the proposed experimental
procedures and analysis pipeline, and to prevent undisclosed flexibility in the
experimental procedures or analysis pipeline.
Stage 1 Primary Criterion #2: Whether the manuscript describes a sufficiently valid (i.e. close) and robust (e.g. statistically powerful) replication of the original study methods and rationale to provide an indication of replicability.
Stage 1 Secondary Criterion #1: The logic, rationale, and plausibility of the proposed hypotheses.
Stage 1 Secondary Criterion #2: The soundness of the methodology and analysis pipeline.
Stage 1 Secondary Criterion #3: Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the results obtained are able to test the stated hypotheses.
Stage 1 Primary Criterion #2: Whether the manuscript describes a sufficiently valid (i.e. close) and robust (e.g. statistically powerful) replication of the original study methods and rationale to provide an indication of replicability.
Stage 1 Secondary Criterion #1: The logic, rationale, and plausibility of the proposed hypotheses.
Stage 1 Secondary Criterion #2: The soundness of the methodology and analysis pipeline.
Stage 1 Secondary Criterion #3: Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the results obtained are able to test the stated hypotheses.
Of the above Stage 1 criteria, editorial decisions will
be based solely on the Primary Criteria. In some cases, the published methods
of the original study may not provide enough detail to permit a close
replication (Primary Criterion #1), and the original authors of the study may
also be unwilling or unable to provide the necessary missing information. In
such cases, appropriate estimates of the original procedures will be accepted
provided the authors can establish that the chosen method is a reasonable
estimation of the original approach, and provided they can supply documentary
evidence (such as emails) proving any attempts to obtain such information from
the original authors.
With the exception of one specific scenario discussed below, reviewers’ comments regarding the Secondary Criteria will not influence the decision to publish. Instead, where the editors believe that any critical issues raised within these categories are valid, additional actions may be required. The specific actions will depend on the severity of the concerns, assessed according to three ‘threat levels’: minor, major, and severe. To prevent publication bias, these assessments will be made before results are known.
For concerns judged by the editors to be minor, authors will be required to discuss the necessary limitations in the introduction (and/or later in the discussion at Stage 2).
For major concerns that substantially limit (but do not eliminate) the interpretability of any potential findings, an editorial cautionary note may also be published alongside the final report. For instance, where reviewers identify a flaw with the methodology of the original study under replication (Stage 1 Secondary Criterion #1 or #2), then in addition to a mandatory discussion of the limitation by the authors, an editorial comment may be published with the final paper highlighting the limitation of the original work. In such cases, authors would also be encouraged (but not required) to pre-register an additional replication study through the Registered Reports track, in which the procedure is matched as closely as possible to the original study but with the flaw corrected. The same author requirements and options would apply in cases where the replication lacks an appropriate positive control or other independent quality check (Stage 1 Secondary Criterion #3).
For severe concerns that prevent meaningful interpretation of the results, the course of action will be determined by the source publication of the original study. For studies published in journals other than Royal Society Open Science, Stage 1 submissions with fatal flaws will be accepted only if the authors agree to pre-register (via the Registered Reports track) an additional corrective study within the same paper that attempts to correct the identified problems. In cases where authors are unwilling or unable to do so, such Stage 1 submissions are likely to be rejected. However, to ensure that Royal Society Open Science remains accountable for the standard of work that it publishes, replication attempts even of severely flawed studies that were originally published within the journal will be permitted without the requirement for authors to pre-register an additional corrective study. Instead, as with submissions that are judged to contain major (but not severe) concerns, authors will be required to include a discussion of the fatal flaw, and the final article will be accompanied by a cautionary editorial note.
These contingencies are summarised in the table below (and see also the workflow diagam above).
With the exception of one specific scenario discussed below, reviewers’ comments regarding the Secondary Criteria will not influence the decision to publish. Instead, where the editors believe that any critical issues raised within these categories are valid, additional actions may be required. The specific actions will depend on the severity of the concerns, assessed according to three ‘threat levels’: minor, major, and severe. To prevent publication bias, these assessments will be made before results are known.
For concerns judged by the editors to be minor, authors will be required to discuss the necessary limitations in the introduction (and/or later in the discussion at Stage 2).
For major concerns that substantially limit (but do not eliminate) the interpretability of any potential findings, an editorial cautionary note may also be published alongside the final report. For instance, where reviewers identify a flaw with the methodology of the original study under replication (Stage 1 Secondary Criterion #1 or #2), then in addition to a mandatory discussion of the limitation by the authors, an editorial comment may be published with the final paper highlighting the limitation of the original work. In such cases, authors would also be encouraged (but not required) to pre-register an additional replication study through the Registered Reports track, in which the procedure is matched as closely as possible to the original study but with the flaw corrected. The same author requirements and options would apply in cases where the replication lacks an appropriate positive control or other independent quality check (Stage 1 Secondary Criterion #3).
For severe concerns that prevent meaningful interpretation of the results, the course of action will be determined by the source publication of the original study. For studies published in journals other than Royal Society Open Science, Stage 1 submissions with fatal flaws will be accepted only if the authors agree to pre-register (via the Registered Reports track) an additional corrective study within the same paper that attempts to correct the identified problems. In cases where authors are unwilling or unable to do so, such Stage 1 submissions are likely to be rejected. However, to ensure that Royal Society Open Science remains accountable for the standard of work that it publishes, replication attempts even of severely flawed studies that were originally published within the journal will be permitted without the requirement for authors to pre-register an additional corrective study. Instead, as with submissions that are judged to contain major (but not severe) concerns, authors will be required to include a discussion of the fatal flaw, and the final article will be accompanied by a cautionary editorial note.
These contingencies are summarised in the table below (and see also the workflow diagam above).
Threat level of methodological concerns
under Stage 1 Secondary Criteria #1, #2 or #3
|
||||
Minor
|
Major
|
Severe
|
||
Original
study published in Royal Society Open
Science
|
Mandatory discussion of the concern in
final paper by authors
|
✓
|
✓
|
✓
|
Publication of cautionary note by editors
|
✘
|
✓
|
✓
|
|
Authors encouraged (but not required) to
pre-register an additional corrective replication attempt
|
✘
|
✓
|
✓
|
|
Authors required to conduct a corrective
replication attempt to achieve Stage 1 acceptance
|
✘
|
✘
|
✘
|
|
Original study not published in Royal
Society Open Science
|
Mandatory discussion of the concern in
final paper by authors
|
✓
|
✓
|
✓
|
Publication of cautionary note by editors
|
✘
|
✓
|
✓
|
|
Authors encouraged but not required to
pre-register an additional corrective replication attempt
|
✘
|
✓
|
✘
|
|
Authors required to conduct a corrective
replication attempt to achieve Stage 1 acceptance
|
✘
|
✘
|
✓
|
|
Other than the scenario described above for severe
concerns, Stage 1 manuscripts will be rejected outright only if reviews
indicate that the replication attempt failed to meet sufficient methodological
standards, such as insufficient statistical power or critical deviations from
the original methodology (Stage 1 Primary Criteria #1 and #2). No fixed level
of statistical power is required, but will be assessed on an individual basis
before results are known. Authors may also be required to revise their Stage 1
submission to add necessary detail to the reported methodology or to alter the
analyses. The description of methods should be sufficiently elaborate to
provide an internally complete account of all employed methods, without the
reader needing to examine the specific methodological details of the original
study.
If the manuscript is reviewed favourably according to the Stage 1 Primary Criteria then it will be awarded in principle acceptance (IPA). Authors are then invited to submit a Stage 2 manuscript that includes the content of the approved Stage 1 manuscript plus the results and discussion.
3.3. At
Stage 2 the manuscript is returned to the original Stage 1 reviewers who assess
the content according to the following Primary
and Secondary criteria:
Stage 2 Primary Criterion #1: Whether
the introduction and methods are the same as the approved Stage1 submission
(required).
Stage 2 Primary Criterion #2: Whether the authors’ conclusions are justified given the data.
Stage 2 Secondary Criterion #1: Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects or success of positive controls or other quality checks). Failure to pass these conditions will not lead to manuscript rejection but could require authors to explicitly acknowledge such limitations in their discussion, and in severe cases could also lead to the publication of an accompanying editorial cautionary note.
Stage 2 Primary Criterion #2: Whether the authors’ conclusions are justified given the data.
Stage 2 Secondary Criterion #1: Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects or success of positive controls or other quality checks). Failure to pass these conditions will not lead to manuscript rejection but could require authors to explicitly acknowledge such limitations in their discussion, and in severe cases could also lead to the publication of an accompanying editorial cautionary note.
Of the above
Stage 2 criteria, editorial decisions will be based solely on the Primary
Criteria. Comments regarding Secondary Criterion #1 will not influence
the decision to publish but could require additional caveats by the authors (in
the discussion) or may be noted by the editors in an accompanying cautionary
note.
Stage 2 manuscripts will be rejected only where the submission includes unapproved changes to the approved Stage 1 manuscript or where the authors’ conclusions are judged to extend beyond those permitted by the results, AND where any such differences of opinion between the authors and reviewers/editors cannot be reconciled.
Stage 2 manuscripts will be rejected only where the submission includes unapproved changes to the approved Stage 1 manuscript or where the authors’ conclusions are judged to extend beyond those permitted by the results, AND where any such differences of opinion between the authors and reviewers/editors cannot be reconciled.
4.
Registered Reports track
4.1. This track is similar to the results-blind process but intended for authors who have yet to attempt a replication of a study published within Royal Society Open Science, within one of the source journals listed in section 1, or any other journal approved by the editors. Authors should initially submit a partial manuscript through the Registered Reports article type containing the abstract, introduction and method sections, written in the future tense. In all other respects, the options and requirements are the same as for the results-blind track.
4.2. The
Stage 1 and Stage 2 submissions will assessed according to the same Primary and
Secondary Criteria as for the results-blind track (see 3.2 and 3.3 above)
4.3. The
final Stage 2 paper will be published as a Registered Report.
No comments:
Post a Comment