In this special guest post, Rob McIntosh, associate editor at Cortex and long-time member of the Registered Reports editorial team, foreshadows a new article type that will celebrate scientific exploration in its native form.Exploratory Reports will launch next month and, now, we need your input to get it right.
Chris has kindly allowed me to
crash his blog, to publicise and to gather ideas and opinions for a new article
type at Cortex. The working name is Exploratory Reports. As far back as 2014,
in his witterings and twitterings, Chris
trailered a plan for Cortex to develop
a format for open-ended science, a kind of louche, relaxed half-cousin to the
buttoned-up and locked-down Registered
Reports. Easier tweeted than done. We are now preparing to launch this
brave new format, but even as we do so, we are still wrestling with some basic
questions. Does it have a worthwhile role to play in the publishing landscape? Can
it make a meaningful contribution to openness in science? What should its
boundaries and criteria be? And is there a better name than Exploratory Reports?
Visitors to this blog will have a more-than-nodding familiarity with misaligned incentives in science, with the ‘deadly sin’ of hidden flexibility, and with the damage done to reliability when research conducted in an open-ended, see-what-we-can-find way, is written into the record as a pre-planned test of specific hypotheses. No one doubts that exploratory research has a vital role to play in empirical discovery and hypothesis generation, nor that it can be rigorous and powerful (see recent blog discussions here and here). But severe problems can arise from a failure to distinguish between exploratory and confirmatory modes of enquiry, and most perniciously from the misrepresentation of exploratory research as confirmatory.
A major driver of this misrepresentation is the pervasive idealisation of hypothesis-testing, throughout our scientific training, funding agencies, and journals. Statistical confirmation (or disconfirmation) of prior predictions is inferentially stronger than the ‘mere’ delineation of interesting patterns, and top journals prefer neat packages of strong evidence with firm impactful conclusions, even if our actual science is often more messy and… exploratory. Given a more-or-less-explicit pressure to publish in a confirmatory mode, it is unsurprising that individual scientists more-or-less-wittingly resort to p-hacking, HARKing, and other ‘questionable research practices’.
Regulars of this blog will need no further education on such QRPs, or on the mighty and multi-pronged Open Science movement to reform them. Still less will you need reminding of the key role that study pre-registration can play by keeping researchers honest about what was planned in advance. Pre-registration does not preclude further exploration of the data, but it keeps this clearly distinct from the pre-planned aspects, eliminating p-hacking, HARKing, and several other gremlins, at a stroke. The promise of enhanced truth value earns pre-registered studies an Open Practices badge at a growing number of journals, and it has even been suggested that there should be an automatic bonus star in the UK Government’s Research Excellence Framework (where stars mean money).
This is fine progress, but it does little to combat the perceived pre-eminence of confirmatory research, one of the most distorting forces in our science. Indeed, a privileged status for pre-registered studies could potentially intensify the idealisation of the confirmatory mode, given that pre-registration is practically synonymous with a priori hypothesis testing. A complementary strategy would therefore be for journals to better value and serve more open-ended research, in which data exploration and hypothesis generation can take precedence over hypothesis-testing. A paper that is openly exploratory, which shows its working and shares its data, is arguably as transparent in its own way as a pre-registered confirmatory study. One could even envisage an Open Practices badge for explicitly exploratory studies.
Some journal editors may believe that it is typically inappropriate to publish exploratory work. But this is not the case at Cortex, where the field of study (brain-and-behaviour) is relatively uncharted, where many research questions are open-ended (e.g. What are the fMRI or EEG correlates of task X? What characterises patient group Y across test battery Z?), and where data collection is often costly because expensive technologies are involved or a rare or fleeting neuropsychological condition is studied. It is hard to estimate how much of the journal’s output is really exploratory because, whilst some authors have the confidence to make exploratory work explicit, others may still dress it in confirmatory clothing. If publication is their aim, then they are wise to do so, because the Action Editor or reviewers could be unsympathetic to an exploratory approach.
Hence, a new article type for exploratory science, where pattern-finding and hypothesis generation are paramount, and where the generative value of a paper can even outweigh its necessary truth value. A dedicated format is a commitment to the centrality of exploratory research in discovery. It also promotes transparency, because the incentives to misrepresentation are reduced, and the claims and conclusions can be appropriate to the methods. Some exploratory work might provide strong enough evidence to boldly assert a new discovery, but most will make provisional cases, seeding testable hypotheses and predictions for further (confirmatory) studies. The main requirements are that the work should be rigorous, novel, and generative.
Or that is the general idea. The devil, as ever, is in the detail. Will scientists – as authors, reviewers and readers - engage with the format? What should exploratory articles look like, and can we define clear guidelines for such an open-ended and potentially diverse format? How do we exercise the quality control to make this a high-status format of value to the field, not a salvage yard for failed experiments, or a soapbox for unfettered speculation? Below, a few of the questions keeping us awake at night are unpacked a little further. Your opinions and suggestions on these questions, and any aspect of this venture, would be most welcome.
1. Scope of the format. At the most restrictive end, the format would be specific for studies that take an exploratory approach to open-ended questions. Less restrictive definitions might allow for experimental work with no strong a priori predictions, or even for experiments that had prior predictions but in which the most interesting outcomes were unanticipated. At the most inclusive end, any research might be eligible that was willing to waive all claims dependent upon pre-planning. Are there clear boundaries that can be drawn?
2. Exploration and review. A requirement for submission to this format will be that the full data are uploaded at the point of submission, sufficient to reproduce the analyses reported. To what extent should reviewers, with access to the data, be allowed to recommend/insist that further analyses, of their own suggestion, should be included in the final paper?
3. Statistical standards. Conventional significance testing is arguably meaningless in the exploratory mode, and it has even been suggested that this format should have no p-values at all. There will be a strong emphasis on clear data visualisation, showing (where feasible) complete observations. But some means of quantifying the strength of apparent patterns will still be required, and it may be just too radical to exclude p values altogether. When using conventional significance testing, should more stringent criteria for suggestive and significant evidence be used? More generally, what statistical recommendations would you make for this format, and what reporting standards should be required (e.g. confidence intervals, effect sizes, adjusted and non-adjusted coefficients etc.)?
4. Evidence vs. theory. Ideally, a good submission presents a solid statistical case from a large dataset, generating novel hypotheses, making testable predictions. The reality is often liable to be more fragmentary (e.g. data from rare neuropsychological patients may be limited, and not easily increased). Can weaker evidence be acceptable in the context of a novel generative theoretical proposal, provided that the claims do not exceed the data?
5. The name game. The working title for this format has been Exploratory Reports. The ambition is to ‘reclaim’ the term ‘exploratory’ from a slightly pejorative sense it has acquired in some circles. Let’s make exploration great again! But does this set up too much of an uphill struggle to make this a high-status format; and is there anyway a better, fresher term (Discovery Reports; Open Research)?
Visitors to this blog will have a more-than-nodding familiarity with misaligned incentives in science, with the ‘deadly sin’ of hidden flexibility, and with the damage done to reliability when research conducted in an open-ended, see-what-we-can-find way, is written into the record as a pre-planned test of specific hypotheses. No one doubts that exploratory research has a vital role to play in empirical discovery and hypothesis generation, nor that it can be rigorous and powerful (see recent blog discussions here and here). But severe problems can arise from a failure to distinguish between exploratory and confirmatory modes of enquiry, and most perniciously from the misrepresentation of exploratory research as confirmatory.
A major driver of this misrepresentation is the pervasive idealisation of hypothesis-testing, throughout our scientific training, funding agencies, and journals. Statistical confirmation (or disconfirmation) of prior predictions is inferentially stronger than the ‘mere’ delineation of interesting patterns, and top journals prefer neat packages of strong evidence with firm impactful conclusions, even if our actual science is often more messy and… exploratory. Given a more-or-less-explicit pressure to publish in a confirmatory mode, it is unsurprising that individual scientists more-or-less-wittingly resort to p-hacking, HARKing, and other ‘questionable research practices’.
Regulars of this blog will need no further education on such QRPs, or on the mighty and multi-pronged Open Science movement to reform them. Still less will you need reminding of the key role that study pre-registration can play by keeping researchers honest about what was planned in advance. Pre-registration does not preclude further exploration of the data, but it keeps this clearly distinct from the pre-planned aspects, eliminating p-hacking, HARKing, and several other gremlins, at a stroke. The promise of enhanced truth value earns pre-registered studies an Open Practices badge at a growing number of journals, and it has even been suggested that there should be an automatic bonus star in the UK Government’s Research Excellence Framework (where stars mean money).
This is fine progress, but it does little to combat the perceived pre-eminence of confirmatory research, one of the most distorting forces in our science. Indeed, a privileged status for pre-registered studies could potentially intensify the idealisation of the confirmatory mode, given that pre-registration is practically synonymous with a priori hypothesis testing. A complementary strategy would therefore be for journals to better value and serve more open-ended research, in which data exploration and hypothesis generation can take precedence over hypothesis-testing. A paper that is openly exploratory, which shows its working and shares its data, is arguably as transparent in its own way as a pre-registered confirmatory study. One could even envisage an Open Practices badge for explicitly exploratory studies.
Some journal editors may believe that it is typically inappropriate to publish exploratory work. But this is not the case at Cortex, where the field of study (brain-and-behaviour) is relatively uncharted, where many research questions are open-ended (e.g. What are the fMRI or EEG correlates of task X? What characterises patient group Y across test battery Z?), and where data collection is often costly because expensive technologies are involved or a rare or fleeting neuropsychological condition is studied. It is hard to estimate how much of the journal’s output is really exploratory because, whilst some authors have the confidence to make exploratory work explicit, others may still dress it in confirmatory clothing. If publication is their aim, then they are wise to do so, because the Action Editor or reviewers could be unsympathetic to an exploratory approach.
Hence, a new article type for exploratory science, where pattern-finding and hypothesis generation are paramount, and where the generative value of a paper can even outweigh its necessary truth value. A dedicated format is a commitment to the centrality of exploratory research in discovery. It also promotes transparency, because the incentives to misrepresentation are reduced, and the claims and conclusions can be appropriate to the methods. Some exploratory work might provide strong enough evidence to boldly assert a new discovery, but most will make provisional cases, seeding testable hypotheses and predictions for further (confirmatory) studies. The main requirements are that the work should be rigorous, novel, and generative.
Or that is the general idea. The devil, as ever, is in the detail. Will scientists – as authors, reviewers and readers - engage with the format? What should exploratory articles look like, and can we define clear guidelines for such an open-ended and potentially diverse format? How do we exercise the quality control to make this a high-status format of value to the field, not a salvage yard for failed experiments, or a soapbox for unfettered speculation? Below, a few of the questions keeping us awake at night are unpacked a little further. Your opinions and suggestions on these questions, and any aspect of this venture, would be most welcome.
1. Scope of the format. At the most restrictive end, the format would be specific for studies that take an exploratory approach to open-ended questions. Less restrictive definitions might allow for experimental work with no strong a priori predictions, or even for experiments that had prior predictions but in which the most interesting outcomes were unanticipated. At the most inclusive end, any research might be eligible that was willing to waive all claims dependent upon pre-planning. Are there clear boundaries that can be drawn?
2. Exploration and review. A requirement for submission to this format will be that the full data are uploaded at the point of submission, sufficient to reproduce the analyses reported. To what extent should reviewers, with access to the data, be allowed to recommend/insist that further analyses, of their own suggestion, should be included in the final paper?
3. Statistical standards. Conventional significance testing is arguably meaningless in the exploratory mode, and it has even been suggested that this format should have no p-values at all. There will be a strong emphasis on clear data visualisation, showing (where feasible) complete observations. But some means of quantifying the strength of apparent patterns will still be required, and it may be just too radical to exclude p values altogether. When using conventional significance testing, should more stringent criteria for suggestive and significant evidence be used? More generally, what statistical recommendations would you make for this format, and what reporting standards should be required (e.g. confidence intervals, effect sizes, adjusted and non-adjusted coefficients etc.)?
4. Evidence vs. theory. Ideally, a good submission presents a solid statistical case from a large dataset, generating novel hypotheses, making testable predictions. The reality is often liable to be more fragmentary (e.g. data from rare neuropsychological patients may be limited, and not easily increased). Can weaker evidence be acceptable in the context of a novel generative theoretical proposal, provided that the claims do not exceed the data?
5. The name game. The working title for this format has been Exploratory Reports. The ambition is to ‘reclaim’ the term ‘exploratory’ from a slightly pejorative sense it has acquired in some circles. Let’s make exploration great again! But does this set up too much of an uphill struggle to make this a high-status format; and is there anyway a better, fresher term (Discovery Reports; Open Research)?