Here is my twitter thread on our upcoming Discussion Forum on reproducibility in cognitive neuroscience at the journal Cortex. I've posted it to my blog because, weirdly, it appears on twitter to be broken on some browsers (but not others!) To see it on twitter, start here.
--------------
A late-night thread on reproducibility and
Here is my editorial introducing the seven commentaries. I’m going to move through each of them here in turn, and stick around to the end of the thread to hear about two new initiatives we’re launching this year in response https://psyarxiv.com/shryx /1
First up, Huber et al . report how they tried to replicate a study published in @NatureNeuro. After being invited beforehand to run & submit the study by one editor, a different editor then desk rejected them once the (non-replication) results were in. /2
Sidebar: we later published Huber et al’s replication study at Cortex (thanks @NatureNeuro, we’re happy to help you out any time). You can read the paper here: https://drive.google.com/file/d/1PhWHaAwO2BK8cVu0VASJS__txzYlMDEY/view?usp=sharing … /3
Good for them, but Huber & co believe the problem w/ replication in cog neurosci is deep & serious. They call for more stringent checks on reproducibility *before* publication & dynamic tracking of rep attempts & outcomes. Their full comment here: https://psyarxiv.com/5q9ma/ /4
Next, @sampendu pushes back a little at the suggestion to select what gets published based on results, even when doing so is based on replicability. Instead he calls for a “pending replication” stamp to be placed on unverified exploratory studies https://psyarxiv.com/h49a6 /5
But wait...what about the tools we’re using? @m_wall argues that the reliability of our research cannot exceed the reliability of the methods we employ. And in cognitive neuroscience this is poorly understood. It's not just about publication culture. https://psyarxiv.com/upynr/ /6
Nevertheless the often obstructive nature of peer review isn’t terribly helpful. @HannahMBuxton
weighs in to point out the value of adversarial collaborations for
reducing bias & encouraging better theory, especially when submitted
as Registered Reports https://psyarxiv.com/82sr3/ /7
Do reforms to how science works take into account the scientists who DO the work – the early career researchers? @LeahMaizey & @LTzavella argue that unless reforms work for ECRs, they will fail. M&T suggest “replication & extension” as one solution https://psyarxiv.com/dzsh4 /8
But it’s not all about incentives. @minzlicht calls for cognitive neuroscientists to rise above their egos and fallibilities, embrace error correction & champion reproducibility over reputation. And he is someone who practices what he preaches https://psyarxiv.com/8zvc3 /9
In particular, you can read @minzlicht’s
recent Registered Report at Cortex where he tests the reproducibility
of one of his own previous findings & concludes that the original
result may be a false positive https://osf.io/473kd/
Almost nobody ever does this in cogneuro. /10
And finally, @NeuroMinded, a former @NatureNeuro
editor, takes on the newsroom culture of sci publishing. Huber et al.’s
fixes will help but only superficially. To really fix these problems,
he says, scientists need to take back control from publishers https://psyarxiv.com/d59me /11
Where does all this leave us? Cortex has been at the forefront of #openscience initiatives such as @RegReports, Exploratory Reports, TOP guidelines & #openscience badges.
But these are NOT enough and this year we’ll be launching two new initiatives. /12
The first is an Accountable Replications policy – @hardsci’s now famous "pottery barn rule" of publishing, which we recently introduced at @RoyalSociety Open Science. In a nutshell: if Cortex published the original study we’ll publish the replications of that study. /13
The second is an entirely new initiative, again the creation of @hardsci:
Verification Reports. Short articles with the sole purpose of testing
the reproducibility & robustness of original studies using the exact
SAME data. /14
These steps aren’t a total answer but they move us in the right direction. The recent launch of @ukrepro – together with the wide support the network is receiving from funders, publishers & regulators – means that reproducibility is going to be a Big Deal for many years. /15
That’s why cognitive neuroscientists need to be at the forefront of those discussions. And it’s why cog neurosci journals need to work harder to support reproducibility. That means adopting @RegReports, Exploratory Reports, TOP guidelines, replication initiatives & more. /16
I will end this very long thread there!
Hope you enjoy the articles (which are all available as preprints in the
tweets above) and thanks to all the wonderful contributors for weighing
in. Onward.
/end
