Tuesday 19 March 2019

The battle for reproducibility over storytelling in cognitive neuroscience


Here is my twitter thread on our upcoming Discussion Forum on reproducibility in cognitive neuroscience at the journal Cortex. I've posted it to my blog because, weirdly, it appears on twitter to be broken on some browsers (but not others!) To see it on twitter, start here.

--------------

A late-night thread on reproducibility and in cognitive neuroscience, including our upcoming series of (rather punchy) comment pieces at the journal Cortex. Gather round all ye.

Here is my editorial introducing the seven commentaries. I’m going to move through each of them here in turn, and stick around to the end of the thread to hear about two new initiatives we’re launching this year in response /1

First up, Huber et al . report how they tried to replicate a study published in . After being invited beforehand to run & submit the study by one editor, a different editor then desk rejected them once the (non-replication) results were in. /2
 
Sidebar: we later published Huber et al’s replication study at Cortex (thanks , we’re happy to help you out any time). You can read the paper here: /3

Good for them, but Huber & co believe the problem w/ replication in cog neurosci is deep & serious. They call for more stringent checks on reproducibility *before* publication & dynamic tracking of rep attempts & outcomes. Their full comment here: /4

Next, pushes back a little at the suggestion to select what gets published based on results, even when doing so is based on replicability. Instead he calls for a “pending replication” stamp to be placed on unverified exploratory studies /5

But wait...what about the tools we’re using? argues that the reliability of our research cannot exceed the reliability of the methods we employ. And in cognitive neuroscience this is poorly understood. It's not just about publication culture. /6

Nevertheless the often obstructive nature of peer review isn’t terribly helpful. weighs in to point out the value of adversarial collaborations for reducing bias & encouraging better theory, especially when submitted as Registered Reports /7

Do reforms to how science works take into account the scientists who DO the work – the early career researchers? & argue that unless reforms work for ECRs, they will fail. M&T suggest “replication & extension” as one solution /8

But it’s not all about incentives. calls for cognitive neuroscientists to rise above their egos and fallibilities, embrace error correction & champion reproducibility over reputation. And he is someone who practices what he preaches /9

In particular, you can read ’s recent Registered Report at Cortex where he tests the reproducibility of one of his own previous findings & concludes that the original result may be a false positive Almost nobody ever does this in cogneuro. /10 

And finally, , a former editor, takes on the newsroom culture of sci publishing. Huber et al.’s fixes will help but only superficially. To really fix these problems, he says, scientists need to take back control from publishers /11

Where does all this leave us? Cortex has been at the forefront of initiatives such as , Exploratory Reports, TOP guidelines & badges. But these are NOT enough and this year we’ll be launching two new initiatives. /12

The first is an Accountable Replications policy – ’s now famous "pottery barn rule" of publishing, which we recently introduced at Open Science. In a nutshell: if Cortex published the original study we’ll publish the replications of that study. /13

The second is an entirely new initiative, again the creation of : Verification Reports. Short articles with the sole purpose of testing the reproducibility & robustness of original studies using the exact SAME data. /14

These steps aren’t a total answer but they move us in the right direction. The recent launch of – together with the wide support the network is receiving from funders, publishers & regulators – means that reproducibility is going to be a Big Deal for many years. /15

That’s why cognitive neuroscientists need to be at the forefront of those discussions. And it’s why cog neurosci journals need to work harder to support reproducibility. That means adopting , Exploratory Reports, TOP guidelines, replication initiatives & more. /16

I will end this very long thread there! Hope you enjoy the articles (which are all available as preprints in the tweets above) and thanks to all the wonderful contributors for weighing in. Onward. /end
Posted by at

1 comment:

Subscribe to: Post Comments (Atom)