Guest post by Claudio Tennie: Why I am resigning as Associate Editor from Proceedings B today

The following is a guest post published at the request of my colleague, Dr Claudio Tennie, University of Tübingen.

 

I have always been fond of Royal Society Proceedings B. And yet, today I am resigning in protest as one of their Associate Editors. What happened? 

Earlier this year, a group of people, spearheaded by Dr. Ljerka Ostojic, approached Proceedings B with a well-versed request: that it should adopt Registered Reports. Yet, to our dismay, they declined to do so.

We are now very aware of the various replication crises in many fields. A lack of robust findings is not surprising, and is indeed the logical outcome, of the current system. To be blunt, this system actively selects for bad science. In order to (once again) explain how and why Registered Reports can drastically help this situation, a comparison between science and car crash testing might be helpful.

It is safe to say that none of us would like to live in a world where all cars are advertised as having five star crash test ratings, but where, in reality, many should really rate as zero stars. This would be the expected case in a world where car crash outcomes were measured by car makers and selected by car-sellers. Why? Because capitalistic forces would select both for invalid crash testing and biased crash test reporting. Allowing Registered Reports is the logical equivalent to checking crash test dummies before they are being used in car crashes and also then publicising all outcomes of all crash tests.

Likewise, in science, we want to know which hypotheses find support and which do not. And we want to use the best methods to arrive at these conclusions. Currently, we often use sub-ideal methods, which alongside the inherent biases towards publishing positive findings, selects for bad science. As a result, it is even not clear what proportion of positive findings within the suspiciously large mountain of positive findings are valid. The current situation is an absurd and truly unbearable situation – wasting time, money and energy galore. We urgently need to change it.

Of course, an especially efficient policy is to properly check crash test dummies pre-test; and to publish all crash test results. Registered reports creates exactly this situation for the scientific field. In Registered Reports, methods are properly checked before they are applied. And the eventual publication must report all results – and will be published regardless the specific outcomes. While this does not mean that every study can be a Registered Report – there are exceptions to the rule (see the FAQ section here) – many should be. As a result of this simple and compelling logic, the number of journals adopting Registered Reports is constantly increasing.

I was therefore very disappointed to witness Proceedings B refusing to adopt Registered Reports. Moreover they did so on the very unconvincing grounds that one of their sister journals (Royal Society Open Science) already allows them. The general problem persists with every (suitable) journal that refuses to allow Registered Reports. Proceedings B should adopt Registered Reports. But because they refuse to do so, I must protest. I am therefore resigning as an Associate Editor at Proceedings B.

Goodbye.

Claudio Tennie

The battle for reproducibility over storytelling in cognitive neuroscience

Here is my twitter thread on our upcoming Discussion Forum on reproducibility in cognitive neuroscience at the journal Cortex. I've posted it to my blog because, weirdly, it appears on twitter to be broken on some browsers (but not others!) To see it on twitter, start here.

--------------

A late-night thread on reproducibility and #openscience in cognitive neuroscience, including our upcoming series of (rather punchy) comment pieces at the journal Cortex. Gather round all ye.

Here is my editorial introducing the seven commentaries. I’m going to move through each of them here in turn, and stick around to the end of the thread to hear about two new initiatives we’re launching this year in response https://psyarxiv.com/shryx  /1

First up, Huber et al . report how they tried to replicate a study published in @NatureNeuro. After being invited beforehand to run & submit the study by one editor, a different editor then desk rejected them once the (non-replication) results were in. /2

 

Sidebar: we later published Huber et al’s replication study at Cortex (thanks @NatureNeuro, we’re happy to help you out any time). You can read the paper here: https://drive.google.com/file/d/1PhWHaAwO2BK8cVu0VASJS__txzYlMDEY/view?usp=sharing … /3

Good for them, but Huber & co believe the problem w/ replication in cog neurosci is deep & serious. They call for more stringent checks on reproducibility *before* publication & dynamic tracking of rep attempts & outcomes. Their full comment here: https://psyarxiv.com/5q9ma/  /4

Next, @sampendu pushes back a little at the suggestion to select what gets published based on results, even when doing so is based on replicability. Instead he calls for a “pending replication” stamp to be placed on unverified exploratory studies https://psyarxiv.com/h49a6  /5

But wait...what about the tools we’re using? @m_wall argues that the reliability of our research cannot exceed the reliability of the methods we employ. And in cognitive neuroscience this is poorly understood. It's not just about publication culture. https://psyarxiv.com/upynr/  /6

Nevertheless the often obstructive nature of peer review isn’t terribly helpful. @HannahMBuxton weighs in to point out the value of adversarial collaborations for reducing bias & encouraging better theory, especially when submitted as Registered Reports https://psyarxiv.com/82sr3/  /7

Do reforms to how science works take into account the scientists who DO the work – the early career researchers? @LeahMaizey & @LTzavella argue that unless reforms work for ECRs, they will fail. M&T suggest “replication & extension” as one solution https://psyarxiv.com/dzsh4  /8

But it’s not all about incentives. @minzlicht calls for cognitive neuroscientists to rise above their egos and fallibilities, embrace error correction & champion reproducibility over reputation. And he is someone who practices what he preaches https://psyarxiv.com/8zvc3  /9

In particular, you can read @minzlicht’s recent Registered Report at Cortex where he tests the reproducibility of one of his own previous findings & concludes that the original result may be a false positive https://osf.io/473kd/  Almost nobody ever does this in cogneuro. /10 

And finally, @NeuroMinded, a former @NatureNeuro editor, takes on the newsroom culture of sci publishing. Huber et al.’s fixes will help but only superficially. To really fix these problems, he says, scientists need to take back control from publishers https://psyarxiv.com/d59me  /11

Where does all this leave us? Cortex has been at the forefront of #openscience initiatives such as @RegReports, Exploratory Reports, TOP guidelines & #openscience badges. But these are NOT enough and this year we’ll be launching two new initiatives. /12

The first is an Accountable Replications policy – @hardsci’s now famous "pottery barn rule" of publishing, which we recently introduced at @RoyalSociety Open Science. In a nutshell: if Cortex published the original study we’ll publish the replications of that study. /13

The second is an entirely new initiative, again the creation of @hardsci: Verification Reports. Short articles with the sole purpose of testing the reproducibility & robustness of original studies using the exact SAME data. /14

These steps aren’t a total answer but they move us in the right direction. The recent launch of @ukrepro – together with the wide support the network is receiving from funders, publishers & regulators – means that reproducibility is going to be a Big Deal for many years. /15

That’s why cognitive neuroscientists need to be at the forefront of those discussions. And it’s why cog neurosci journals need to work harder to support reproducibility. That means adopting @RegReports, Exploratory Reports, TOP guidelines, replication initiatives & more. /16

I will end this very long thread there! Hope you enjoy the articles (which are all available as preprints in the tweets above) and thanks to all the wonderful contributors for weighing in. Onward. /end