Last night I asked you brilliant folks to give me your strongest counterarguments
to the following proposition: that all registered clinical trials should be published in
journals only
if submitted as Registered Reports (RRs).
It was a really interesting discussion, and thanks to everyone who engaged.
Here are the reasons that were put forward.
My tl;dr verdict: I’m still waiting
for a good reason!
1. RRs require presenting more methodological detail than standard clinical
trial registration and so expose authors to a potential competitive
disadvantage.
This isn’t really a scientific objection (or least, it’s a very weak
scientific objection) but I understand the strategic argument.
My response is that if all registered trials have to published as RRs then
everyone faces the same disadvantage, so there is no relative disadvantage.
2. Clinical trial registration is sufficient for controlling bias.
It’s not. Around 50% of
clinical trials never report results, ~14% are registered after data
collection is complete, and somewhere between 30-85% engage in hidden outcome
switching. With depressing statistics like this, how can standard trial
registration be seen as anything close to sufficient?
3. OK, clinical trial registration used to be
insufficient but it’s sufficient now because clinicaltrials.gov requires
authors to specify a primary outcome measure.
Still nope. The COMPARE project finds
that authors routinely engage in hidden outcome switching even when primary
outcome measures are specified. There is no logical reason why requiring
something that already fails to prevent hidden outcome switching should prevent
hidden outcome switching.
4. Ok fine, but a signed declaration at submission that outcomes haven’t been switched
would solve hidden outcome switching.
It would probably have some effect, but then it remains easy to specify an outcome sufficiently
vaguely to enable one of several variables to be cherry picked as the primary
outcome measure, and so allow researchers to tick this box even when they switched. And even if this measure did reduce
outcome switching, it would not reduce publication bias. RRs reduce both
hidden outcome switching and publication bias. So why should any kind of declaration be
preferable to all registered clinical trials being published as RRs?
5. Small companies often live or die by the results of trials. The RR model
presents a risk to their livelihoods if they have to publicly admit that an
intervention failed to work.
The model suggested here applies only to registered trials. If companies
want to do their own internal unregistered trials and choose what to publish
(where they can) based on the results, that’s up to them. The argument here is
that the price of attaining credibility within the pages of a reputable
peer-reviewed journal should be to register the trial as a RR.
6. RRs involve one paper arising per protocol. But a single protocol may
need to produce multiple papers addressing different questions. This is also
important to support the careers of early career researchers.
This sounds to me like an argument for salami slicing in the interests of
careerism. But I accept that in the reality of academia, careers matter. My
initial reaction is that if the research question and method are complementary
enough to go in the same protocol, why aren’t the results complementary enough
to go in the same paper? The easy solution to this is to separate protocols
that address different questions into different RRs. That way there are as many
papers to publish as there are separate research questions.
7. The RR model doesn’t force authors to publish their results. Therefore
there is no guarantee that it will prevent publication bias.
This is the strongest objection so far, but even so it is virtually
guaranteed to be less of a problem for RRs than under the status quo. Authors
of RRs are indeed free to withdraw their papers after the protocol is
provisionally accepted, but doing so triggers the publication of at least part
of the registered protocol plus a reason for the
withdrawal. So what is an author going to say, that they withdrew their
peer-reviewed RR because the trial outcomes were negative? I suspect the
research community (including the reviewers who invested time in assessing the
protocol) would take a dim view on such a strategy, and it is probably for this
reason that there has yet to be a single case of a withdrawn RR at any journal. In any case,
if this were considered to be a serious risk, it would be straightforward to
strengthen the RR model for clinical trials so that it requires authors
to publish the results. If every journal published registered clinical trials
only as RRs, and all RRs bore this mandate, virtually all registered trials
would be published.
So that’s seven reasons, none of which I think are particularly strong.
Got anything stronger?