Why science – and open science – matters now more than ever

The internet is abound with thinkpieces about the election of Donald Trump, and I won’t bore you with my own views, aside from one: I hear a lot of scientists saying that their work is the least important thing now; that science has slipped down their personal agenda; that there are bigger fish to fry. 

I can understand why many of us feel this way, particularly in the immediate aftermath of such a calamitous election, but it’s mistaken.

Science matters now more than ever. Despair is an indulgence we cannot afford.

There are two existential threats posed by Trump – one that I think is mostly hyperbole, and another that is deadly and very real. The first is nuclear holocaust; the second is climate change. The threat of nuclear war is a Hollywood fantasy that will almost certainly never happen because all sides know that it guarantees mutual, permanent annihilation. The second is globally disastrous and nearly certain if the world doesn’t commit to mass action. If Trump withdraws the US from the Paris agreement, and global temperatures continue to rise, then the consequences for life on Earth are appalling. If temperatures rise by just 6 degrees then most life on Earth will become extinct, including us. None of the shit we think is important now – nothing – will matter a damn.

This really brings home for me why open science and evidence-based policy are so vital. Science must be open to be reproducible and, in turn, to have any chance of being valued in politics. And policy based on evidence is our ticket – our one ticket – to survival. I know now that attempting to make these normative will be my life's work in academia. Every day I will focus my efforts even more on our ongoing projects to promote open science and the use of evidence in policy-making.

To all scientists: now is the time to go all in and make your work as open as possible – if not for yourself then for your children and for their children. The more scientists who adopt open practices, the more normative they will become and the better chance humans have to still be here in a century’s time.

Before Brexit and Trump we could perhaps think that the wisdom of crowds would prevail and civilisation would proceed inexorably toward a Star Trek future. We now know that this is bullshit. The wisdom of crowds is bullshit. Civilisation is fragile, a thin veneer hard fought for and easily shattered.

It doesn’t matter if your work is in a field that you consider relevant to global problems like climate change – we can never predict which areas of science might one day form the basis of a solution to a problem. Adopting open practices across as many fields as possible sends the message that science needs to be transparent, reproducible and usable by anyone.

So, please, if you are not already doing so, publicly archive your data, materials and code, pre-register your hypotheses and analysis plans, and publish your work through open access routes. Focus on getting your science right rather than hacking your way to glam papers, climbing the greasy pole, or building your empire. Recognise your own bias: don't confuse the window to the natural world with a mirror. Be vocal in communicating your work in schools and to the public, particularly outside your echo chambers, to the people in your community who are the most skeptical or suspicious about science. Be politically active in highlighting the importance of evidence-based policy. Science funding is likely to become increasingly limited so make your science count. As scientists we are indoctrinated with the notion that competition matters but it doesn't. Cooperation is far more important, and we are all on the same team now. Team survival.

Whatever field you work in, your work counts and so does your voice. The world needs to survive at least four years of Donald Trump, and hell knows whatever worse may follow him.

I look at my daughter, who is ten months old, and I realise that she and her children will be the ones who will live with the choices we make today. She is why open science matters.

An "Accountable Replication Policy" at Royal Society Open Science

tl;dr: we've drafted a new journal policy for increasing the publication of replication studies and we need your help to get it right.

Many readers will be aware of Sanjay Srivastava's 2012 proposal for a Pottery Barn rule to support replications at scientific journals. The basic idea is that journals should commit to publishing replication attempts of previous studies published within the journal (i.e. "you publish it, you buy it"). As Sanjay wrote at the time: "Once a journal has published a study, it becomes responsible for publishing direct replications of that study. Publication is subject to editorial review of technical merit but is not dependent on outcome."

I have always felt that this was a brilliant idea - truly one of the most innovative and concrete proposals for not only increasing the volume of replication studies but also incentivizing journals to publish original work that is more replicable in the first place. It's a little disappointing that no journal has yet taken up the policy.

That is about to change. Essi Viding and I have written a fully specified Accountable Replications Policy, to be implemented at the journal Royal Society Open Science where we both serve as editors. 

We plan to capture the core features of Sanjay's proposal while also extending it in two ways.

First, the commitment to publish replications of previous work will include not only articles published within Royal Society Open Science but in more than 20 other major psychology and neuroscience journals. 

Second, the Accountable Replications mechanism will be open to both completed replication studies (unlocking the file drawer of hitherto unpublished replications) AND to proposals for yet-to-be completed replications. In the former case, the manuscripts will be initially stripped of results and reviewed "results-blind". In the latter case, submissions will be assessed via the journal's existing Registered Report route. In both cases, the decision to publish will be based on peer review before results are known, immunising against publication bias. 

Our draft policy has been approved by the journal editorial board and Royal Society. Drafting it was a tricky process and we are now seeking to further improve it based on your views. Is the policy fit for purpose? Is it clear? Have we missed any risks or opportunities? Would you use it? Please let us know what you think in the comments below, on social media, or by email. 

______________

Policy Overview   

We believe that when a journal publishes an empirical study it assumes a degree of accountability for the replicability of that study. This responsibility is reflected by the extent to which the journal commits to publishing attempted replications of previous studies that appear within its pages.

The Accountable Replication Policy commits the Psychology and Cognitive Neuroscience section of Royal Society Open Science to publishing replications of studies previously published within the journal. Authors can either submit a replication study that is already completed or a proposal to replicate a previous study. To ensure that the review process is unbiased by the results, submissions will be reviewed with existing results initially redacted (where applicable), or in the case of study proposals, before the results exist. Submissions that report close, clear and valid replications of the original methodology will be offered in principle acceptance, which virtually guarantees publication of the replication regardless of the study outcome.

The diagram below illustrates the editorial process and manuscript handling pipeline (and see here for a higher resolution image).

 

Detailed Policy

1.   Royal Society Open Science commits to publishing any methodologically sound attempt to replicate any study that has been published within the Psychology and Cognitive Neuroscience section of the journal, as well as any study within the remit of the Psychology and Cognitive Neuroscience section that has been published in any one of the following journals: Biological Psychiatry, Brain, Cerebral Cortex, Cortex, Cognitive Psychology, Cognition, Current Biology, European Journal of Neuroscience, Journal of Cognitive Neuroscience, any journal within the Journal of Experimental Psychology group, Journal of Experimental Social Psychology, Journal of Neuroscience, Journal of Personality and Social Psychology, Nature, Nature Neuroscience, NeuroImage, Neuron, PLOS Biology, Proceedings of the National Academy of Sciences of the USA, Psychological Science, Science, Social Psychological and Personality Science, and any journal published by the Royal Society. We will also consider, on a case-by-case basis, pre-submission enquiries for replications of studies published in journals additional to those listed here.

2.   Under this policy, authors have the choice between two routes for consideration of submissions:  a “results-blind” track in which the introduction and method of a completed study are peer reviewed while the results and discussion are temporarily withheld, or the Registered Reports track where the study protocol is reviewed and pre-accepted before the commencement of data collection.

3.   Results-blind track

3.1.  This track is intended for authors who have already completed a direct (close) replication attempt of a study published within Royal Society Open Science, within one of the source journals listed in section 1, or in any other journal approved on a case-by-case basis by the editors. Authors should initially submit a partial manuscript containing the abstract, introduction and method sections, with the results and discussion redacted together with any other information (e.g. in the abstract) that states or implies the research outcomes. The method section must include a comprehensive and detailed account of all experimental procedures (including data inclusion and exclusion criteria) and analyses undertaken. The later Stage 2 submission cannot include the outcomes of any analyses that are not stated in the Stage 1 submission at the point of in principle acceptance. The Stage 1 manuscript should be submitted under the standard research article category.

3.2.  At Stage 1 the submission is considered by reviewers who assess the content according to the following Primary and Secondary criteria:

Stage 1 Primary Criterion #1: Whether the authors provide a sufficiently clear and detailed description of the methods for another researcher to replicate exactly the proposed experimental procedures and analysis pipeline, and to prevent undisclosed flexibility in the experimental procedures or analysis pipeline. 

Stage 1 Primary Criterion #2: Whether the manuscript describes a sufficiently valid (i.e. close) and robust (e.g. statistically powerful) replication of the original study methods and rationale to provide an indication of replicability. 

Stage 1 Secondary Criterion #1: The logic, rationale, and plausibility of the proposed hypotheses. 

Stage 1 Secondary Criterion #2: The soundness of the methodology and analysis pipeline. 

Stage 1 Secondary Criterion #3: Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the results obtained are able to test the stated hypotheses.

Of the above Stage 1 criteria, editorial decisions will be based solely on the Primary Criteria. In some cases, the published methods of the original study may not provide enough detail to permit a close replication (Primary Criterion #1), and the original authors of the study may also be unwilling or unable to provide the necessary missing information. In such cases, appropriate estimates of the original procedures will be accepted provided the authors can establish that the chosen method is a reasonable estimation of the original approach, and provided they can supply documentary evidence (such as emails) proving any attempts to obtain such information from the original authors.

With the exception of one specific scenario discussed below, reviewers’ comments regarding the Secondary Criteria will not influence the decision to publish. Instead, where the editors believe that any critical issues raised within these categories are valid, additional actions may be required. The specific actions will depend on the severity of the concerns, assessed according to three ‘threat levels’: minor, major, and severe. To prevent publication bias, these assessments will be made before results are known.

For concerns judged by the editors to be minor, authors will be required to discuss the necessary limitations in the introduction (and/or later in the discussion at Stage 2).

For major concerns that substantially limit (but do not eliminate) the interpretability of any potential findings, an editorial cautionary note may also be published alongside the final report. For instance, where reviewers identify a flaw with the methodology of the original study under replication (Stage 1 Secondary Criterion #1 or #2), then in addition to a mandatory discussion of the limitation by the authors, an editorial comment may be published with the final paper highlighting the limitation of the original work. In such cases, authors would also be encouraged (but not required) to pre-register an additional replication study through the Registered Reports track, in which the procedure is matched as closely as possible to the original study but with the flaw corrected. The same author requirements and options would apply in cases where the replication lacks an appropriate positive control or other independent quality check (Stage 1 Secondary Criterion #3).

For severe concerns that prevent meaningful interpretation of the results, the course of action will be determined by the source publication of the original study. For studies published in journals other than Royal Society Open Science, Stage 1 submissions with fatal flaws will be accepted only if the authors agree to pre-register (via the Registered Reports track) an additional corrective study within the same paper that attempts to correct the identified problems. In cases where authors are unwilling or unable to do so, such Stage 1 submissions are likely to be rejected. However, to ensure that Royal Society Open Science remains accountable for the standard of work that it publishes, replication attempts even of severely flawed studies that were originally published within the journal will be permitted without the requirement for authors to pre-register an additional corrective study. Instead, as with submissions that are judged to contain major (but not severe) concerns, authors will be required to include a discussion of the fatal flaw, and the final article will be accompanied by a cautionary editorial note.

These contingencies are summarised in the table below (and see also the workflow diagam above).

		Threat level of methodological concerns under Stage 1 Secondary Criteria #1, #2 or #3
		Minor	Major	Severe
Original study published in Royal Society Open Science	Mandatory discussion of the concern in final paper by authors	✓	✓	✓
	Publication of cautionary note by editors	✘	✓	✓
	Authors encouraged (but not required) to pre-register an additional corrective replication attempt	✘	✓	✓
	Authors required to conduct a corrective replication attempt to achieve Stage 1 acceptance	✘	✘	✘
Original study not published in Royal Society Open Science	Mandatory discussion of the concern in final paper by authors	✓	✓	✓
	Publication of cautionary note by editors	✘	✓	✓
	Authors encouraged but not required to pre-register an additional corrective replication attempt	✘	✓	✘
	Authors required to conduct a corrective replication attempt to achieve Stage 1 acceptance	✘	✘	✓

Other than the scenario described above for severe concerns, Stage 1 manuscripts will be rejected outright only if reviews indicate that the replication attempt failed to meet sufficient methodological standards, such as insufficient statistical power or critical deviations from the original methodology (Stage 1 Primary Criteria #1 and #2). No fixed level of statistical power is required, but will be assessed on an individual basis before results are known. Authors may also be required to revise their Stage 1 submission to add necessary detail to the reported methodology or to alter the analyses. The description of methods should be sufficiently elaborate to provide an internally complete account of all employed methods, without the reader needing to examine the specific methodological details of the original study.

If the manuscript is reviewed favourably according to the Stage 1 Primary Criteria then it will be awarded in principle acceptance (IPA). Authors are then invited to submit a Stage 2 manuscript that includes the content of the approved Stage 1 manuscript plus the results and discussion.

3.3.  At Stage 2 the manuscript is returned to the original Stage 1 reviewers who assess the content according to the following Primary and Secondary criteria:

Stage 2 Primary Criterion #1: Whether the introduction and methods are the same as the approved Stage1 submission (required). 

Stage 2 Primary Criterion #2: Whether the authors’ conclusions are justified given the data. 

Stage 2 Secondary Criterion #1: Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects or success of positive controls or other quality checks). Failure to pass these conditions will not lead to manuscript rejection but could require authors to explicitly acknowledge such limitations in their discussion, and in severe cases could also lead to the publication of an accompanying editorial cautionary note.

Of the above Stage 2 criteria, editorial decisions will be based solely on the Primary Criteria. Comments  regarding Secondary Criterion #1 will not influence the decision to publish but could require additional caveats by the authors (in the discussion) or may be noted by the editors in an accompanying cautionary note.

Stage 2 manuscripts will be rejected only where the submission includes unapproved changes to the approved Stage 1 manuscript or where the authors’ conclusions are judged to extend beyond those permitted by the results, AND where any such differences of opinion between the authors and reviewers/editors cannot be reconciled.

4.     Registered Reports track

4.1.  This track is similar to the results-blind process but intended for authors who have yet to attempt a replication of a study published within Royal Society Open Science, within one of the source journals listed in section 1, or any other journal approved by the editors. Authors should initially submit a partial manuscript through the Registered Reports article type containing the abstract, introduction and method sections, written in the future tense. In all other respects, the options and requirements are the same as for the results-blind track.

4.2.  The Stage 1 and Stage 2 submissions will assessed according to the same Primary and Secondary Criteria as for the results-blind track (see 3.2 and 3.3 above)

4.3.  The final Stage 2 paper will be published as a Registered Report.

My personal tribute to Alex Danchev

On Saturday 29th October I spoke at the University of St Andrews memorial event for my stepfather-in-law Alex Danchev, held at St Salvator's Chapel. I've received a number of requests from family, friends and colleagues of Alex for the transcript of the speech, so I have posted it here. For those who attended, I'm sorry for losing my composure toward the end. This was far and away the most difficult bit of public speaking I have done in my life. From around the word "owlishness" I'm afraid my emotions got the better of me, but it was heartening to see when I looked up that I wasn't alone. Alex was truly irreplaceable and will be greatly missed by all who knew him.

--

It is a great honour to speak about Alex, and on behalf of the family I want thank all of his many friends and colleagues for joining us here today.

It’s nearly three months since we lost Alex, and it remains a heavy loss. But though it borders on cliché, I do feel that what we have lost is outweighed, many times over, by what we all gained from his life.

Personally speaking, my interactions with Alex often crossed between his world of the arts and mine of science, a bridge that Alex managed with a casual grace while I often laboured intensely coming the other way.

I recall last year when I was writing a part of a book that deals with the practice of science, and how quickly Alex found examples in his world that helped me understand mine. That is a rare gift even among scientists, let alone for someone who has never studied science for a day. And it was one that I felt deeply privileged to benefit from.

And, of course, there his own incredible, irreplaceable books that we all know. His many speeches, the accolades and awards.

But – if I may – I would like to put aside for a moment the sweeping achievements and talk about Alex himself. For as much as Alex’s work was at the core of his character, it was only a part of who he really was, and what he meant to his family.

In 2006, Jemma and I spent our first year back in the UK living with Dee and Alex in Oxford. I got to know Alex well. 

Each of us is a complex tapestry. The millions of small threads are, I believe, as vital as the bigger picture. Alex’s life was a rich and colourful tapestry. 

Sure there were high-minded discussions about foreign policy, literature and art. But I also remember, even more vividly, his post-jog reports on the activities of the local canal cats in North Oxford. How many cats, which ones, their general disposition, colour, degree of fluffiness. All as he whooshed past them at high speed. 

His coining of the term “Snicker cat” for the pesky animal that would slice its way into the rubbish bags on bin night. 

His unfathomable love of Turkish delight. 

His excellent recipe for tuna and tomato ragu, which I have still never managed to get right. The secret is in the onions, so he told me. 

In later years, his gentle nature around our children, even as they sometimes terrified him. 

His extraordinary running. One year he finished in the top 100 of the London marathon. And what a patient running companion! The last time I ran with him, I was 30, he was 52, and he zoomed along like a wolf while I felt like I was climbing Everest. 

One time while running together, he ran the entire route then circled back to run alongside me, without a word of ridicule or judgment. I would have thanked him if I could breathe. 

After his family and his work, his next great love was fine wines, and in his youth he worked for a time as a sommelier. This made buying him wine an intimidating prospect, to say the least.

I tried hard to avoid embarrassing myself, but at times failed rather spectacularly. On one occasion, I landed in front him a bottle of what I triumphantly declared would be the next big thing in New Zealand Sauvignon Blancs. After one taste, he simply went “Hmmm, yes”, left the room and promptly reappeared with one from his own cellar. Suffice to say, the wine I bought was never seen again. 

Of course I extracted some measure of balance by teasing him as much as possible about his 1970s Wine Encyclopaedia, a book he bought in his youth that is possibly the most pompous thing ever committed to print.

Alex, as always, absorbed my ridiculous Australian nature with saintly patience and urbane charm.

Some more memories. 

In the years that followed, after Jemma and I had bought our first home, his regular posting of newspaper cutouts containing cats to Jemma – his own, very personal way of reminding her that he loved her as a daughter. 

Hedge trimming with him last summer in Crail using a lethal piece of machinery that could easily have killed us and everyone else within a 10 metre radius. At one point I very nearly electrocuted us but he had a steady hand. 

A remarkably efficient loader and unloader of the dishwasher, yet forever perplexed by the workings of the washing machine, which he would refer to as “the clothes washer” – in his words, a “disturbing contraption”, as though he was a time traveller from the 1850s. 

A man, who if left to do the shopping, would return with two dozen tins of baked beans because they are a handy snack after all. 

One thing few people know is that in later life he planned to turn from writing non-fiction to historical fiction. What incredible novels he would have written! 

Alex had a million special and unique ways about him. His world intersected with ours and yet it was also set apart by his quest for scholarly understanding. 

He was, I think, Art History’s answer to a deep sea explorer – continually combing the abyss for new sources of understanding. And it wasn’t merely a search undertaken for his own satisfaction: what made his work worthwhile for him was surfacing with so much to report and describe.

He made that act of communication seem easy, yet I believe most of us never come close to seeing the world as he did, and fewer still are able to articulate their visions so vividly. 

When immersed in his writing, he would turn down the volume on the world outside. One time in Crail, while Dee was away, he allowed the lawn to grow so high it went to seed. Suffice to say, Dee was less than amused on her return. I still smile at this memory because his office window overlooks the garden. He literally (in every sense of the word) watched it turn into a jungle.

And yet even at his busiest he never turned the world off completely. One of Alex’s greatest virtues was his generosity. He always made the time to help others.

I still have the first draft of my own book manuscript, full of his detailed notes, and written – of all times – during his intense construction of Magritte. He insisted on editing a printed copy, old school. 

Having made the revisions I was on the verge put the draft in the recycling, but now it has become priceless to me. Especially his patient reminders about split infinitives and his uncanny, almost clairvoyant, ability to suggest the word I MEANT to write rather than the one I actually wrote.

In an email to me in earlier this year he gave me some general advice about writing which, reflecting on it now, is actually pretty good life advice:

“I'd be inclined to look out for ways in which you can add like bits of colour, or spice - maybe in the vocabulary, or formulation, and especially at the conclusion of each chapter. When you let yourself go a little, in the expression, it works well, I think. Too much of that would be inadvisable, no doubt, but perhaps there's room for a judicious dash more.”

One of my favourite writers, David Dobbs, recommends that after a hard day’s writing it is best to stop half way through a thought. Even if you know exactly what you’re going to say, resist the temptation to finish it. That way it’s easier to pick up where you left off tomorrow and so avoid writer’s block. 

Alex left a life unfinished. He won’t wake up tomorrow to carry on, so it’s now up to us to pick up where he left off. 

Few among us would be able to carry on his work but I believe all of us can take forward his kindness and generosity and sense of purpose, and perhaps some of what made him so unique will continue to live through us. 

As I say goodbye to Alex, I’m reminded how fortunate he was to do what he loved, in the place he loved, with the one person he loved above all others – and how fortunate we were to know him. We are all fortunate in another way too, that his words will live on in a way that immortalises him.

But his written words, as extraordinary as they are, are not how I will remember him. For me, it is a simpler picture. 

It is summer in Crail. Alex is sitting at his desk overlooking the sea, working on Magritte with his customary owlishness.

Then my 1-year old son, who is just starting to walk, finishes an epic crawl up the stairs and toddles triumphantly into his office. Without a hint of irritation, Alex emerges from the depths to greet him. My son climbs on his lap and they sit together, for a moment in silence, watching the waves roll in. 

I’m a scientist so, for me, there is no doubt that death is final and we bring nothing of ourselves beyond that door. And yet part of me clings to a conceit, that if I freeze this memory in time – in amber, as it were – then Alex never dies.

Whenever I need Alex, I will know where to find him. He is by the window, sitting with his grandson on a warm summer’s evening. And he is happy.