The internet is abound with thinkpieces about the election of Donald Trump, and I won’t bore you with my own views, aside from one: I hear a lot of scientists saying that their work is the least important thing now; that science has slipped down their personal agenda; that there are bigger fish to fry.
I can understand why many of us feel this way, particularly in the immediate aftermath of such a calamitous election, but it’s mistaken.
Science matters now more than ever. Despair is an indulgence we cannot afford.
There are two existential threats posed by Trump – one that I think is mostly hyperbole, and another that is deadly and very real. The first is nuclear holocaust; the second is climate change. The threat of nuclear war is a Hollywood fantasy that will almost certainly never happen because all sides know that it guarantees mutual, permanent annihilation. The second is globally disastrous and nearly certain if the world doesn’t commit to mass action. If Trump withdraws the US from the Paris agreement, and global temperatures continue to rise, then the consequences for life on Earth are appalling. If temperatures rise by just 6 degrees then most life on Earth will become extinct, including us. None of the shit we think is important now – nothing – will matter a damn.
This really brings home for me why open science and evidence-based policy are so vital. Science must be open to be reproducible and, in turn, to have any chance of being valued in politics. And policy based on evidence is our ticket – our one ticket – to survival. I know now that attempting to make these normative will be my life's work in academia. Every day I will focus my efforts even more on our ongoing projects to promote open science and the use of evidence in policy-making.
To all scientists: now is the time to go all in and make your work as open as possible – if not for yourself then for your children and for their children. The more scientists who adopt open practices, the more normative they will become and the better chance humans have to still be here in a century’s time.
Before Brexit and Trump we could perhaps think that the wisdom of crowds would prevail and civilisation would proceed inexorably toward a Star Trek future. We now know that this is bullshit. The wisdom of crowds is bullshit. Civilisation is fragile, a thin veneer hard fought for and easily shattered.
It doesn’t matter if your work is in a field that you consider relevant to global problems like climate change – we can never predict which areas of science might one day form the basis of a solution to a problem. Adopting open practices across as many fields as possible sends the message that science needs to be transparent, reproducible and usable by anyone.
So, please, if you are not already doing so, publicly archive your data, materials and code, pre-register your hypotheses and analysis plans, and publish your work through open access routes. Focus on getting your science right rather than hacking your way to glam papers, climbing the greasy pole, or building your empire. Recognise your own bias: don't confuse the window to the natural world with a mirror. Be vocal in communicating your work in schools and to the public, particularly outside your echo chambers, to the people in your community who are the most skeptical or suspicious about science. Be politically active in highlighting the importance of evidence-based policy. Science funding is likely to become increasingly limited so make your science count. As scientists we are indoctrinated with the notion that competition matters but it doesn't. Cooperation is far more important, and we are all on the same team now. Team survival.
Whatever field you work in, your work counts and so does your voice. The world needs to survive at least four years of Donald Trump, and hell knows whatever worse may follow him.
I look at my daughter, who is ten months old, and I realise that she and her children will be the ones who will live with the choices we make today. She is why open science matters.
Sunday 13 November 2016
Wednesday 9 November 2016
An "Accountable Replication Policy" at Royal Society Open Science
Many readers will be aware of Sanjay Srivastava's 2012 proposal for a Pottery Barn rule to support replications at scientific journals. The basic idea is that journals should commit to publishing replication attempts of previous studies published within the journal (i.e. "you publish it, you buy it"). As Sanjay wrote at the time: "Once a journal has published a study, it becomes responsible for publishing direct replications of that study. Publication is subject to editorial review of technical merit but is not dependent on outcome."
I have always felt that this was a brilliant idea - truly one of the most innovative and concrete proposals for not only increasing the volume of replication studies but also incentivizing journals to publish original work that is more replicable in the first place. It's a little disappointing that no journal has yet taken up the policy.
That is about to change. Essi Viding and I have written a fully specified Accountable Replications Policy, to be implemented at the journal Royal Society Open Science where we both serve as editors.
We plan to capture the core features of Sanjay's proposal while also extending it in two ways.
First, the commitment to publish replications of previous work will include not only articles published within Royal Society Open Science but in more than 20 other major psychology and neuroscience journals.
Second, the Accountable Replications mechanism will be open to both completed replication studies (unlocking the file drawer of hitherto unpublished replications) AND to proposals for yet-to-be completed replications. In the former case, the manuscripts will be initially stripped of results and reviewed "results-blind". In the latter case, submissions will be assessed via the journal's existing Registered Report route. In both cases, the decision to publish will be based on peer review before results are known, immunising against publication bias.
Our draft policy has been approved by the journal editorial board and Royal Society. Drafting it was a tricky process and we are now seeking to further improve it based on your views. Is the policy fit for purpose? Is it clear? Have we missed any risks or opportunities? Would you use it? Please let us know what you think in the comments below, on social media, or by email.
______________
Policy Overview
We believe that when a journal publishes an empirical study it assumes a degree of accountability for the replicability of that study. This responsibility is reflected by the extent to which the journal commits to publishing attempted replications of previous studies that appear within its pages.
The Accountable Replication Policy commits the Psychology and Cognitive Neuroscience section of Royal Society Open Science to publishing replications of studies previously published within the journal. Authors can either submit a replication study that is already completed or a proposal to replicate a previous study. To ensure that the review process is unbiased by the results, submissions will be reviewed with existing results initially redacted (where applicable), or in the case of study proposals, before the results exist. Submissions that report close, clear and valid replications of the original methodology will be offered in principle acceptance, which virtually guarantees publication of the replication regardless of the study outcome.
The diagram below illustrates the editorial process and manuscript handling pipeline (and see here for a higher resolution image).
Detailed Policy
1. Royal
Society Open Science commits to publishing any methodologically sound
attempt to replicate any study that has been published within the Psychology
and Cognitive Neuroscience section of the journal, as well as any study within
the remit of the Psychology and Cognitive Neuroscience section that has been
published in any one of the following journals: Biological Psychiatry, Brain,
Cerebral Cortex, Cortex, Cognitive
Psychology, Cognition, Current Biology, European Journal of Neuroscience, Journal of Cognitive Neuroscience, any journal within the Journal of Experimental Psychology
group, Journal of Experimental Social
Psychology, Journal of Neuroscience,
Journal of Personality and Social
Psychology, Nature, Nature Neuroscience, NeuroImage, Neuron, PLOS Biology, Proceedings of the National Academy
of Sciences of the USA, Psychological
Science, Science, Social
Psychological and Personality Science, and any journal published by the
Royal Society. We will also consider, on a case-by-case basis, pre-submission
enquiries for replications of studies published in journals additional to those
listed here.
2. Under this policy, authors have the choice
between two routes for consideration of submissions: a “results-blind” track in which the
introduction and method of a completed study are peer reviewed while the
results and discussion are temporarily withheld, or the Registered Reports track where the study protocol
is reviewed and pre-accepted before the commencement of data collection.
3. Results-blind track
3.1. This
track is intended for authors who have already completed a direct (close)
replication attempt of a study published within Royal Society Open Science, within one of the source journals
listed in section 1, or in any other journal approved on a case-by-case basis by
the editors. Authors should initially submit a partial manuscript containing
the abstract, introduction and method sections, with the results and discussion
redacted together with any other information (e.g. in the abstract) that states
or implies the research outcomes. The method section must include a
comprehensive and detailed account of all experimental procedures (including
data inclusion and exclusion criteria) and analyses undertaken. The later Stage
2 submission cannot include the outcomes of any analyses that are not stated in
the Stage 1 submission at the point of in principle acceptance. The Stage 1
manuscript should be submitted under the standard research article category.
3.2. At
Stage 1 the submission is considered by reviewers who assess the content
according to the following Primary
and Secondary criteria:
Stage 1 Primary Criterion #1: Whether
the authors provide a sufficiently clear and detailed description of the
methods for another researcher to replicate exactly the proposed experimental
procedures and analysis pipeline, and to prevent undisclosed flexibility in the
experimental procedures or analysis pipeline.
Stage 1 Primary Criterion #2: Whether the manuscript describes a sufficiently valid (i.e. close) and robust (e.g. statistically powerful) replication of the original study methods and rationale to provide an indication of replicability.
Stage 1 Secondary Criterion #1: The logic, rationale, and plausibility of the proposed hypotheses.
Stage 1 Secondary Criterion #2: The soundness of the methodology and analysis pipeline.
Stage 1 Secondary Criterion #3: Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the results obtained are able to test the stated hypotheses.
Stage 1 Primary Criterion #2: Whether the manuscript describes a sufficiently valid (i.e. close) and robust (e.g. statistically powerful) replication of the original study methods and rationale to provide an indication of replicability.
Stage 1 Secondary Criterion #1: The logic, rationale, and plausibility of the proposed hypotheses.
Stage 1 Secondary Criterion #2: The soundness of the methodology and analysis pipeline.
Stage 1 Secondary Criterion #3: Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the results obtained are able to test the stated hypotheses.
Of the above Stage 1 criteria, editorial decisions will
be based solely on the Primary Criteria. In some cases, the published methods
of the original study may not provide enough detail to permit a close
replication (Primary Criterion #1), and the original authors of the study may
also be unwilling or unable to provide the necessary missing information. In
such cases, appropriate estimates of the original procedures will be accepted
provided the authors can establish that the chosen method is a reasonable
estimation of the original approach, and provided they can supply documentary
evidence (such as emails) proving any attempts to obtain such information from
the original authors.
With the exception of one specific scenario discussed below, reviewers’ comments regarding the Secondary Criteria will not influence the decision to publish. Instead, where the editors believe that any critical issues raised within these categories are valid, additional actions may be required. The specific actions will depend on the severity of the concerns, assessed according to three ‘threat levels’: minor, major, and severe. To prevent publication bias, these assessments will be made before results are known.
For concerns judged by the editors to be minor, authors will be required to discuss the necessary limitations in the introduction (and/or later in the discussion at Stage 2).
For major concerns that substantially limit (but do not eliminate) the interpretability of any potential findings, an editorial cautionary note may also be published alongside the final report. For instance, where reviewers identify a flaw with the methodology of the original study under replication (Stage 1 Secondary Criterion #1 or #2), then in addition to a mandatory discussion of the limitation by the authors, an editorial comment may be published with the final paper highlighting the limitation of the original work. In such cases, authors would also be encouraged (but not required) to pre-register an additional replication study through the Registered Reports track, in which the procedure is matched as closely as possible to the original study but with the flaw corrected. The same author requirements and options would apply in cases where the replication lacks an appropriate positive control or other independent quality check (Stage 1 Secondary Criterion #3).
For severe concerns that prevent meaningful interpretation of the results, the course of action will be determined by the source publication of the original study. For studies published in journals other than Royal Society Open Science, Stage 1 submissions with fatal flaws will be accepted only if the authors agree to pre-register (via the Registered Reports track) an additional corrective study within the same paper that attempts to correct the identified problems. In cases where authors are unwilling or unable to do so, such Stage 1 submissions are likely to be rejected. However, to ensure that Royal Society Open Science remains accountable for the standard of work that it publishes, replication attempts even of severely flawed studies that were originally published within the journal will be permitted without the requirement for authors to pre-register an additional corrective study. Instead, as with submissions that are judged to contain major (but not severe) concerns, authors will be required to include a discussion of the fatal flaw, and the final article will be accompanied by a cautionary editorial note.
These contingencies are summarised in the table below (and see also the workflow diagam above).
With the exception of one specific scenario discussed below, reviewers’ comments regarding the Secondary Criteria will not influence the decision to publish. Instead, where the editors believe that any critical issues raised within these categories are valid, additional actions may be required. The specific actions will depend on the severity of the concerns, assessed according to three ‘threat levels’: minor, major, and severe. To prevent publication bias, these assessments will be made before results are known.
For concerns judged by the editors to be minor, authors will be required to discuss the necessary limitations in the introduction (and/or later in the discussion at Stage 2).
For major concerns that substantially limit (but do not eliminate) the interpretability of any potential findings, an editorial cautionary note may also be published alongside the final report. For instance, where reviewers identify a flaw with the methodology of the original study under replication (Stage 1 Secondary Criterion #1 or #2), then in addition to a mandatory discussion of the limitation by the authors, an editorial comment may be published with the final paper highlighting the limitation of the original work. In such cases, authors would also be encouraged (but not required) to pre-register an additional replication study through the Registered Reports track, in which the procedure is matched as closely as possible to the original study but with the flaw corrected. The same author requirements and options would apply in cases where the replication lacks an appropriate positive control or other independent quality check (Stage 1 Secondary Criterion #3).
For severe concerns that prevent meaningful interpretation of the results, the course of action will be determined by the source publication of the original study. For studies published in journals other than Royal Society Open Science, Stage 1 submissions with fatal flaws will be accepted only if the authors agree to pre-register (via the Registered Reports track) an additional corrective study within the same paper that attempts to correct the identified problems. In cases where authors are unwilling or unable to do so, such Stage 1 submissions are likely to be rejected. However, to ensure that Royal Society Open Science remains accountable for the standard of work that it publishes, replication attempts even of severely flawed studies that were originally published within the journal will be permitted without the requirement for authors to pre-register an additional corrective study. Instead, as with submissions that are judged to contain major (but not severe) concerns, authors will be required to include a discussion of the fatal flaw, and the final article will be accompanied by a cautionary editorial note.
These contingencies are summarised in the table below (and see also the workflow diagam above).
Threat level of methodological concerns
under Stage 1 Secondary Criteria #1, #2 or #3
|
||||
Minor
|
Major
|
Severe
|
||
Original
study published in Royal Society Open
Science
|
Mandatory discussion of the concern in
final paper by authors
|
✓
|
✓
|
✓
|
Publication of cautionary note by editors
|
✘
|
✓
|
✓
|
|
Authors encouraged (but not required) to
pre-register an additional corrective replication attempt
|
✘
|
✓
|
✓
|
|
Authors required to conduct a corrective
replication attempt to achieve Stage 1 acceptance
|
✘
|
✘
|
✘
|
|
Original study not published in Royal
Society Open Science
|
Mandatory discussion of the concern in
final paper by authors
|
✓
|
✓
|
✓
|
Publication of cautionary note by editors
|
✘
|
✓
|
✓
|
|
Authors encouraged but not required to
pre-register an additional corrective replication attempt
|
✘
|
✓
|
✘
|
|
Authors required to conduct a corrective
replication attempt to achieve Stage 1 acceptance
|
✘
|
✘
|
✓
|
Other than the scenario described above for severe
concerns, Stage 1 manuscripts will be rejected outright only if reviews
indicate that the replication attempt failed to meet sufficient methodological
standards, such as insufficient statistical power or critical deviations from
the original methodology (Stage 1 Primary Criteria #1 and #2). No fixed level
of statistical power is required, but will be assessed on an individual basis
before results are known. Authors may also be required to revise their Stage 1
submission to add necessary detail to the reported methodology or to alter the
analyses. The description of methods should be sufficiently elaborate to
provide an internally complete account of all employed methods, without the
reader needing to examine the specific methodological details of the original
study.
If the manuscript is reviewed favourably according to the Stage 1 Primary Criteria then it will be awarded in principle acceptance (IPA). Authors are then invited to submit a Stage 2 manuscript that includes the content of the approved Stage 1 manuscript plus the results and discussion.
3.3. At
Stage 2 the manuscript is returned to the original Stage 1 reviewers who assess
the content according to the following Primary
and Secondary criteria:
Stage 2 Primary Criterion #1: Whether
the introduction and methods are the same as the approved Stage1 submission
(required).
Stage 2 Primary Criterion #2: Whether the authors’ conclusions are justified given the data.
Stage 2 Secondary Criterion #1: Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects or success of positive controls or other quality checks). Failure to pass these conditions will not lead to manuscript rejection but could require authors to explicitly acknowledge such limitations in their discussion, and in severe cases could also lead to the publication of an accompanying editorial cautionary note.
Stage 2 Primary Criterion #2: Whether the authors’ conclusions are justified given the data.
Stage 2 Secondary Criterion #1: Whether the data are able to test the authors’ proposed hypotheses by passing the approved outcome-neutral criteria (such as absence of floor and ceiling effects or success of positive controls or other quality checks). Failure to pass these conditions will not lead to manuscript rejection but could require authors to explicitly acknowledge such limitations in their discussion, and in severe cases could also lead to the publication of an accompanying editorial cautionary note.
Of the above
Stage 2 criteria, editorial decisions will be based solely on the Primary
Criteria. Comments regarding Secondary Criterion #1 will not influence
the decision to publish but could require additional caveats by the authors (in
the discussion) or may be noted by the editors in an accompanying cautionary
note.
Stage 2 manuscripts will be rejected only where the submission includes unapproved changes to the approved Stage 1 manuscript or where the authors’ conclusions are judged to extend beyond those permitted by the results, AND where any such differences of opinion between the authors and reviewers/editors cannot be reconciled.
Stage 2 manuscripts will be rejected only where the submission includes unapproved changes to the approved Stage 1 manuscript or where the authors’ conclusions are judged to extend beyond those permitted by the results, AND where any such differences of opinion between the authors and reviewers/editors cannot be reconciled.
4.
Registered Reports track
4.1. This track is similar to the results-blind process but intended for authors who have yet to attempt a replication of a study published within Royal Society Open Science, within one of the source journals listed in section 1, or any other journal approved by the editors. Authors should initially submit a partial manuscript through the Registered Reports article type containing the abstract, introduction and method sections, written in the future tense. In all other respects, the options and requirements are the same as for the results-blind track.
4.2. The
Stage 1 and Stage 2 submissions will assessed according to the same Primary and
Secondary Criteria as for the results-blind track (see 3.2 and 3.3 above)
4.3. The
final Stage 2 paper will be published as a Registered Report.
Tuesday 1 November 2016
My personal tribute to Alex Danchev
On Saturday 29th October I spoke at the University of St Andrews memorial event for my stepfather-in-law Alex Danchev, held at St Salvator's Chapel. I've received a number of requests from family, friends and colleagues of Alex for the transcript of the speech, so I have posted it here. For those who attended, I'm sorry for losing my composure toward the end. This was far and away the most difficult bit of public speaking I have done in my life. From around the word "owlishness" I'm afraid my emotions got the better of me, but it was heartening to see when I looked up that I wasn't alone. Alex was truly irreplaceable and will be greatly missed by all who knew him.
--
It is a great honour to
speak about Alex, and on behalf of the family I want thank all of his many friends
and colleagues for joining us here today.
It’s nearly three months
since we lost Alex, and it remains a heavy loss. But though it borders on
cliché, I do feel that what we have lost is outweighed, many times over, by
what we all gained from his life.
Personally speaking, my interactions
with Alex often crossed between his world of the arts and mine of science, a
bridge that Alex managed with a casual grace while I often laboured intensely
coming the other way.
I recall last year when I
was writing a part of a book that deals with the practice of science, and how
quickly Alex found examples in his world that helped me understand mine. That
is a rare gift even among scientists, let alone for someone who has never
studied science for a day. And it was one that I felt deeply privileged to benefit
from.
And, of course, there his
own incredible, irreplaceable books that we all know. His many speeches, the accolades
and awards.
But – if I may – I would
like to put aside for a moment the sweeping achievements and talk about Alex himself.
For as much as Alex’s work was at the core of his character, it was only
a part of who he really was, and what he meant to his family.
In 2006, Jemma and I spent
our first year back in the UK living with Dee and Alex in Oxford. I got to know
Alex well.
Each of us is a complex tapestry.
The millions of small threads are, I believe, as vital as the bigger picture. Alex’s
life was a rich and colourful tapestry.
Sure there were high-minded discussions
about foreign policy, literature and art. But I also remember, even more
vividly, his post-jog reports on the activities of the local canal cats in
North Oxford. How many cats, which ones, their general disposition, colour,
degree of fluffiness. All as he whooshed past them at high speed.
His coining of the term
“Snicker cat” for the pesky animal that would slice its way into the rubbish
bags on bin night.
His unfathomable love of
Turkish delight.
His excellent recipe for
tuna and tomato ragu, which I have still never managed to get right. The secret
is in the onions, so he told me.
In later years, his gentle
nature around our children, even as they sometimes terrified him.
His extraordinary running.
One year he finished in the top 100 of the London marathon. And what a patient
running companion! The last time I ran with him, I was 30, he was 52, and he zoomed
along like a wolf while I felt like I was climbing Everest.
One time while running
together, he ran the entire route then circled back to run alongside me,
without a word of ridicule or judgment. I would have thanked him if I could
breathe.
After his family and his work,
his next great love was fine wines, and in his youth he worked for a time as a
sommelier. This made buying him wine an intimidating prospect, to say the least.
I tried hard to avoid embarrassing
myself, but at times failed rather spectacularly. On one occasion, I landed in
front him a bottle of what I triumphantly declared would be the next big thing
in New Zealand Sauvignon Blancs. After one taste, he simply went “Hmmm, yes”,
left the room and promptly reappeared with one from his own cellar. Suffice to
say, the wine I bought was never seen again.
Of course I extracted some
measure of balance by teasing him as much as possible about his 1970s Wine Encyclopaedia,
a book he bought in his youth that is possibly the most pompous thing ever committed
to print.
Alex, as always, absorbed my
ridiculous Australian nature with saintly patience and urbane charm.
Some more memories.
In the years that followed,
after Jemma and I had bought our first home, his regular posting of newspaper
cutouts containing cats to Jemma – his own, very personal way of reminding her
that he loved her as a daughter.
Hedge trimming with him last
summer in Crail using a lethal piece of machinery that could easily have killed
us and everyone else within a 10 metre radius. At one point I very nearly
electrocuted us but he had a steady hand.
A remarkably efficient
loader and unloader of the dishwasher, yet forever perplexed by the workings of
the washing machine, which he would refer to as “the clothes washer” – in his words,
a “disturbing contraption”, as though he was a time traveller from the 1850s.
A man, who if left to do the
shopping, would return with two dozen tins of baked beans because they are a
handy snack after all.
One thing few people know is
that in later life he planned to turn from writing non-fiction to historical
fiction. What incredible novels he would have written!
Alex had a million special
and unique ways about him. His world intersected with ours and yet it was also set
apart by his quest for scholarly understanding.
He was, I think, Art History’s
answer to a deep sea explorer – continually combing the abyss for new sources
of understanding. And it wasn’t merely a search undertaken for his own
satisfaction: what made his work worthwhile for him was surfacing with so much
to report and describe.
He made that act of communication
seem easy, yet I believe most of us never come close to seeing the world as he
did, and fewer still are able to articulate their visions so vividly.
When immersed in his
writing, he would turn down the volume on the world outside. One time in Crail,
while Dee was away, he allowed the lawn to grow so high it went to seed. Suffice
to say, Dee was less than amused on her
return. I still smile at this memory because his office window overlooks the
garden. He literally (in every sense of the word) watched it turn into a
jungle.
And yet even at his busiest he
never turned the world off completely. One of Alex’s greatest virtues was his
generosity. He always made the time to help others.
I still have the first draft
of my own book manuscript, full of his detailed notes, and written – of all
times – during his intense construction of Magritte. He insisted on editing a
printed copy, old school.
Having made the revisions I
was on the verge put the draft in the recycling, but now it has become priceless
to me. Especially his patient reminders about split infinitives and his
uncanny, almost clairvoyant, ability to suggest the word I MEANT to write
rather than the one I actually wrote.
In an email to me in earlier
this year he gave me some general advice about writing which, reflecting on it
now, is actually pretty good life advice:
“I'd be inclined to look out for ways in which you can add like bits of colour, or spice - maybe in the vocabulary, or formulation, and especially at the conclusion of each chapter. When you let yourself go a little, in the expression, it works well, I think. Too much of that would be inadvisable, no doubt, but perhaps there's room for a judicious dash more.”
Alex left a life unfinished. He won’t wake up tomorrow to carry on, so it’s now up to us to pick up where he left off.
Few among us would be able to carry on his work but I believe all of us can take forward his kindness and generosity and sense of purpose, and perhaps some of what made him so unique will continue to live through us.
As I say goodbye to Alex, I’m
reminded how fortunate he was to do what he loved, in the place he loved, with
the one person he loved above all others – and how fortunate we were to know
him. We are all fortunate in another way too, that his words will live on in a
way that immortalises him.
But his written words, as
extraordinary as they are, are not how I will remember him. For me, it is a
simpler picture.
It is summer in Crail. Alex
is sitting at his desk overlooking the sea, working on Magritte with his customary
owlishness.
Then my 1-year old son, who
is just starting to walk, finishes an epic crawl up the stairs and toddles triumphantly
into his office. Without a hint of irritation, Alex emerges from the depths to
greet him. My son climbs on his lap and they sit together, for a moment in
silence, watching the waves roll in.
I’m a scientist so, for me,
there is no doubt that death is final and we bring nothing of ourselves beyond
that door. And yet part of me clings to a conceit, that if I freeze this memory
in time – in amber, as it were – then Alex never dies.
Whenever I need Alex, I will
know where to find him. He is by the window, sitting with his grandson on a warm
summer’s evening. And he is happy.
Thursday 29 September 2016
“Methodological terrorism” and other myths
Most readers of my blog will be aware of Professor Susan Fiske’s
leaked letter to the APS Observer last week. In her article, Fiske delivers a blistering invective on the dangers of social media, warning that scrutiny of
psychological research by “online
vigilantes” and “self-appointed data police” is destroying careers. In actions
she describes as “methodological terrorism”, Fiske warns of “graduate students
opting out of academia, assistant professors afraid to come up for tenure,
mid-career people wondering how to protect their labs, and senior faculty
retiring early”. Her solution? Scientific critique should be restricted to
moderated groups and behind the walls of peer reviewed academic journals.
Well.
Fiske refuses to identify the “terrorists” because apparently that’s unprofessional (though see this argument from Neuroskeptic as to why it is vital if such allegations are true). This leaves her in an unenviable Trumpian territory of delivering a broadside against an entire community based on what “people have been saying”. Lots of people, no doubt. Good people.
The whole premise is so flimsy that it would probably have been ignored if not delivered by someone so high ranking.
At the time the article leaked I was at a conference in Germany that Fiske was also attending. I immediately emailed her, pointing out what I saw as some of the problems with her argument, and I invited her to join our session the next morning to publicly debate the issue. She declined.
Two days later, PhD student Anne Scheel had the courage to stand up in front of a packed auditorium and challenge Fiske directly about her “terrorist” charge (if you’re not following Anne, go do that now). In response, Fiske claimed that everyone overreacted. Apparently, she just used that term to get people’s attention, but says she would still defend the “conceptual basis for that wording”.
Oh irony.
Now you may well ask, what evidence is there that Fiske's terrorists even exist in sufficient numbers to warrant such a panic? That answer is, not much. Since 2015, UCL neuroscientist Sam Schwarzkopf has operated a blog inviting those who believe they have been bullied by methodologists or the open science movement to speak up, anonymously if they prefer. How many victims have come forward?
Not one.
So what’s really going on here? The truth is that we are in the midst of a power struggle, and it’s not between Fiske’s “destructo-critics” and their victims, but between reformers who are trying desperately to improve science and a vanguard of traditionalists who, every so often, look down from their thrones to throw a log in the road. As the body count of failed replications continues to climb, a new generation want a different kind of science and they want it now. They're demanding greater transparency in research practices. They want to be rewarded for doing high quality research regardless of the results. They want researchers to be accountable for the quality of the work they produce and for the choices they make as public professionals. It's all very sensible, constructive, and in the public interest.
Placed in this context, Fiske’s argument – as inflammatory as it is – is actually quite boring and more than a little naive. Social media is democratizing science, and nobody, not even the most powerful among us, can stop its inexorable rise. Does anyone really think that Pandora’s box can be closed, returning to the halcyon days of the 1990s (or earlier) when critique of science happened behind closed doors among selected elites?
What’s happened instead is that technology has empowered the new generation to speak freely and publicly on scientific issues, to critique poor quality science and practices, to bust fraud, and to break the bounds of peer review. Twitter in particular has shaken the traditional academic hierarchy to its core. On Twitter a PhD student with thousands of followers suddenly has a greater voice than an Ivy League professor who might have no social media presence at all.
If the traditionalists want to preserve their legacies (and they are worth preserving) then they would do well to remember who they truly serve. Psychology does not exist to support the interests of researchers seeking to command petty empires and prestigious careers, less still to protect the reputations of those who have already “won” in the academic game of life. Psychology has but one master: the public. That’s why psychology featured in over 400 “impact case studies” in a recent audit of UK science, and it’s why millions of dollars, pounds and euros are being invested in new opportunities for all scientists, including psychologists, to do more transparent and robust research.
We remain forever accountable to that public for the quality of research we produce. Those who are prepared to critique science – and scientists too for their choices – are performing a public service that warrants our appreciation not condemnation. To Fiske’s “terrorists” I therefore say thank you. And to Fiske herself, who complains about the tone of scientific debate while calling it terrorism, I say: physician heal thyself.
For other responses to Fiske, see these great posts by Andrew Gelman, Ana Todorović, Xenia Schmalz, Neuroskeptic, Andy Field, and Tal Yarkoni.
Subscribe to:
Posts (Atom)