Friday, 9 September 2022

The day I met the Queen

I’m no fan of monarchies. I’m Australian and I voted (on the losing side) of the 1999 referendum to become a republic. Still, I think that even for many republicans, the Queen surpassed politics. She was a professional and a seemingly permanent fixture of our lives who cared about her responsibilities, and she certainly put the work in.


With that in mind, a brief story about the day I met her during the grand opening of CUBRIC in summer 2016.


First, the lead up. For weeks beforehand, we had Men in Black wandering the corridors, opening random doors and asking random questions. Yes, I work here. No, that’s a cleaner’s closet. No I’m not going to murder the monarch. Chill out guys, killing famous visitors reduces our funding rates.


We also had to deal with the Queen’s private secretary who I remember as an absurdly pompous character. I was already permanently sleep deprived from having a 2 year-old and a 6 month-old, and I had nothing left to give to the pretentious trivialities and condescending micromanagement of a "Royal opening". I stopped attending the endless "rehearsals".


The Queen’s chief of police was a nice guy though. One day I was sitting in my office and he dropped in for a chat. Over the course of a general conversation about life and family, he told me that the Queen was particularly looking forward to this visit because it was about science, which she loved, and she was getting bored of art galleries.


As part of the visit we had to empty one of our testing labs and convert it into a room containing nothing but a chair and a phone and a bottle of water, just in case she needed to make an urgent call. The table had to be in a certain position in the room. The phone had to be in a certain position on the table, the water on the left. We rolled our eyes at the fussiness of it all.


Then the actual visit. My job was to explain the technique of transcranial magnetic stimulation, or TMS, which involves delivering electromagnetic pulses to the brain in living people. I was the director of this particular method in the centre, so I made myself the guinea pig and the brilliant and steady-handed Jemma Sedgmond (my then lab manager) had the job of zapping my brain.


We’d done this many times before in student lectures so in a lot of ways it was just another day at the office, and in every other way it was the strangest thing we had ever done.


We were doing the demo in the control room of our 7T MRI scanner, the flagship technology of the centre. In came the Queen and Prince Philip.


For some reason Philip immediately started faffing about with some storage lockers in the corner while various dignitaries humoured him and tried to gently steer him back on track.


The Queen, however, was having none of that. In a way that seemed quite practised, she ignored Philip’s rummaging and instead made a bee line for the key stuff. She knew what she was doing.


From memory this photo was taken while Philip had his head in a locker. There’s my blurry bald head at the back. In the foreground is our chief physicist John Evans and our (then) director of fMRI, Richard Wise. There’s a photo somewhere of Jemma and me with the TMS but I can’t find it now.



We eventually got to the TMS demo. There I was talking the Queen and Prince Philip through live electromagnetic stimulation of my brain. Jemma administered TMS to my right and left motor cortex, and I explained the contractions of my hands and arms in real time, how they arose, how the technique could be applied in other ways, and how it could help us understand brain function in health and disease.


I’m virtually certain this is the one and only time someone from the Royal family will see a scientist talk them through live electrical stimulation of their own brain.


Sidebar: sometimes watching TMS makes people feel sick. I once had a student in the front row of a lecture pull a whitey on me, and it was in the back of my mind that this could happen again. Imagine the nightmare of paperwork if the Queen fainted in front of us. But she only recoiled ever so slightly as my arms jumped around.


After we’d finished the demo, Philip mumbled something unintelligible and the Queen asked a bunch of very good questions. She had an easy and inquisitive way about her, and as I was answering her, it occurred to me that all the pomposity and ceremony of her visit seemed to be more about the people around her feeling important, and less to do with her as a person.


All she wanted to do was talk about the science. Why was the effect of TMS contralateral? (that is, you stimulate right side of the brain but see the effect on the left side of the body). How could you use this to understand the effects of stroke or brain injury? Could this help the brain adapt after damage? Yes ma'am (as in ham, not ma'am as in palm). Thank you pompous private secretary for helping me remember that particular triviality.


Then after weeks of lead up, it was over. It was an intensely weird experience, but like Janeway says to Harry Kim in Star Trek Voyager, "We're Starfleet officers. Weird is part of the job." 


RIP HMQ, a woman with possibly the weirdest job of all.

Monday, 16 March 2020

CALLING ALL SCIENTISTS: Rapid evaluation of COVID19-related Registered Reports at Royal Society Open Science

Each of us has a part to play in combatting the COVID-19 pandemic, including journals and Registered Reports (RRs).

For those unfamiliar with RRs, they are a form of empirical article offered by more than 300 journals in which study proposals are peer reviewed and pre-accepted before research is undertaken. This article type offers a powerful tool for advancing research on COVID-19 by reducing publication bias and reporting bias in the growing evidence base.

As the RR editor at Royal Society Open Science, I am announcing a special call for submissions that are relevant to any aspect of COVID-19 in any field, including but not limited to biological, medical, economic, and psychological research. All RRs at Royal Society Open Science are published open access (CC-BY) with accompanying open review.

To maximise accessibility of the call, all article processing charges for these submissions are automatically waived, and to ensure that high quality protocols can be implemented as swiftly as possibly, the journal will strive to complete initial Stage 1 review within 7 days of receiving a submission. 

This form of rapid review for RRs has never been attempted, and for it to succeed we need your help. 

To reviewers: If you are willing to review a Stage 1 RR in your field of expertise within 24-48 hours of accepting a review request, then please complete this online form. You don't need to be a virologist or epidemiologist -- research in a wide range of fields is relevant to COVID-19, including maths, statistics, psychology and the social sciences. All expert reviewers are welcome, from PhD students to professors. 

To authors: To accelerate handling of submissions, we encourage you to complete your Stage 1 RR using this online template which will help ensure clear linking of the research question, hypotheses, analysis plans and prospective interpretation. We recognise that not all parts of the template will be relevant for all fields, but as a minimum please ensure that your manuscript includes the design planner in Section 9 of the template. Full guidelines for RRs at Royal Society Open Science can be found here. Please mention COVID-19 prominently in your Stage 1 manuscript title and/or cover letter to assist the journal administrative team in expediting your submission.

Further updates to this initiative will be reported on my blog, on my twitter feed, or on the Royal Society publishing blog.

** UPDATE 17 March: PLOS Biology have now also joined this initiative

** UPDATE 18 March: Nature Human Behaviour have now also joined

** UPDATE 20 March: All seven PeerJ journals have joined

** UPDATE 23 March: 450 reviewers, and rising, have now joined the rapid review network 

** UPDATE 23 March: Collabra: Psychology have joined

** UPDATE 25 March: We have issued our first Stage 1 in-principle acceptance for this initiative at Royal Society Open Science

** UPDATE 8 April: We have issued our second Stage 1 in-principle accepance for this initiative at Royal Society Open Science. It is also the first example of an accepted RR in chemistry. 

** UPDATE 15 April: We have issued our third Stage 1 in-principle acceptance, and the first example of an accepted RR in immunology/microbiology.

** UPDATE 30 April: We have accepted our first completed Stage 2 submission (with results). The rapid response network is also approaching 800 reviewers.

** UPDATE 1 May: We have issued our fourth Stage 1 in-principle acceptance, which is also the first example of an accepted RR in viral bioinformatics. 

** UPDATE 18 May: We have issued our fifth Stage 1 in-principle acceptance 

** UPDATE 27 May: We have accepted our second completed Stage 2 submission (with results). The rapid response network is now approaching 850 reviewers. 

** UPDATE 2 June: We have issued our sixth Stage 1 in-principle acceptance. Protocol is temporarily embargoed until Stage 2 submission.

** UPDATE 9 June: We have issued our seventh Stage 1 in-principle acceptance

** UPDATE 10 June: We have issued our eighth Stage 1 in-principle acceptance. Protocol is temporarily embargoed until Stage 2 submission.

** UPDATE 9 July: Nature Communications have now joined the initiative. 

** UPDATE: 27 September: We have published our third completed Stage 2 submission (with results). The rapid response nework is now approaching 900 reviewers.

** UPDATE: 8 October: We have published our fourth completed Stage 2 submission (with results).

 ** UPDATE: 21 October: We have published our fifth completed Stage 2 submission (with results).

** UPDATE: 10 February 2021: We have published our sixth completed Stage 2 submission (with results).

** UPDATE 29 September 2021: We have published our seventh completed Stage 2 submission (with results)

** UPDATE 25 November 2021: We have published our eighth completed Stage 2 submission (with results).

Monday, 11 November 2019

Guest post by Claudio Tennie: Why I am resigning as Associate Editor from Proceedings B today

The following is a guest post published at the request of my colleague, Dr Claudio Tennie, University of Tübingen.
I have always been fond of Royal Society Proceedings B. And yet, today I am resigning in protest as one of their Associate Editors. What happened? 
Earlier this year, a group of people, spearheaded by Dr. Ljerka Ostojic, approached Proceedings B with a well-versed request: that it should adopt Registered Reports. Yet, to our dismay, they declined to do so.
We are now very aware of the various replication crises in many fields. A lack of robust findings is not surprising, and is indeed the logical outcome, of the current system. To be blunt, this system actively selects for bad science. In order to (once again) explain how and why Registered Reports can drastically help this situation, a comparison between science and car crash testing might be helpful.
It is safe to say that none of us would like to live in a world where all cars are advertised as having five star crash test ratings, but where, in reality, many should really rate as zero stars. This would be the expected case in a world where car crash outcomes were measured by car makers and selected by car-sellers. Why? Because capitalistic forces would select both for invalid crash testing and biased crash test reporting. Allowing Registered Reports is the logical equivalent to checking crash test dummies before they are being used in car crashes and also then publicising all outcomes of all crash tests.
Likewise, in science, we want to know which hypotheses find support and which do not. And we want to use the best methods to arrive at these conclusions. Currently, we often use sub-ideal methods, which alongside the inherent biases towards publishing positive findings, selects for bad science. As a result, it is even not clear what proportion of positive findings within the suspiciously large mountain of positive findings are valid. The current situation is an absurd and truly unbearable situation – wasting time, money and energy galore. We urgently need to change it.
Of course, an especially efficient policy is to properly check crash test dummies pre-test; and to publish all crash test results. Registered reports creates exactly this situation for the scientific field. In Registered Reports, methods are properly checked before they are applied. And the eventual publication must report all results – and will be published regardless the specific outcomes. While this does not mean that every study can be a Registered Report – there are exceptions to the rule (see the FAQ section here) – many should be. As a result of this simple and compelling logic, the number of journals adopting Registered Reports is constantly increasing.
I was therefore very disappointed to witness Proceedings B refusing to adopt Registered Reports. Moreover they did so on the very unconvincing grounds that one of their sister journals (Royal Society Open Science) already allows them. The general problem persists with every (suitable) journal that refuses to allow Registered Reports. Proceedings B should adopt Registered Reports. But because they refuse to do so, I must protest. I am therefore resigning as an Associate Editor at Proceedings B.

Claudio Tennie

Tuesday, 19 March 2019

The battle for reproducibility over storytelling in cognitive neuroscience

Here is my twitter thread on our upcoming Discussion Forum on reproducibility in cognitive neuroscience at the journal Cortex. I've posted it to my blog because, weirdly, it appears on twitter to be broken on some browsers (but not others!) To see it on twitter, start here.


A late-night thread on reproducibility and in cognitive neuroscience, including our upcoming series of (rather punchy) comment pieces at the journal Cortex. Gather round all ye.

Here is my editorial introducing the seven commentaries. I’m going to move through each of them here in turn, and stick around to the end of the thread to hear about two new initiatives we’re launching this year in response /1

First up, Huber et al . report how they tried to replicate a study published in . After being invited beforehand to run & submit the study by one editor, a different editor then desk rejected them once the (non-replication) results were in. /2
Sidebar: we later published Huber et al’s replication study at Cortex (thanks , we’re happy to help you out any time). You can read the paper here: /3

Good for them, but Huber & co believe the problem w/ replication in cog neurosci is deep & serious. They call for more stringent checks on reproducibility *before* publication & dynamic tracking of rep attempts & outcomes. Their full comment here: /4

Next, pushes back a little at the suggestion to select what gets published based on results, even when doing so is based on replicability. Instead he calls for a “pending replication” stamp to be placed on unverified exploratory studies /5

But wait...what about the tools we’re using? argues that the reliability of our research cannot exceed the reliability of the methods we employ. And in cognitive neuroscience this is poorly understood. It's not just about publication culture. /6

Nevertheless the often obstructive nature of peer review isn’t terribly helpful. weighs in to point out the value of adversarial collaborations for reducing bias & encouraging better theory, especially when submitted as Registered Reports /7

Do reforms to how science works take into account the scientists who DO the work – the early career researchers? & argue that unless reforms work for ECRs, they will fail. M&T suggest “replication & extension” as one solution /8

But it’s not all about incentives. calls for cognitive neuroscientists to rise above their egos and fallibilities, embrace error correction & champion reproducibility over reputation. And he is someone who practices what he preaches /9

In particular, you can read ’s recent Registered Report at Cortex where he tests the reproducibility of one of his own previous findings & concludes that the original result may be a false positive Almost nobody ever does this in cogneuro. /10 

And finally, , a former editor, takes on the newsroom culture of sci publishing. Huber et al.’s fixes will help but only superficially. To really fix these problems, he says, scientists need to take back control from publishers /11

Where does all this leave us? Cortex has been at the forefront of initiatives such as , Exploratory Reports, TOP guidelines & badges. But these are NOT enough and this year we’ll be launching two new initiatives. /12

The first is an Accountable Replications policy – ’s now famous "pottery barn rule" of publishing, which we recently introduced at Open Science. In a nutshell: if Cortex published the original study we’ll publish the replications of that study. /13

The second is an entirely new initiative, again the creation of : Verification Reports. Short articles with the sole purpose of testing the reproducibility & robustness of original studies using the exact SAME data. /14

These steps aren’t a total answer but they move us in the right direction. The recent launch of – together with the wide support the network is receiving from funders, publishers & regulators – means that reproducibility is going to be a Big Deal for many years. /15

That’s why cognitive neuroscientists need to be at the forefront of those discussions. And it’s why cog neurosci journals need to work harder to support reproducibility. That means adopting , Exploratory Reports, TOP guidelines, replication initiatives & more. /16

I will end this very long thread there! Hope you enjoy the articles (which are all available as preprints in the tweets above) and thanks to all the wonderful contributors for weighing in. Onward. /end

Wednesday, 19 December 2018

My manifesto as would-be editor of Psychological Science

**Update 8 Feb 2019: After my initial application, I'm happy to report that I've progressed to the next stage of consideration. Obviously I'm still a long way from the destination, and it will undoubtedly be an extremely competitive field of candidates, but I am one step closer. Special thanks to all the colleagues who took the time to support my nomination! I will update again when there are further developments.**

**Update 18 June 2019: I have just heard that the role has been offered to another candidate. A big thank you to everyone who supported me and to the APS for considering me. I wish the new editor -- whoever they are -- the very best of luck and would urge them to consider implementing at least some of the initiatives in my agenda below.**

This week I received a nice email informing me that I have been elected as a Fellow of the Association for Psychological Science. A warm thanks to whoever nominated me -- I have no idea who you are, but I appreciate your faith in me. 

In the spirit of using this position to achieve something meaningful, I have put myself forward for consideration as Editor-in-Chief of the journal Psychological Science. The turnover of the Journal's editorship offers the opportunity to elevate Psychological Science from being the flagship journal of the APS to becoming a global beacon for the most important, open and reliable research in psychology -- an example not just for other journals in psychology but for science as a whole.

On December 18, I submitted the following statement to the Search Committee:

I am a professor of psychology and cognitive neuroscience at the School of Psychology, Cardiff University (see here for homepage including basic CV). I currently serve as a senior section editor at six peer reviewed academic journals, including BMJ Open Science, Collabra: Psychology, Cortex, European Journal of Neuroscience, NeuroImage, and Royal Society Open Science and I previously served on the editorial board at PLOS ONE and AIMS Neuroscience. Among other initiatives, I co-founded Registered Reports, the Transparency and Openness Promotion (TOP) guidelines, the Peer Reviewers’ Openness Initiative, and the accountable replications policy at the Royal Society. In total I have edited ~200 submissions, including ~120 Registered Reports. As a senior editor of Registered Reports, I am experienced at managing teams of editors. I am a fellow of the British Psychological Society (BPS) and was recently awarded fellowship of the Association for Psychological Science (member #119281). In 2007 I was awarded the BPS Spearman Medal and in 2018 my book on the need for reform in psychology won the BPS Book Award (Best Academic Monograph category.) As chief editor of Psychological Science, I would complete the important mission that Steve Lindsay began, implementing a range of policy reforms to maximise the quality and impact of research published in the Journal.

Steve Lindsay, and Eric Eich before him, have done a superb job introducing the APS and Psychological Science to the world of open science. I am standing for consideration as chief editor on a manifesto that will consolidate and extend the mission that they began.  

1. Full implementation of Registered Reports

Psychological Science currently offers a limited version of Registered Reports in which the format is available only for direct replications of selected previous studies published in the Journal. I will expand the format to offer full Registered Reports and I will appoint a dedicated Registered Reports editor. 

2. Registered Reports Funding Models

I will commence discussions with funding agencies to support Registered Reports grant models in which a Stage 1 Registered Report to Psychological Science is simultaneously assessed by the Journal and the funder, with successful proposals achieving provisional acceptance and funding support on the same day. 

3. Accountable Replications Policy

I will introduce an Accountable Replications Policy in which Psychological Science guarantees to publish any rigorous, methodologically sound replication of any previous study published in the Journal. This initiative will be similar to the policy I recently launched at Royal Society Open Science and the European Journal of Neuroscience. 

4. Exploratory Reports

Hypothesis-testing is just one way of doing science. I will introduce a new Exploratory Reports format, similar to the initiative I helped shape at Cortex, to provide a dedicated home for transparent exploratory research employing inductive or abductive methods. This format will focus on generating ideas and testable predictions for future studies. 

5. OSF Badges

I will review the Journal’s current policy concerning OSF Badges, seeking to raise standards for the awarding of the Open Data, Open Materials and Preregistered badges. I will appoint a dedicated Reproducibility Editor to oversee this review and the badges programme. 

6. TOP Guidelines

Psychological Science is a signatory of the Transparency and Openness Promotion (TOP) guidelines. I will implement the TOP guidelines at the Journal, achieving a minimum of Level 2 across all eight standards. Among other requirements, this will mean that all empirical articles must either make anonymised study data, analytic code, and digital study materials freely available in a publicly accessible repository, or the authors must explain in the article the legal and/or ethical barriers to archiving. The appointed Reproducibility Editor will oversee the implementation and compliance with TOP. 

7. Open Peer Review

I will implement a simple policy of open review in which all reviews and editorial decision letters are published alongside the corresponding articles, with the action editor identified and reviewers retaining the choice to either sign their reviews or remain anonymous. 

8. Verification Reports

I will launch a new ultra-short report format in which independent authors are given the opportunity to repeat and expand the analyses of original data in published articles in the Journal. The format will serve to verify or challenge the original authors’ conclusions and subject the results to robustness checks. 

I am standing for this role because I believe that psychology faces one of two possible futures. In one, we fail to reform our research culture and diminish. The legacy of psychology will eventually be forgotten, along with its enormous potential in understanding the mind and helping society. In an alternate future, we seize this moment -- right now -- and lead the way in placing quality and reproducibility at the heart of our scientific mission. I’m reminded of that signature episode of Star Trek Voyager when the Doctor says to Harry Kim, desperately trying to alter the timeline and save his crew: “Somebody has got to knuckle down and change history, and that somebody is you”.

If that somebody is you then let’s do this together. Email and support my nomination as the future editor of Psychological Science.

Tuesday, 6 November 2018