Invisible police in senior academia

I have this recurring experience on social media where I say something that, to me, seems bleedingly obvious but ends up offending senior academics. What happens next is always the same: a series of private communications, admonishing me for my tone, holding me responsible for the failure of other scientists to choose to do better quality science, and asking me to rethink my “strategy”.

As one professor wrote to me privately the other day (and no, I will never reveal their identity or the identity of anyone else who contacts me privately): “I think you are really sabotaging your cause with this kind of behavior. I recently was discussing [Registered Reports] with someone and they basically said that they have a bad taste in their mouth about them simply because of your behavior as their primary advocate. I hope you will take a breath and think about how to best achieve your goals (which I largely share) - I don't think belittling the people you want to convince is the way.”

It was a heartfelt response from someone I like and respect to this tweet, where I referred to senior academics in neuroimaging as “unique and beautiful snowflakes”. This isn’t alt-right rhetoric – it’s a quote from Fight Club (one of my favourite films) that I think aptly and ironically summarises the primitive feudal system in the life sciences. As I wrote nearly five years ago, “Cognitive neuroscience is littered with petty fiefdoms doing one small study after another – making small, noisy advances.” I also write about this in my book. My message is always the same: scientists asking important questions do what’s necessary to answer them, and if that requires working in consortia then that’s what they do. Those who need to take these steps but refuse to try are usually driven by a combination of careerism and delusion.

The nature of these responses I get from professors tells me something interesting: that a lot of scientists – and for whatever reason they usually tend to be senior academics – misunderstand why I use social media. They assume that it’s part of some strategy or tactic, or a tool in an ongoing programme of open science advocacy. I know that’s how some people use twitter (and they do it very well), and perhaps my twitter feed looks like that from time to time, but it’s NOT how I use it and it’s not why I joined twitter in the first place.

Going right back to 2012, the reason I joined twitter was to offer the public (and non-scientists especially) some behind the scenes insights into the life of a working scientist. If people are interested in hearing these thoughts then they can tune in. If not they can ignore me or even block me, which is absolutely fine. I’ve tried to stay true to this mission by following three simple rules.

1.    Always be honest. On a lot of issues I say nothing, but when I do say something I always give my honest view, and provided that opinion isn’t libelous I would say exactly the same thing if you asked me in a one-to-one conversation, in a media interview, or in the question time of a talk. When I respond to you on twitter I imagine that you are sitting in front of me. I don’t censor my own speech online except to avoid ending up in court. “But that’s a bad strategy!”, exclaim the professors. “You are attacking the people who have power, rather than diplomatically trying to change their minds” – as another professor put it to me in an email. NO. Again, you’re confusing me for a politician who uses social media to persuade people to agree with them. I use social media to explain to people what I think, and to learn what they think, not to tell them what I believe they should think. 

2.    As a general rule, I tend to avoid criticizing individual scientists for their statements or practices, not because I have any problem with people who do (provided they aren’t  abusive or threatening I think this is a vital and much under-valued function in science), but because most of my interests lie in how groups behave and changing large-scale systems. I do make exceptions to this, e.g. as recently in my response to Brian Wansink.

3.    I don’t have an “internet persona”. The very thought of this makes me cringe. For better or worse (probably worse), what you see on here is exactly what I’m like in real life.

I’m happy to take personal criticism. Maybe it's an Australian trait, but as I said to one professor, I actually hugely appreciate this criticism, especially when it’s personal. In the past, this kind of criticism has given me pause. That said, I prefer it to be public because that better serves the purpose of why I joined twitter in the first place – to pull back the curtain on academia. I don’t really see the point in having extensive one-on-one debates with senior academics about what I say on twitter. Let’s have these debates outside and then those who are interested can learn something about what senior academics care about and how they interact. As I was discussing with Tracy King last night, there is a huge web of senior power in academia that is invisible and intractable to most people. Let’s reveal it.

Now you may well say that my approach on social media works against the broader “strategic goals” of open science but - brace yourself - I don’t really care if it does, and there’s not much point trying to convince me otherwise. The day you convinced me of that would be the day I left twitter because it wouldn’t be me anymore doing the tweeting – it would be some faceless “open science” entity named Chris Chambers.

And although it's not the point of this post, while we're on the subject: from a "strategic" point of view I think we’re doing ok. Registered Reports have hit nearly 150 journals and appear to be working as intended; we’ve launched the TOP guidelines across 5000 journals and organisations; the PRO initiative is changing journal policies to favour open data and materials; we’ve launched a new Accountable Replications policy at Royal Society Open Science (and had two submissions already in the last two weeks); we’ve established the UK Reproducibility Network and the UK Network of Open Science Working Groups; we’re working with the British Neuroscience Association to advance their impressive agenda in supporting reproducibility and transparency; I spend a lot of time helping people (on twitter and privately) with their Registered Reports submissions, policy launches and other open science challenges, and much more. We are steering the ship literally every day toward more rigorous, reliable science, so that the next generation can be rewarded and required to do high quality work. To most (not all) of the professors who email me to complain about my tone: remind me again what you're doing to further this mission?

Perhaps this is all in spite of me being on social media and my tone. If so then so be it – my tone on here is who I am. But I doubt it. Within reasonable bounds, I doubt anything I say on here matters very much in the grand scheme. What each of us does out there is so much more important. We’re witnessing a remarkable passage in the history of science, and for better or worse my words are just one human’s humble observations of that history as it unfolds. Onward.

Are there special rules for Greeks entering the UK?

This is a guest post by Cardiff University PhD student (and EU national) Loukia Tzavella, reporting how she was required to do a language test to pass immigration controls enroute to the UK. If you have had a similar experience, please share in the comments below.  

Last week I attended a scientific conference in Lyon, France and was ready for my journey back to Gatwick Airport, UK. The first ‘special treatment’ of the day came from the employee at bag drop. She kept looking at my passport and flicking through the pages. She told me that I couldn’t go to London without a visa and after a few minutes she looked up and said, well we don’t really know how to “handle” these type of passports (i.e., Greek) which of course caused me to react with the reply “It’s an EU passport!”

I had to wait a few minutes because she still couldn’t deal with the situation and had to consult with other colleagues. After having to explain that Greeks have the right of free movement in EU countries, including the UK (!), I left my bag and met my colleagues who were waiting for me to move on to security. I had already had quite a rough day for personal reasons so at that point I couldn’t wait to reach the gate and be on my way back home. Writing this, I deleted the word “home” a few times. I have been in the UK for almost 8 years. Should I call it home? 

We were rushing to get to the gate when we saw passport control and I thought “Will I have to feel different again for being Greek, maybe apologise for the inconvenience”?

I don’t know if this was passport control for EasyJet or UK destinations in general, but there were no lines and it seemed like we were going to go through fast and reach the gate, which was right behind passport control. My UK colleagues went through instantly. Many airport employees so far were not very keen to speak or listen to English, so I spoke French to the passport inspector and put on a smile. He looked at my passport and then stared at me for a few seconds, and then did so repeatedly several times.

I checked with him if everything was okay and he requested my driving license. No explanation offered. After I gave him the license I asked why this was required and he ignored me. He then pulled out a paper titled “Test of Greek Language”, written in Greek. Of course I reacted to this as I had no idea why this was happening and asked why I needed to complete a test. He then squeaked “Why, why” mockingly and told me to stop asking questions. At that point I was extremely frustrated, surprised this was happening and overall furious. However, I was worried that if I made a scene I could be held for questioning and miss my flight. I couldn’t help it, though, and as I was going through the test I asked again why, in French this time.

He finally told me that immigrants obtain fake passports from Greece and travel to France to get entry into the UK illegally and so they have to check Greek passengers travelling to UK destinations. He thus needed to know if I was an illegal immigrant. Okay, so what would happen if I made a stress-induced mistake on the test and seemed to make corrections (that would look suspicious) or didn’t have my driver’s license? What are the guidelines on this? My rights, as I know them, may include rejecting, or failing, to do the test as I only need my valid EU passport to travel, so what would happen if I did? I still have no idea! He kept staring at me and after 2 mins, before I had finished the language test, he said “OK, leave”. I packed my things and ran to my colleagues to explain what had happened.

Are language tests for Greek citizens part of an official agreement between France and Britain? Does this only occur in French airports, and if so under whose instruction? Does this happen in other countries when trying to reach the UK? Is this practice targeted at Greeks only? If so, is this a practice for all Greek nationals or does it depend on how we look? In any case, this policy should be explained upfront so that citizens don’t feel discriminated against and distressed during the whole process.

Anyone should be happy to comply with such procedures when the reasons are explained in a polite (i.e., professional) manner. Previous reports of fake passports don’t include any information about special procedures for UK destinations. Reports mention that fake passports can be Mexican, Brazilian, Argentine, Syrian and German, so I would be curious to know whether there were also language tests for these citizens.This sounds like a serious issue, and if regulations are needed for stricter border controls, advance notice to citizens holding Greek passports should be provided or at least a brief explanation should be given before all the extra checks are requested.

At UK airports I am frequently asked why I am travelling to the UK and how long I am planning to stay, but border staff have always been professional, so I am not sure whether this is an issue solely for Lyon Airport. I don’t know whether this unofficial ‘directive’ has anything to do with the upcoming Brexit, or whether other EU nationals have experienced similar treatment. But one thing is clear: there should be official information on this ‘special treatment package’ for Greek citizens entering the UK and this apparently new procedure of imposing language tests. 

Why I hate the ‘tone debate’ in psychology and you should too

There is an extremely boring, seemingly indestructible, debate going on in psychology about the ‘tone’ of scientific debate in our field. The root of this peculiar argument is that the nature of some discussion surrounding open science and reform is apparently too negative, too critical, and too personal, and that this pushes people away from doing better science.

The entire debate is so nauseatingly self-absorbed. It really pains me to see so many smart colleagues who I respect and admire devoting so much of their time to being concerned about this issue. Hey, it’s their time, who am I to judge. But what a waste it is of our collective energy, and I wish people could see why.

A few unfiltered thoughts, and then I will never speak of this again.

1. There’s this idea that open science will attract more 'disciples' if it comes across as having a more positive, inclusive tone. Goodness me, what a load of honking bullshit this is. Open science will attract individual adopters for three reasons: (1) when scientists grow a conscience and appreciate that their public mission demands transparency and reproducibility; (2) when scientists decide to take advantage of individual incentives (career and social) for being open (e.g. Registered Reports, joining SIPS etc.); (3) when funders, journals and institutions make it a requirement. All of these are in progress. The cold embrace of open science by gatekeepers and regulators is in the post – it is only a matter of time before transparent, reproducible practices will be required if you want to spend public money. That’s why I tell early career researchers to get ahead now because the ground is shifting under your feet, and I am one of the people shifting it. What ‘tone’ we set in public discourse about science (and open science) is thoroughly irrelevant to all of this; indeed, thinking that this changes anything is nothing more than hubristic navel gazing.

2. Open science isn't an evangelist movement that needs to save souls from damnation. Yes it is a political (r)evolution. Yes, the more people adopt it the better. But you won’t get them to do it through the tone of critical discussion or by preaching. We get scientists to adopt open practices by changing the system to make openness attractive and essential.  

3. Leave Satire Alone. If we are so thin skinned that we can’t laugh at ourselves from time to time (and, jesus, even learn something from it), we are already lost. 

4. Don’t fall into the trap of being concerned about tone on behalf of others. No, you are not the voice of early career researchers or the vulnerable or Nixon's silent majority. All of these communities can speak for themselves, and they do, especially in the flat landscape of social media. Yes, your anecdotes about some researcher who really wanted to be open but was put off by a sharp twitter discussion count for precisely nada.

5. Psychology is a discipline that has no problem praising the living daylights out of individuals when they publish some stunning result or get a big grant. Hey guess what: critique follows the same rules. You live by the sword...

6. Ignoring the ‘tone debate’ isn’t the same as turning a blind eye to abuse. There is painfully little evidence that any of the much touted tone infractions constitute abuse, or that abuse in the open science community is any greater than the baseline in academia. It isn’t the job of the open science community to solve all problems in academia.

7. Follow Wheaton’s Law but don’t let demands on civility stand in the way of delivering the criticism that is required in service of science. Sometimes this will require setting a tone that risks causing offence. That's ok, so long as it is in service of the mission. I say this repeatedly, but remember who you serve. It isn’t yourself and it isn’t your colleagues. It isn’t early career researchers or vulnerable scientists, or scientists without a lot of resources. Science serves the public who pay for science. Ask yourself how our debates about ‘tone’ appear to the people who fund the work we do. Ask yourself how pathetic this all looks and why other sciences aren’t doing it. Then snap out of it.

8. Finally, and this is the kicker: Discussions about ‘tone’ were originally orchestrated (rather shrewdly, it must be said) by opponents of reform as a way of diverting attention from systemic changes to science that (a) threaten the organic structures that underpin their own power and (b) would expose their own sloppy research practices. We are in an intensely political period and whenever you argue about tone you are playing your opponents’ game by your opponents’ rules. While we tear ourselves sideways worrying about such trival nonsense, they are smirking from their thrones.

This is the last thing I will ever say about the ‘tone’ debate. We need to change the way psychological science works, not the way some people talk about psychological science.

Open-ended, Open Science

In this special guest post, Rob McIntosh, associate editor at Cortex and long-time member of the Registered Reports editorial team, foreshadows a new article type that will celebrate scientific exploration in its native form.

Exploratory Reports will launch next month and, now, we need your input to get it right.

Chris has kindly allowed me to crash his blog, to publicise and to gather ideas and opinions for a new article type at Cortex. The working name is Exploratory Reports. As far back as 2014, in his witterings and twitterings, Chris trailered a plan for Cortex to develop a format for open-ended science, a kind of louche, relaxed half-cousin to the buttoned-up and locked-down Registered Reports. Easier tweeted than done. We are now preparing to launch this brave new format, but even as we do so, we are still wrestling with some basic questions. Does it have a worthwhile role to play in the publishing landscape? Can it make a meaningful contribution to openness in science? What should its boundaries and criteria be? And is there a better name than Exploratory Reports?

Visitors to this blog will have a more-than-nodding familiarity with misaligned incentives in science, with the ‘deadly sin’ of hidden flexibility, and with the damage done to reliability when research conducted in an open-ended, see-what-we-can-find way, is written into the record as a pre-planned test of specific hypotheses. No one doubts that exploratory research has a vital role to play in empirical discovery and hypothesis generation, nor that it can be rigorous and powerful (see recent blog discussions here and here). But severe problems can arise from a failure to distinguish between exploratory and confirmatory modes of enquiry, and most perniciously from the misrepresentation of exploratory research as confirmatory.

A major driver of this misrepresentation is the pervasive idealisation of hypothesis-testing, throughout our scientific training, funding agencies, and journals. Statistical confirmation (or disconfirmation) of prior predictions is inferentially stronger than the ‘mere’ delineation of interesting patterns, and top journals prefer neat packages of strong evidence with firm impactful conclusions, even if our actual science is often more messy and… exploratory. Given a more-or-less-explicit pressure to publish in a confirmatory mode, it is unsurprising that individual scientists more-or-less-wittingly resort to p-hacking, HARKing, and other ‘questionable research practices’.

Regulars of this blog will need no further education on such QRPs, or on the mighty and multi-pronged Open Science movement to reform them. Still less will you need reminding of the key role that study pre-registration can play by keeping researchers honest about what was planned in advance. Pre-registration does not preclude further exploration of the data, but it keeps this clearly distinct from the pre-planned aspects, eliminating p-hacking, HARKing, and several other gremlins, at a stroke. The promise of enhanced truth value earns pre-registered studies an Open Practices badge at a growing number of journals, and it has even been suggested that there should be an automatic bonus star in the UK Government’s Research Excellence Framework (where stars mean money).

This is fine progress, but it does little to combat the perceived pre-eminence of confirmatory research, one of the most distorting forces in our science. Indeed, a privileged status for pre-registered studies could potentially intensify the idealisation of the confirmatory mode, given that pre-registration is practically synonymous with a priori hypothesis testing. A complementary strategy would therefore be for journals to better value and serve more open-ended research, in which data exploration and hypothesis generation can take precedence over hypothesis-testing. A paper that is openly exploratory, which shows its working and shares its data, is arguably as transparent in its own way as a pre-registered confirmatory study. One could even envisage an Open Practices badge for explicitly exploratory studies. 

Some journal editors may believe that it is typically inappropriate to publish exploratory work. But this is not the case at Cortex, where the field of study (brain-and-behaviour) is relatively uncharted, where many research questions are open-ended (e.g. What are the fMRI or EEG correlates of task X? What characterises patient group Y across test battery Z?), and where data collection is often costly because expensive technologies are involved or a rare or fleeting neuropsychological condition is studied. It is hard to estimate how much of the journal’s output is really exploratory because, whilst some authors have the confidence to make exploratory work explicit, others may still dress it in confirmatory clothing. If publication is their aim, then they are wise to do so, because the Action Editor or reviewers could be unsympathetic to an exploratory approach.

Hence, a new article type for exploratory science, where pattern-finding and hypothesis generation are paramount, and where the generative value of a paper can even outweigh its necessary truth value. A dedicated format is a commitment to the centrality of exploratory research in discovery. It also promotes transparency, because the incentives to misrepresentation are reduced, and the claims and conclusions can be appropriate to the methods. Some exploratory work might provide strong enough evidence to boldly assert a new discovery, but most will make provisional cases, seeding testable hypotheses and predictions for further (confirmatory) studies. The main requirements are that the work should be rigorous, novel, and generative.

Or that is the general idea. The devil, as ever, is in the detail. Will scientists – as authors, reviewers and readers - engage with the format? What should exploratory articles look like, and can we define clear guidelines for such an open-ended and potentially diverse format? How do we exercise the quality control to make this a high-status format of value to the field, not a salvage yard for failed experiments, or a soapbox for unfettered speculation? Below, a few of the questions keeping us awake at night are unpacked a little further. Your opinions and suggestions on these questions, and any aspect of this venture, would be most welcome. 

1. Scope of the format. At the most restrictive end, the format would be specific for studies that take an exploratory approach to open-ended questions. Less restrictive definitions might allow for experimental work with no strong a priori predictions, or even for experiments that had prior predictions but in which the most interesting outcomes were unanticipated. At the most inclusive end, any research might be eligible that was willing to waive all claims dependent upon pre-planning. Are there clear boundaries that can be drawn? 

2. Exploration and review. A requirement for submission to this format will be that the full data are uploaded at the point of submission, sufficient to reproduce the analyses reported. To what extent should reviewers, with access to the data, be allowed to recommend/insist that further analyses, of their own suggestion, should be included in the final paper? 

3. Statistical standards. Conventional significance testing is arguably meaningless in the exploratory mode, and it has even been suggested that this format should have no p-values at all. There will be a strong emphasis on clear data visualisation, showing (where feasible) complete observations. But some means of quantifying the strength of apparent patterns will still be required, and it may be just too radical to exclude p values altogether. When using conventional significance testing, should more stringent criteria for suggestive and significant evidence be used? More generally, what statistical recommendations would you make for this format, and what reporting standards should be required (e.g. confidence intervals, effect sizes, adjusted and non-adjusted coefficients etc.)? 

4. Evidence vs. theory. Ideally, a good submission presents a solid statistical case from a large dataset, generating novel hypotheses, making testable predictions. The reality is often liable to be more fragmentary (e.g. data from rare neuropsychological patients may be limited, and not easily increased). Can weaker evidence be acceptable in the context of a novel generative theoretical proposal, provided that the claims do not exceed the data? 

5. The name game. The working title for this format has been Exploratory Reports. The ambition is to ‘reclaim’ the term ‘exploratory’ from a slightly pejorative sense it has acquired in some circles. Let’s make exploration great again! But does this set up too much of an uphill struggle to make this a high-status format; and is there anyway a better, fresher term (Discovery Reports; Open Research)?

Rob will oversee this new article format when it launches next month. Please share your views in the comments below or you can feed back directly via email to Rob or Chris, or on twitter.