Friday 19 August 2016

Registered Reports for Qualitative Research: A call for feedback from humanities and social science researchers


tl;dr – in 2017 we are expanding the Registered Reports publication model into qualitative research for the first time. Since we’ve never done this before, we’re very interested to hear from qualitative researchers about how we can make the format attractive, and what obstacles or drawbacks we might encounter with pre-registration of qualitative studies.

Registered Reports are developing some serious momentum. What began in 2013 as a radical (and somewhat controversial) idea to curb bias in published research is rapidly becoming a standard addition to the publishing landscape. Twenty-eight journals have now signed on to offer the format, with more in the pipeline, and across a wide range of social, life and physical sciences.

For the uninitiated, Registered Reports are form of empirical publication where the peer review happens, in part, before the results of studies are known (full details here). Proposals that are scientifically robust, and which receive positive peer reviews, are then provisionally accepted for publication. Once the research is complete, the authors resubmit their manuscript with the results and discussion; it is re-reviewed and ultimately published, provided the authors adhered to their protocol, interpreted the evidence reasonably, and met any pre-agreed standards for assuring data quality.

The great strength of Registered Reports is the way it tackles bias. By accepting studies in advance of results, it prevents publication bias – the tendency for journals to selectively publish results that are considered clear or more attractive. And by requiring authors to pre-specify their research hypotheses and analysis methods, it also greatly reduces the capacity for biased reporting by authors.

As the popularity of Registered Reports has increased, so has its scholarly reach: from the initial launch within psychology and neuroscience it has expanded into political science, biology, and even physical sciences of physics and chemistry. But within every area, so far, that Registered Reports have ventured, it has been under the banner of quantitative research.

To lay my cards on the table, I'm a firm believer in the value of qualitative research. I gained a deep respect for it after being involved in this study, where we used qualitative methods (specifically, grounded theory) to explore the way parliamentarians use science and evidence in decision-making. Qualitative research can generate remarkably rich information, and in some ways it is more transparent than quantitative research. For instance, the qualitative researcher explicitly acknowledges their own bias, and – in our work at least – we publicly archived all of our data to enable qualitative replication.

Publication bias also appears to be just as much a problem in qualitative research as in quantitative research. As the authors of this study conclude:

"This suggests a mechanism by which "qualitative publication bias" might work: qualitative studies that do not show clear, or striking, or easily described findings may simply disappear from view. One implication of this is that, as with quantitative research, systematic reviews of qualitative studies may be biased if they rely only on published papers."

If publication bias is as much a problem for qualitative research as quantitative research, then there is clearly room for Registered Reports to become a popular option among qualitative researchers. This opens the door for Registered Reports to be taken up by fields such as anthropology, cultural studies, criminology, sociology, and beyond. In 2017 we will be launching Registered Reports in at least one journal that publishes papers within these fields, among other social sciences and humanities.

All that said, I am far from being an expert on qualitative methods, so if you are a qualitative researcher I would greatly appreciate your input into the design of a qualitative Registered Reports model. We need your help to get this right and to make it useful.

I am interested in any critical feedback, however positive or negative, and in a completely open-ended way. You can email me with any thoughts or leave a comment below this post. Anonymous comments are welcome. I am also interested in the views of qualitative researchers on certain specific issues, particularly: 

-- Do you believe that qualitative research within your field suffers from publication bias?

-- Have you ever had a paper rejected because, independently of the methods, the qualitative outcomes were judged to be insufficiently interesting, novel or conclusive?

-- Would you consider using Registered Reports as a submission option if it were offered at your preferred journal?

-- Which journals would you like to see Registered Reports offered within?

-- What do you see as the main benefits or drawbacks of Registered Reports in qualitative research?

7 comments:

  1. It's a good thing you are doing--I applaud it.

    And I think it is great that you are exploring the inclusion of qualitative research. I'm glad you had a good experience with grounded theory. Your openness and respect is much appreciated.

    No, I don't have a sense that qualitative research suffers from publication bias. It's interesting to wonder why, apart from the obvious point that it is most usually not hypothesis testing, so there are no cases of unsupported hypotheses. I have never had a submission rejected because the outcomes were insufficiently interesting, novel, or conclusive. I suppose that could happen.

    Sure. I would consider using Registered Reports.

    I certainly support replications of all sorts in qualitative research, and if findings were similar to past findings, I would consider that well worth publishing, provided that the past findings were of high quality (no need to publish any more low quality, trivial, or mediocre work similar to what is already out there). Typically in qualitative research, given that the data is almost invariably new and different from what was previously collected, there would be interesting individual cases introduced even if the conceptual findings were replicated. I suppose some journal editors might use original conceptual findings is a criteria for publication, in which case publication bias would be a problem, but I am not aware of that. We have not had so many replications, and when they are done, they are usually systematic involving differences of some sort from the original study. I'd love to see different researchers analyze the same data, as I have explored to some extent in my own projects. Original findings are quite possible using the same data. Researchers inevitably bring different interests, stocks of knowledge, sensibilities, analytic abilities, styles of thought and writing, and so on. Indeed it seems to me that it is often quite possible to push even an analysis of previously gathered and analyzed data forward to new knowledge. My approach is phenomenological, and so a crucial analytic operation is imaginative variation of data, which may vary among researchers and therefore afford various results even when one begins with the same empirical data.

    One challenge that comes to mind is that qualitative research depends in a great part on the researcher, and what qualitative analysis actually involves. Of course what analysis involves varies considerably with the approach used. However, the researcher is the sense making instrument. Although analytic procedures are specifiable and have been delineated, they are not mechanical and so correct results are not automatic. Indeed it is an iterative process and there are many gradations from confusion and a lack of significance to profound significance and strong conclusions. Presuming that the research is well designed and the data is collected competently, confusion and lack of significant findings is always a result of the limitations of the person doing the analysis, the level of development and insight which the analysis has achieved. A good researcher will learn from data that is well constituted to answer the research question, and whatever is learned will be significant and worth reporting. Therefore it is probably warranted to reject a submission that lacks clear results.

    (cont')

    ReplyDelete
  2. On dissertation committees, I have noticed a huge difference between quantitative and qualitative research. In quantitative research, the researcher proposes the research with great confidence--the hypotheses often even seem like a slam dunk, so well justified that in some cases, were it not the importance of science for all knowledge, one might even wonder if research is needed to support the hypotheses. However, after the analyses are performed, the situation is often quite different. It is not uncommon that few if not any of the hypotheses are supported, and then, were it not for the assuredness of the competence of the design and analytic procedures, the researcher would be completely devastated (sometimes they are in fact demoralized, always disappointed under such circumstances). Their confidence in their study drops to a low point, and confusion can only be remedied by speculation and the promise of future research. The situation is the exact opposite with qualitative research. At the outset, despite the best preparation and proposal, the qualitative research typically has grave doubts about what they will find and actually has no idea at all what it will be. Any hunches are rather pathetic, and they are not typically encouraged. Even well into the analysis, while being immersed in data, qualitative researchers can be very lacking in assurance, confidence, and any sense that the research will yield valuable results. Students invariably now believe that good results are forthcoming, because the data is good and crafted to answer the research questions. In fact it is the opposite of quantitative research in that there is always an embarrassment of riches and the challenge is not that significant results might be lacking but rather how to clearly conceptualize the overwhelming significance of what is being found. And this is no easy task. Very sadly, I have had students who quit projects--simply couldn't do it and others who insisted that because they had done sufficient work, often much more than their quantitative peers, that their confused results were sufficient to defend. It can be a long and arduous process, and the standards to which mentors hold their students are quite variable. If the student is capable and the research is done well, the final result is invariably excellent, but that simply cannot be guaranteed given the nature of the process and specifically what is required.

    Researcher flaws can be due to the lack of education of researchers--qualitative research is still not sufficiently well institutionalized in psychology curricula. They are definitely also due to a lack of talent unfortunately, sad as that is. They can also be due to a lack of effort, time spent, and willingness to tolerate ambiguity and struggle through the process of establishing emergent knowledge. It's a disturbing and ugly truth. I'm not sure what can be done about it. Standards of journals vary, according to a lot of things, and I'd imagine one of them is simply the quality of the work, including the level of insight demonstrated in the findings. And that is fair. In my view, insignificant findings is a basis for rejecting a submission, because it invariably means that the analysis has not been conducted sufficiently well. Over many years, I have been involved as an editor of and reviewer in nurturing work at various developmental stages to being publication-worthy, as is often required because of the paucity of submissions of high quality work, which almost always means high quality analysis and findings. It's very laborious and time consuming, as researchers are often without sufficient education and talent. Nevertheless, some good publications have come out of very laborious mentoring, a very different situation than one finds in quantitative research. Editors probably vary considerably in qualitative journals regarding their willingness to mentor and help develop research that is not up to publication standards.

    (cont')

    ReplyDelete
  3. It may be a big commitment to guarantee publication based on a good proposal/design. Who is going to oversee the process and actually help the researchers achieve publishable results? I think this is difficult to legislate. I suppose that editors might be willing to do the work of providing feedback that would guarantee publication quality and then overseeing the process to completion. However, quite honestly, I don't think I would be inclined to offer this kind of deal at least in the kind of research I do--phenomenological, given how time consuming, frustrating, and possibly even contentious it might be for all involved.

    I hope I am not being too negative. What I say is based on 40 years of experience.

    Fred Wertz
    Professor, Department of Psychology
    Fordham University

    ReplyDelete
  4. I have only had one publication rejected in over a decade, and that was because the BioMedCentral editor disagreed with our conclusions. She then raised a methodology objection and wouldn't back down. If you can believe it, this was just an abstract of a conference presentation. She refused to even publish the abstract.

    A year later the complete paper was published as an invited paper in a higher impact journal.

    So I have a particular interest in what you are trying to do here, especially as it involves Neuroimmune topics at the intersection of Humanities and Physical Sciences - such as the above paper exploring potential hazards from wireless-operated gadgets. Denial is something even editors have to grapple with...

    Trevor Marshall, Autoimmunity Research Foundation
    (TrevorMarshall dot com)

    ReplyDelete
  5. I would really like to follow your blog, rather than book mark it as it will only be lost in sea of bookmarks. Can you consider adding a follow button to your blog at all.

    ReplyDelete
  6. From the submission guidelines (http://www.tandf.co.uk/journals/authors/rrsp-submission-guidelines.pdf):

    "Publication of the manuscript by default comes with online supplementary material, first and foremost the pre-
    registration protocol."

    Why not include links to pre-registration in the actual paper, so everyone who reads the paper can check the pre-registration protocol?! What's the purpose of including pre-registration information only in the online supplemental material?!

    Aside from that, if i look at some of the articles "Comprehensive Results in Psychological Science" has published, i could not find any pre-registration protocols for any of them:

    http://www.tandfonline.com/doi/full/10.1080/23743603.2016.1248079?scroll=top&needAccess=true

    http://www.tandfonline.com/doi/full/10.1080/23743603.2016.1248081


    Am i missing something?

    ReplyDelete
    Replies
    1. I worry Registered Reports-format will be nearly useless when it will be allowed to keep pre-registration hidden from the readers. It starts with one journal, and next thing you know you have a whole conglomerate of journals that say they adhere to "open practices", but in the meantime keep the essential pre-registration hidden from the readers, and let it become a black-box.

      Like you state yourself in your article introducing Registered Reports: http://orca.cf.ac.uk/59475/1/AN2.pdf

      "This is particularly salient given that only 1 in 3 peer reviewers of clinical research compare authors’
      protocols to their final submitted manuscripts"

      If you allow journals to keep the pre-registration protocol hidden from the reader (e.g. by not mandating authors to simply provide a link to the pre-registration information in the actual paper), it will create a black-box where only a few people will check the pre-registration, if that many at all (perhaps only a single editor will check it, who knows).

      I sincerely hope you will think about what is happening/could happen with journals that provide Registered Reports. I reason it could be(-come) extremely important for true open science that 1) authors of Registered Reports are mandated to provide links to a) publically accessible and b) to be preserved pre-registration information in the actual article, and 2) only allow journals who adhere to this rule should be able to use/advertise that they publish Registered Reports.

      That is all. Thank you for your efforts.

      Delete