tag:blogger.com,1999:blog-5747401412673787565.post3479686370385548990..comments2024-03-18T06:19:28.852+00:00Comments on NeuroChambers: The Dirty Dozen: A wish list for psychology and cognitive neuroscienceChris Chambershttp://www.blogger.com/profile/10437328364681252945noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-5747401412673787565.post-36658555591197637152012-07-23T15:29:54.967+01:002012-07-23T15:29:54.967+01:00How would these items, particularly 1 and 4, work ...How would these items, particularly 1 and 4, work for qualitative studies? Making these datasets anonymous can be difficult, and sometimes impossible, for example. I am worried these kind of stringent measures would work towards further blocking this important side of psychological research, though it does merit consideration. Perhaps we should work towards fostering an atmosphere of responsibility rather than trying to police it. I'm not sure of the stats, but what is the ratio of known research frauds committed today to research article published, compared to years past?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-73957863039231770492012-07-20T04:30:19.283+01:002012-07-20T04:30:19.283+01:00It is great to see the increased awareness about p...It is great to see the increased awareness about problems in psychological research and suggestions for improvements. <br /><br />The problems are not new (Sterling, 1959) and one solution is also quite old, namely to increase statistical power (Cohen, 1962). <br /><br />Whereas a priori power analysis can reduce the need for data fudging, post-hoc poewr analysis can be used to detect data fudging. <br /><br />http://www.utm.utoronto.ca/~w3psyuli/PReprints/IC.pdfUlrich Schimmackhttp://www.erin.utoronto.ca/~w3psyuli/homepage.htmnoreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-76906925719874899282012-07-19T17:08:22.748+01:002012-07-19T17:08:22.748+01:00Great post Chris!
I agree with most of your point...Great post Chris!<br /> I agree with most of your points you raised except for the speed review thing. I can't see a point & click system working. How about limiting the length of reviews to one page with clear section to fill in?. <br />I can't see the MCQ version working if reviewers don't justify their choice (even briefly).<br /><br />I really enjoyed reading it!Frednoreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-42237826781845044462012-07-19T14:41:13.325+01:002012-07-19T14:41:13.325+01:00As far as I know Simonsohn's method hasn't...As far as I know Simonsohn's method hasn't been published yet. Ed Yong's article (linked above) mentions that the paper on the technique will soon be submitted, so I imagine it will be at least a couple of months before we see it in accepted form - and that's assuming it is accepted quickly.<br /><br />I think your reservations are completely sensible. In Simonsohn's defence, the 'one-man IRB' quip is me (being facetious) rather than any kind of position he's given himself.<br /><br />Will be interesting to see how it all pans out.Chris Chambershttps://www.blogger.com/profile/10437328364681252945noreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-17753378930479757142012-07-19T14:13:58.384+01:002012-07-19T14:13:58.384+01:00one-man internal affairs bureau
Even reading this...<i>one-man internal affairs bureau</i><br /><br />Even reading this line makes me shudder. Has his paper outlining his method come out yet? Even if he is 3/3 in catching real fraud it's not a tenable situation that he be the sole arbiter of which studies are investigated. I will admire his work when he is open about it.Bashirnoreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-44368319612997678792012-07-18T19:04:25.804+01:002012-07-18T19:04:25.804+01:00Hi Tim, thanks for commenting.
1 - couldn't a...Hi Tim, thanks for commenting.<br /><br />1 - couldn't agree more. Excellent idea to tie in with Masters research. As you say, we really need a dedicated research fund to sponsor direct replications. I'll keep this in mind, I'm sure it is something we can push!<br /><br />2 - I expected speed-reviewing to be the most controversial suggestion. As scientists we've come to both love and hate in-depth review. We love it when a reviewer helps us improve a manuscript; and we hate it when they make fatal errors and kill our paper unfairly. What I'm suggesting is a kind of middle ground where trade off some of this in-depth reviewing for crowd wisdom.<br /><br />Re 2b - I really like the idea of reviewers being reviewed, but it needs to be done carefully and independently. Reviewers already do this as a favour and it would be easy for a mechanism of meta-assessment to deter them from doing anything at all. At the same time, such assessment is important!Chris Chambershttps://www.blogger.com/profile/10437328364681252945noreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-58719376974655200442012-07-18T18:57:17.674+01:002012-07-18T18:57:17.674+01:00Wow, thanks for the detailed feedback. Re the poin...Wow, thanks for the detailed feedback. Re the points of disagreement/discussion:<br /><br />2 - I agree; this is essentially Blackstone's principle in criminal law, and it is basically irrefutable in my book: http://en.wikipedia.org/wiki/Blackstone%27s_formulation<br /><br />4 - Good point. Perhaps there could be exceptions, as you point out, where Observations could be published without internal replication (provided the original experiment provides strong enough evidence)<br /><br />5 - This is just my personal view, but I think that if we're going to release data then we need to surrender ownership of it once published (I realise that is an extreme view that many will disagree with). I just feel that data should be in the public domain for all to see and use as they see fit. It's for reviewers to decide if the data was used appropriately, and naturally the source should be acknowledged whenever the data is used. To me this seems no different from citing other people's published papers. Using someone's data without acknowledging them would be tantamount to data plagiarism and would be academic misconduct.<br /><br />7 - That's a fair point. The last thing we want to do is disadvantage young scientists. So perhaps the cap could be per staff member/PhD student rather than per lab. Without imposing something that limits the quantity of output, we're never going to be able to give quality the attention it deserves.<br /><br />12 - Yes, reviewers can (and often do) help improve manuscripts. But they also get things wrong (often) and make mistakes which lead to erroneous rejections. By sampling a small number of error-prone reviews we guarantee a noisy selection mechanism. I'd like to see a combination of in-depth review and rapid-review. For instance, perhaps at first submission a larger number of speed reviewers could give ratings (including on replication value) and then if the average is high enough the paper is selected for in-depth review by one or two of those reviewers who - crucially - gave ratings that were closest to the mean.Chris Chambershttps://www.blogger.com/profile/10437328364681252945noreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-73311449614922303752012-07-18T18:43:33.845+01:002012-07-18T18:43:33.845+01:00Hi Chris, I just read your post and tweeted it. Ve...Hi Chris, I just read your post and tweeted it. Very well said, I agree that these pressures need to come from outside, and what better external force than funding agencies! As you say, we've already proven as a community that we're incapable of making these changes on our own.Chris Chambershttps://www.blogger.com/profile/10437328364681252945noreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-16301855113638008772012-07-18T16:24:14.275+01:002012-07-18T16:24:14.275+01:00Great read! Thanks.
My two cents:
1. Top journals ...Great read! Thanks.<br />My two cents:<br />1. Top journals should make their names not through high IFs, but rather through openness. I propose they transform into databases where each article is an entry that has following contents: the paper itself, possibly reviewer comments, the raw data, replication study reports.<br />Furthermore, they should sponsor replication studies: replications are actually perfect material for masters' theses or even graduate students in their first year learning the experimentation skills. Why not award prizes or small funds to such people?<br /><br />2. I have reservations against the speed reviewing as well. On the other hand, there should be some changes in the review system. First of all, not all reviewers are selected randomly. Nice articles (fashionable topics; written by high profile researchers; ...) seem to "attract" other reviewers than not-so-nice articles. That is a bias at the editor side.<br />2b. Review/evaluation of reviewers, as suggested by others, is a great idea. And such evaluations should become an integral part of track records, next to the output you deliver as an author.Anonymoushttps://www.blogger.com/profile/06408559623511378013noreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-28419751338145500872012-07-18T15:33:41.775+01:002012-07-18T15:33:41.775+01:00A nice post, Chris, as always.
Here's my two ...A nice post, Chris, as always.<br /><br />Here's my two cents' worth:<br /><br />1. Agreed<br />2. Agreed, but as you say, we must be very careful to guard against false accusations. It would be far better, in my opinion, to let fraudulent data be published (and it might then be subject to failed replications etc.) than to accuse someone innocent.<br />3. Agreed<br />4. Agreed – but sometimes this might be tricky in cases involving patients. What if patients of this particular type are very rare? And/Or the patient deteriorates/recovers after the first testing session (and how would you prove that rather than report that your replication attempt had failed?)? It’s not really practical in every case to provide a replication, in-house, or otherwise.<br />5. YES! I have no problem with data exploration, but as said above we should replicate the “successful” results in a new set of participants, and justify arbitrary decisions (e.g. outlier removal) and provide supplemental data confirming that the results were qualitatively the same if different decisions were made.<br />However, I’m not sure about making raw data freely available to all. This could mean re-analysis of my data to answer a question for which it is not suited, and without due acknowledgement of where the data came from, and could perhaps be in competition with my own on-going work. By all means submit the raw data with the publication – perhaps reviewers should be encouraged (required?) to check the analyses/alternatives. Individual readers should be allowed to request raw data, but perhaps only for particular uses or with permission from the original author(s). Or perhaps we could make raw data available only after some delay so that the original authors get first crack at any further analyses they may want to conduct (perhaps use their published data as a control group for some on-going work?).<br />6. Agreed<br />7. Your proposed cap makes me nervous. A PI might decide that he/she would prefer to (strategically) publish papers on various aspects of their lab’s work over others. Perhaps PIs have a grant they’re about to submit and want to give the impression that their lab is expert in this area. Or perhaps they’d have to choose between publishing grander work from their senior postdoc who is further ahead in their project than a new PhD student. I think this suggestion has the potential to disproportionately hurt junior scientists (PhDs/postdocs).<br />8. Agreed<br />9. Agreed <br />10. Agreed<br />11. I see your point, but I don’t really have a problem with the current system. I think it’s clear and does the job most of the time (and my surname does not begin with an A or a Z!).<br />12. Reviewing does take time, and perhaps the current system isn’t perfect. But the speed reviewing you suggest might hurt the scientific process. I have (almost) always found reviewers’ suggestions to be helpful and IMPROVE my work. Speed-reviewing, as you describe it, could mean that I’d lose this. And junior researchers who are still finding their feet might lose out the most if we lost reviewers’ comments. I’d prefer to see reviewers acknowledged in some way for their time, and contribution to the paper (some Journals e.g. Frontiers already do something like this).Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-44995405117657040152012-07-18T15:30:34.093+01:002012-07-18T15:30:34.093+01:00Hi Chris - very nice and thoughtful post. I agree ...Hi Chris - very nice and thoughtful post. I agree on all 12 points. Of course, the hard part is getting these changes implemented. Many of the ideas your proposed have been around since the 1960s, and we have had the technology to implement them for over a decade, and yet we are still a long way from reform. We face a classic collective action problem: While the system as a whole would benefit from these changes, individual actors/journals/authors do not benefit from being the first mover. <br /><br />By far the best way to solve a collective action problem is if an external force changes the incentive structure so that individual actors benefit from reforming. It is clear to me that only the granting agencies can apply this force, by rewarding scientists (using grant preferences) who follow good practices and who submit to good-practice journals.<br />http://filedrawer.wordpress.com/2012/04/17/its-the-incentives-structure-people-why-science-reform-must-come-from-the-granting-agencies/Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-14048676294965250142012-07-18T15:08:51.270+01:002012-07-18T15:08:51.270+01:00Thanks Dave. I was very much inspired by your piec...Thanks Dave. I was very much inspired by your piece, which readers can find here:<br />http://www.davenussbaum.com/crimes-and-misdemeanors-reforming-social-psychology/<br /><br />I wonder if any journals would consider trialling a speed-reviewing system. In the first instance, they could keep their initial approach untouched and simply contact additional speed reviewers. It would be interesting to see if the final decision (rendered purely based on the detailed reviews) was predictable from a sample of speedy reviews. <br /><br />I couldn't agree more that we need data before making any systematic changes to the peer review system. At the same time, my instinct is that we would gain more by having a greater number of less in-depth reviews than under the current system (which, let's face it, often results in fairly superficial reviews anyway)Chris Chambershttps://www.blogger.com/profile/10437328364681252945noreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-67742794685717944572012-07-18T15:03:26.317+01:002012-07-18T15:03:26.317+01:00Yes I agree, exploratory work is crucial - just lo...Yes I agree, exploratory work is crucial - just look at all the major discoveries that stemmed from serendipity. By splitting up the categories of publications like this we can have the best of both worlds...Chris Chambershttps://www.blogger.com/profile/10437328364681252945noreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-71879387185233097132012-07-18T14:49:14.882+01:002012-07-18T14:49:14.882+01:00A great list, Chris!
The big question to me is ho...A great list, Chris!<br /><br />The big question to me is how do we implement these changes. One important starting place is to discuss and debate them and build consensus around which changes are necessary and how they are best executed. Posts like this one are an important step in that direction.<br /><br />One of the interesting things that Simmons, Nelson, and Simonsohn bring up is that even if journals, universities, and funding agencies move slowly to adopt reforms, researchers themselves can announce in their own papers the steps they've taken (obviously you can't randomly inspect your own data, but you can determine your sample size, outlier cutoffs, and data analysis strategy in advance). The hope is to create a social norm -- an environment in which it is understood that these are the expectations for good research, leading more people to adopt them and empowering reviewers to ask whether these steps have been followed.<br /><br />Lastly, I find your speed-reviewing technique intriguing. I have to admit I would have some reservations about simply introducing it. Still, it would be really interesting to do some pilot testing. For example, why not have normal review serve as a control group and compare the resulting acceptances, rejections, and ratings to a sample of speed reviews. If they come out the same then you've got a decent argument for considering a switch.Dave Nussbaumhttp://www.davenussbaum.comnoreply@blogger.comtag:blogger.com,1999:blog-5747401412673787565.post-65102346984481182012-07-18T14:33:32.650+01:002012-07-18T14:33:32.650+01:00Great post! I especially like the division between...Great post! I especially like the division between experiments and observations. I believe strongly in exploring data, which leads to invaluable observations. Obviously, science would be completely lifeless without exploration! But such observations could certainly be evaluated using a different criteria than those borrowed from hypothesis testing (though, of course, correcting for multiple comparisons is already a part solution, if done properly). Also, any interesting observations should be subjected to further replication using the less flexible experiment paradigm (including pre-registration). This could also deal with the issue of parameter fitting in imaging. While it may be OK to explore lots of way of looking at your data, the final approach should also work with new unseen data. I.e., replicated using exactly the same data acquisition and analysis procedure.StokesBloghttps://www.blogger.com/profile/00890404304081225894noreply@blogger.com