Different types of unreliable research call for different solutions

Our readings this week looked at multiple facets of unreliable research: Deception/cheating, unreplicability, and un-generalizability.  

 

Dan Ariely discusses cheating in the context of Wall Street, but it is easy to see how the behaviors he teases out in a laboratory setting with undergraduates could be relevant for data faking in science as well. Ariely discusses the finding that the propensity to cheat increases dramatically when one believes members of one’s “in-group” (in this case, Carnegie Melon students who see a student with a Carnegie Melon hoodie cheating). Ariely also discusses how under most circumstances, people would only cheat a little bit. He described this as a “personal fudge factor”, related to an individual’s desire to see his or herself as a good, moral person. You can easily imagine how this might translate into a research setting; the belief that your idea is correct (even if the data don’t support that) could lead one to believe that they are only cheating a little bit rather than a lot (which, to be clear, they are.)

Unreliable research can also result from from sloppy, improper use of statistics or poor data collection processes which find “significance” where there is none. An article for the Economist summarizes this problem quite well. This is in many ways the most frustrating area of unreliable research because it is the one that feels most correctable. We can disincentivize cheating, though realistically there will still be people who cheat. But rigorous and appropriate statistical analysis should be achievable for every published paper, if journals are willing to have every paper reviewed by a statistician (and realistically, every University is willing to foot the bill via increased paper submission fees).

Finally, un-generalizable research is research that was properly collected and analyzed, but is sensitive to small changes in experimental conditions. I would argue that these studies are the “features” of unreliable research, as discussed in Jared Horvath’s piece for Scientific American. While research that doesn’t generalize may be disappointing to those who thought they were studying a widespread phenomenon, this research still ultimately adds to our knowledge base.

These three types of unreliability are different in both their causes and solutions, and thus should not be lumped together. Cheating, or faking data, is completely unacceptable and should have serious professional consequences. Poor use of statistics can be corrected with education, and through structural mechanisms such as professional statisticians being employed by journals specifically to review the data analysis methodology of the paper. Un-generalizable research should be revealed through replications specifically designed to test the generalizability of the phenomenon: Using a different mouse line, a different age range, data collected at a different time of day. And unlike the previous two types of unreliable research, un-generalizable research shouldn’t reflect poorly on the researchers who conducted the experiments; instead it’s a familiar call back to the drawing board.

The unseen biases by the peer review process

In the article “Why you can’t always believe what you reading scientific journals” the author discussed one of the major issues in life science research, the peer review process. The article focuses on the benefits and disadvantages of a new tool called the “PubPeer.” The reporter was able to ask and discuss many questions with the PubPeer founders. There were many issues that were particularly interesting to the general scope of the problems with bias in sciences.

The founders began by stating this this website has served to be very useful since it allows for a centralized commentary for published work. Interestingly just recently, this tool was able to uncover a controversial stem cell fiasco. With the help of an anonymous commentator the public was able to uncover the published fraud. I thought this was exceptional since I always wondered what the process was to uncover misconduct in already publish work. What I believe is even more interesting, is the fact that this work was accepted and published –showing the flaws in the peer review process.

Another point brought up in this article was the fact that most misconduct occurs in life sciences. However, now this makes sense since it would be very easy to show that for example a mathematics equation was faulty by making a visible mistake. On the other hand, in biology there are many areas were misconduct can present itself. One of the ones that matches the classrooms interests is the idea of science bias. This might be even more difficult to pick up since there might be many different types of biases like in the experimental design, the data collection, the analysis and even the interpretation. Which are difficult to identify once the data has been published and the authors was not detailed enough in their documentation.

Overall, this was a very interesting read and pointed out new tools that could one day replace the faulty peer review process currently begin used today, to avoid biases in science.

Scientific bias and other shenanigans

Bias is a very common problem in scientific research.  I was surprised to find that there are so many types of bias that may occur in any given study. It is important to identify bias at all levels of a study and these levels include: pre-study stage, study stage and data analysis stage. The overall problem with bias is that results can become skewed and create results that are not correct. One interesting aspect of bias at these different stages is that they can be limited through careful planning and awareness. For example, selection bias is a very common form of bias that occurs during the pre-study stage. In this form of bias a study may become compromised due to recruitment strategies for selecting study participants that rely on criteria that favor one group over another. This may lead to decreased probability of identify a difference between groups when one is present. One way of combating this bias is to use random selection. By randomly assigning individuals to groups in the study the chance of having a selection bias for one group over the other is greatly reduced. 

One aspect of Bias that is important to understand is the difference behind intentional and un-intentional bias. Essentially bias is never completely accounted for in a study. No study is perfect. But what is interesting is when the bias is known to exist or even created by by the researchers but they do not correct for it in their study. This is what I refer to as intentional bias also known as academic misconduct. I think the primary reason for not making these essential corrections are largely due to the pressures put on researchers to publish. Without the papers it is hard to get funding. It is not hard to believe that someone would be tempted to publish tampered results if it means they will be able to support themselves financially. Another reason could be the fame. An example of fame seeking would be the story of Hwang Woo-suk, a stem cell researcher who in 2005 was found to have fabricating a large number of experiments leading to papers published in top-tier journals. This was a heavily reported incident. In this case, it is shown that bias can have a negative effect on public opinion of scientific research. This opinion is important for continued growth and support of scientific research.