Incentive to Care

Scientific discovery and technological innovation can do and have managed extraordinary feats. However, today we hear so much questioning the reliability of the findings, and countless resources have been essentially wasted funding projects that never reach fruition. When we examine the system of scientific discovery and publication on paper, we find that it is a rigorous process that requires careful planning and execution of experiments meant to answer questions. The same question must be answered from multiple angles, proven and re-proven with each proof repeated to ensure that the manuscript sent to the reviewers is the best work the lab can offer. Multiple reviewers must then scrutinize the results and methods and send feedback often involving the original authors to run more experiments to cover any holes that might exist in the work. Finally, after publication, the article in question offers just one small answer to a problem still layered in questions, and it is the responsibility of other researchers to retest these data as they try to find their own answer to the problem.

Why then, with so many checks and balances, does this system seem to fail? In the article from The Economist “Trouble at the lab”, the author explores some of the specific issues that lead to the above problems with one of the major problems being the lack of incentive for researchers to engage in proper scientific practice. The culture of science, especially in the academic setting, follows a mantra of “publish or perish”, and journals incentivize positive and novel findings over replications of experiments or negative findings. These positive findings are much more likely to have a lower statistical power than the negative results that are found, meaning that more bias is published and fewer useful results. Additionally, other researches spend countless hours and dollars trying similar kinds of experiments not knowing that those methods have already been tried. But what researcher can afford to try to publish all their negative results or try every replication that’s in the relevant literature?

Looking at Dan Ariely’s “The Honest Truth about Dishonesty”, we can see that the human tendency to look after one’s own interest is phenomenon that is as omnipresent as it is complex. Applying some of the experimental conditions to those of the everyday conditions that many scientists face, I cannot blame any one scientist for behavior. In Ariely’s experiments, when the participants see another test taker (the actor) who very obviously cheated on the short math exam and easily profited from it, the incidence of cheating rose drastically. People compete with each other, not with integrity, for survival, and the same applies to a scientist. He might know he needs to replicate an experiment, but there’s only so many lab hours and reagents, and the draft to be sent out needs that last final spark to push it through as opposed to another replicant of a previous Western Blot. In a system where we feel like we’re being wronged, the laboratory lifestyle being very easy to imagine as one of those systems, it’s much more conceivable to justify self-promoting behavior because it’s the only way to compete with one’s colleagues who are engaging in the same practices.

However, this does not need to be the end-all for this story. More and more today, there are resources and entities seeking to remedy these problems we find in the scientific community by incentivizing behavior such as publishing methods, data, and negative results. A fellow blogger, Katherine Bricker, references in her piece the journal Cell’s mandate for investigators to list their exact methods and reagents in their “Star Methods” tab. In Ariely’s work, he found that when he asked students to recite the Ten Commandments before taking the exam, the incidence of cheating dropped to 0% astoundingly. Taking the time to remind investigators and scientists of their obligations to truthful and rigorous scientific practice and actually offering incentive for them to do so, we can change this tendency and start using our time and money more effectively and lay the foundation for stable and meaningful science in the future.

https://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

http://www.cell.com/star-methods

https://www.youtube.com/watch?v=G2RKQkAoY3k

The Double-Edged Blade of Occam's Razor

Occam's Razor is the concept that the simplest explanation is the most likely explanation. Unless you are particularly susceptible to magical thinking, you likely employ this principle frequently in your everyday life. It is also, probably, quite useful to you.

Can't find your keys? Odds are a unicorn didn't eat them, you just forgot where you put them. 

End up at the Clermont Lounge instead of the Clairmont Inn? You'll probably want to check your GPS instead of looking for a wormhole. (Or just enjoy the Clermont Lounge... But I digress.)

Occam's Razor is also incredibly important to developing models to explain biological data.  Statisticians often speak of overfitting models (see blog post by Ashley Cross) as a common temptation and problem for scientists. Yes, it may be possible to construct an equation that perfectly explains each and every one of your data points. This strategy not only disregards the inherent variability within biology, it also makes it much more difficult to apply the model to other systems. 

However, the apparent simplicity of making a simple model should also be taken with a grain of salt. Simplicity is easy and easy explanations are comforting. It is much more reassuring to believe that you mistyped into your GPS than to believe you fell into a worm hole. But imagine the experience you'd have disregarded if you actually did fall into a worm hole. 

Models are derived from your sample which a) by chance, may not represent the population you hope to extrapolate the results to, or b) could be impacted by an enormous number of factors that you have no control over, or don't know exist. Good hypotheses are based off of educated predictions and previous knowledge, but absolutely nothing is completely understood.  Biology is complex, and we often take complexity for granted. The people over at LessWrong give several examples of this. Most relevant to this discussion is the example of Thor, the angry god to which ancient people attributed lightning strikes:

"The human brain is the most complex artifact in the known universe.  If anger seems simple, it's because we don't see all the neural circuitry that's implementing the emotion...  The complexity of anger, and indeed the complexity of intelligence, was glossed over by the humans who hypothesized Thor the thunder-agent."

Though most of us probably don't pray to Thor during every rainstorm, we may still be equally likely to oversimplify as to overcomplicate. Models are incredibly useful and important. They save time and energy, but they are descriptions and not explanations. If you have read Part G of Intuitive Biostatistics, it is easy to see that choosing how to construct or compare models can be a tenuous process if not given enough though. It is always important to understand the problem you are trying to address, but we must be careful as scientists to understand we have limited understanding. 

Giving Credit Where Credit is Due, or How We Incentivize Bad Science

The scientific community has recently come under fire for irreproducibility and flawed methodology. Both of these are serious accusations in a community that prides itself on the rigorous and self-correcting system that has been built over the last several hundred years. Numerous articles have detailed the holes in the fabric of the system, including one particularly thorough piece that appeared in the Economist. But few have addressed how and why these holes have come to be, and in order to do that we need to take a hard look at how a good scientist’s career is formed.

            The vast majority of benchwork science is performed by graduate students, specifically PhD students in the early stages of their careers. This is a make it or break it time for them, when good results may mean a job after graduation, so naturally they are vastly overworked and, on average, horrendously underpaid. Why they choose to work under such conditions varies, but if you choose to ask them about their projects, you’ll find that they are remarkably passionate about their work. Their goals seem to center around being able to do the science they love, regardless of whether it pays well or not. This passion seems innocent until you combine it with the lax requirements for training in statistics and rigor found in many graduate programs. The NIH has taken the first steps by requiring that all predoctoral and postdoctoral grants address how reproducibility will be handled in the proposed study. But not everyone actually performs the necessary tests of their data, and it takes a lot to change a community with such an intransient culture. Unchecked, these flaws in mentoring result in hasty and botched data analysis in the short term, but even more disturbing issues down the road.

            After obtaining their PhDs, postdoctoral fellows have many demands on their time. They must run projects independently, write papers on the results of those projects, secure their own funding, and review papers that have been submitted to journals in which they themselves have published. Because their future careers depend on their funding and publishing rates, it is no wonder that these tend to be the focus of their attention. The pile of papers to be reviewed, which often gets added to by their mentoring professors, often gets overlooked in the mad scramble of eighty-hour workweeks and fast-approaching grant deadlines. This results in another hole for bad data to slip through: a hasty reviewer is less likely to catch mistakes in methods or analysis than one that is properly incentivized to perform this holy task of peer review.

            And finally we reach the final stage of a scientist’s career, the tenured professor. Their job is, if possible, more hectic than a postdoc’s. They have to combine the obligations of teaching with that of writing grants, directing multiple research projects, working with the administration, mentoring graduate students, and reviewing papers submitted to various journals. Like postdocs, they are incentivized based on their grants and publications, with some incentives for teaching. As a result, mentoring and peer review can fall to the wayside in a culmination of a career that started with failure of instruction, thus creating a self-perpetuating cycle. Students are taught very little statistics if any, and are not incentivized for participating in the peer review process, so they do not pass on those traits to their future students.

            In the end, the solution is remarkably simple. We need to include peer review, quality mentoring, and good statistical analysis as skills necessary to obtain a professorial position. Once professors are forced to embrace these qualities and pass them on to their graduate students, the system will fix itself relatively quickly. The key is forcing PhD programs to put their money where their mouth is and train their adherents in proper data analysis. This will fix the problem of accidental misrepresentation of data. Properly incentivized peer review will fix the problem of purposeful misrepresentation of data. With these loopholes closed, we can turn to the other matter of fixing field-level standards for significance, methodology, and insignificant results that were brought up in the Economist.