Giving Credit Where Credit is Due, or How We Incentivize Bad Science

The scientific community has recently come under fire for irreproducibility and flawed methodology. Both of these are serious accusations in a community that prides itself on the rigorous and self-correcting system that has been built over the last several hundred years. Numerous articles have detailed the holes in the fabric of the system, including one particularly thorough piece that appeared in the Economist. But few have addressed how and why these holes have come to be, and in order to do that we need to take a hard look at how a good scientist’s career is formed.

            The vast majority of benchwork science is performed by graduate students, specifically PhD students in the early stages of their careers. This is a make it or break it time for them, when good results may mean a job after graduation, so naturally they are vastly overworked and, on average, horrendously underpaid. Why they choose to work under such conditions varies, but if you choose to ask them about their projects, you’ll find that they are remarkably passionate about their work. Their goals seem to center around being able to do the science they love, regardless of whether it pays well or not. This passion seems innocent until you combine it with the lax requirements for training in statistics and rigor found in many graduate programs. The NIH has taken the first steps by requiring that all predoctoral and postdoctoral grants address how reproducibility will be handled in the proposed study. But not everyone actually performs the necessary tests of their data, and it takes a lot to change a community with such an intransient culture. Unchecked, these flaws in mentoring result in hasty and botched data analysis in the short term, but even more disturbing issues down the road.

            After obtaining their PhDs, postdoctoral fellows have many demands on their time. They must run projects independently, write papers on the results of those projects, secure their own funding, and review papers that have been submitted to journals in which they themselves have published. Because their future careers depend on their funding and publishing rates, it is no wonder that these tend to be the focus of their attention. The pile of papers to be reviewed, which often gets added to by their mentoring professors, often gets overlooked in the mad scramble of eighty-hour workweeks and fast-approaching grant deadlines. This results in another hole for bad data to slip through: a hasty reviewer is less likely to catch mistakes in methods or analysis than one that is properly incentivized to perform this holy task of peer review.

            And finally we reach the final stage of a scientist’s career, the tenured professor. Their job is, if possible, more hectic than a postdoc’s. They have to combine the obligations of teaching with that of writing grants, directing multiple research projects, working with the administration, mentoring graduate students, and reviewing papers submitted to various journals. Like postdocs, they are incentivized based on their grants and publications, with some incentives for teaching. As a result, mentoring and peer review can fall to the wayside in a culmination of a career that started with failure of instruction, thus creating a self-perpetuating cycle. Students are taught very little statistics if any, and are not incentivized for participating in the peer review process, so they do not pass on those traits to their future students.

            In the end, the solution is remarkably simple. We need to include peer review, quality mentoring, and good statistical analysis as skills necessary to obtain a professorial position. Once professors are forced to embrace these qualities and pass them on to their graduate students, the system will fix itself relatively quickly. The key is forcing PhD programs to put their money where their mouth is and train their adherents in proper data analysis. This will fix the problem of accidental misrepresentation of data. Properly incentivized peer review will fix the problem of purposeful misrepresentation of data. With these loopholes closed, we can turn to the other matter of fixing field-level standards for significance, methodology, and insignificant results that were brought up in the Economist.