Is The Strained Economics of the Scientific Enterprise A Significant Cause for Scientific Bias?

Economists are the most peculiar kind of scientists.  In fact, if I were not drawn to the world of medical technology innovation, I would certainly entertain my nerdy-streak by becoming an economist.  Anyone who has read the award-winning book Freakonomics might agree (aside: Freakonomics is a brilliant podcast for the intellectual-at-heart). So what exactly does the economy and the scientific enterprise have to do with bias in academic research?

I may be biased, yet I believe economics has everything to do with it.  We as human beings are susceptible to incentives, however benign or malignant. Any science-minded individual who keeps a beat on the news will know that irreproducibility and moreover, retractions of manuscripts is on the rise globally.  Indeed, while a recent article .  Wherein does the culpability lay?

I argue, incentives unduly influence individual investigators yet publishers are also to blame.  It is well known that funding for the scientific enterprise in the United States is at an all-time low when controlled for costs of inflation since the termination of the NIH budget doubling in the early 2000s.  With less access to funding and a glut of Ph.D’s entering the academic job market (a worth subject for another discussion), researchers must to more with less in order to publish. Fellow blogger Austin Nuckols is wise to note “the culture of science, especially in the academic setting, follows a mantra of “publish or perish”.  The circle of life for academic research is an ultra tenous one driven by supply and demand of the NIH dollar: Win grantàperform researchàpublish à repeat.  One break in that chain is enough to sink a mid-career academic’s productivity (not to mention salary support). When jobs are uncertain every few years, it is easy to see where bias can come top-down, influencing the un-empowered graduate student to conduct research with significant bias, leading to conclusions “in our own image”.   Publishers are similarly incentivized to avoid reducing bias, despite calls to do so in high profile journals (e.g. Nature, Cell).  “Novelty” sells; and who can remember the last time a reproducibility study was featured in the high-impact “Vanity” journals?

Looking at this dismal state of affairs for the budding researcher, I feel incentivized to begin the inaugural edition of The Journal of Research Reproducibility or better yet, The Journal of Failed Experiments (And How to Avoid Doing Them).  Perhaps then, the odds of academic success in research will be in my favor. 

Cancer: The Disease of Irreproducible Results?

The U.S. Government spends $5 billion every year on cancer research yet it is no secret that this return on investment has been quite disappointing. What lies at the crux of this attitude, is the growing mistrust that permeates lab findings - corrupt data due to sloppy analysis, unstable results, poor experimental design, and most of all, a replication crisis. When a cancer study spirals into a wrong conclusion, individuals suffer, a multibillion-dollar industry of treatment loses money, and biologists get jaded. This only propagates the cycle of researchers favoring efficiency in publication over the validity of results. As a testament to this, in 2011, a team from Bayer pharmaceuticals reported that only 20-25% of studies they attempted to reproduce generated results "completely in line" with those of original publications.

It's not only a problem of results, repeated findings, etc. It's also a problem of consistent methodology. We cannot solely focus on standardizing analyses of data outputs if a given experiment cannot even be performed in another lab. A comprehensive review of current research literature suggested that essential steps in a protocol are frequently omitted from published papers. Specifically a 2013 survey of several hundred journal articles that referenced >1,700 different laboratory materials revealed that only about 50% of such materials could be identified by reading the original papers.

Ultimately, this flawed cycle of irreproducibility has shaken the grounds of public trust for research science and the advances it offers society, however science's substantive progress is highly dependent on winning back this trust. The publication and dissemination of not only provocative yet imprecise studies, but also of findings that cannot be re-performed for whatever the reason, is only further contributing to the public's lack of confidence in the veracity of a branch of science that has the potential of being revolutionary for biomedicine. And we will never elucidate the inherent problems in such findings until we've figured out a way to make them reproducible.

A Stock Solution; Asset Pricing Theory

Alright ya'll it's about to get stuffy in here. Something I have strong interest in is global markets and the stock exchange (well, strong for a biochemist with no economics training past high school). Naturally, this sector is perfect for exploring the widespread use of statistics and the unique and powerful ways they can be applied. While disciples of Benjamin Graham will warn that the valuation of a company's stock is not 100% tied to potential for an upward trend, this undoubtedly plays some role in the type of trading done on the market today. More than 50% of trading on the NYSE is done via something called High-Frequency Trading, which uses complex algorithms to buy and sell securities on the millisecond time scale for an overwhelming addition of small differences, resulting in a large profit for the companies employing these buying and trading algorithms. I'm not nearly savvy enough to describe these algorithms with sufficient detail, but I do want to discuss some aspects of these algorithms and other probability/statistics related applications in stock trading.

Investors deal with an immense amount of data, and more is generated every couple of milliseconds. As a result, a combination of instinct, experience, and sound statistical models are the professional trader's bread and butter. Various statistical tests are utilized to assess risk and confidence.

The first technique (and arguably the most central) application of statistics in stock trading is in Asset Pricing Theory. This branch of investment theory uses the calculated effects (effect size!) of various macro-economic factors, or the behavior of theoretical indices (indices track many different stocks, or sectors and are like a mean value representing how a sector is doing. Common examples are the S&P 500 or the Dow Jones Industrial Average) to forecast the expected return of an asset. I realize that all sounds a bit vague, so let's focus on a specific example:

We are all familiar with the T-statistic (departure of a parameter from its notional value and its standard error) which we use in Student's T-Tests. In the case of investing, the utility of this statistic is almost exactly the same as when we would use it to compare means. In fact, one of the foundations of investment statistics is formation and testing of a null hypothesis. In Asset Pricing Theory, the null hypothesis would propose that "the expected return of the asset is not different from the risk-free rate of return". In other words, they compare the asset in question to the performance of risk-free investments like some bonds or savings accounts. Given the historical returns of an asset and the risk-free investment of choice, an investor may find a T-statistic describing the difference between their asset (our sample of interest) and the risk-free investment (background, WT, negative control, etc.). Investors will then use the T-statistic as an indicator of the probability of observing the asset's returns under the assumption of the null hypothesis. Another similarity is that, in this branch of economics, they set their statistically significant p-value as 0.05, and utilize confidence intervals to have a better understanding of how this asset is likely to behave. Similar to our experiments, these calculations also rely on a certain "N", where a single N could be individual transactions involving this asset (in this case, higher volume stocks would be advantaged, due to a greater number of values), but this is not always the case.

This type of statistical testing is vital to many stock analysts, who will use the likelihood that a stock will perform better than a risk-free investment as part of their assessment of how to score the stock (what they should advise their advisees or their firm to do with regard to the stock). It can also indicate when a stock may be "overpriced" or "on sale". In many trading circles, the direction that a stock is likely to go short-term is less important than the absolute value of the company and what a stock of that company should be "worth", hence "Asset Pricing Theory".