When deciding where to go to dinner many factors come into
play: the cuisine, the location, the décor, the price, the circumstances or
occasion for dining and the quality to name a few. Furthermore the resources available to the
patrons to allocate toward both the decision of where to dine and the act of
dining itself carry considerable weight, as they can dictate the importance
that must be placed on making a ‘correct’ decision. On one end of the spectrum we may have a
fiscally well off individual with an abundance of time trying to decide where
to grab a midweek bite. It would be
reasonable to assume this decision carries little weight and the wrong decision
can be made without a risk of significant consequence. On the other end we may have a financially
strapped person, with little free time trying to pick a restaurant for an
anniversary dinner. Here the resources
that are being spent, being more limited, most likely carry more value and as
such the decision probably carries more pressure to be correct. Now let us say the person lives in a city
with hundreds of restaurant options.
Given this person’s aforementioned lack of resources to devote to
collecting all the data to make this decision from each source, let’s say the
restaurant’s literature, they have to turn to tools to assist them, restaurant
reviews. Years ago, the trend was to
trust expert opinion, such as restaurant critics, societal awards, and infamous
guides such as Michelin. However, these
organizations also had limited resources and could only review a select number
of establishments and usually with typically only one or two visits per
restaurant. What if the perfect
restaurant for our individual restaurant was not reviewed? Then Zagat was
launched, where diner’s opinions and reviews were collected and then
reviews/ratings curated from this pool, thus allowing more restaurants to be
reviewed and more samples taken per restaurant, albeit with the sacrifice of decreasing
the specialized knowledge of the experts in drafting reviews. Now let us operate with the assumption that
there is an objective if not absolute quality being measured and make the
further assumption that expert knowledge and experience lends itself to a more
accurate measurement of said quality.
For this, let us assume experts actively try to avoid bias, such as
recall bias, whereas the proletariat would not, presenting more subjective
opinions and measuring the wrong metrics. In order to offset the drawbacks of using
crowd opinion, the reviews were curated by experts and furthermore the
selection bias exists that it was only people who knew of Zagat and actively
sought to contribute who submitted reviews.
Yet the aim of Zagat was still clear and the people that contributed
ideally did so to help create a useful guide that aspired to be more toward the objective side on the objective-subjective spectrum. Additionally, you had to review a minimum
number of restaurants to submit to further select the pool. However, still not every restaurant was
reviewed. Then Yelp came along, where
any location was reviewed by anyone and full reviews were posted, giving
everyone a wealth of data to use in their decision making process. Quality was
further sacrificed as now almost all barriers to entry and expert opinion were
eliminated. Now a person who had never
dined at a three star Michelin and had no frame of reference of quality could
award a 5 star rating to a restaurant.
Let us go back to the individual trying to make a decision for the
anniversary dinner. If they turn to Yelp,
they have a wider selection to choose from, and while the reviews lack internal
validity, they are a wider sampling of people and may be more generalizable to
a larger population and have more external validity. If they turn to Michelin,
they have more internal validity, yet given the selection bias of it being only
reviews of the expert population it may not have external validity and the
population of restaurants to choose from is much smaller. As a side note, Yelp has the caveat of
considerable selection bias, one may even see it as citation bias, as reviewers
elect to review, and are more likely to make the effort to review a notable
experience, be it positive or negative while the non-crowd sourced guides
reviewers are hired and instructed to review restaurants no mater how notable
they were. I feel it comes down to the
individual at this point and if they feel their own opinions are more in line
with restaurant critics or ‘yelpers.’
In science a similar problem is faced; there is a wealth of
information, considerable pressure to make the correct decisions as to what to
accept and what to reject and not enough resources to allocate to making these
decisions using all available information.
To solve this issue systems of reviews and curating were put in place so
ideally only the best quality work makes it into the collective scientific
knowledge pool. Currently, this is the
peer-review system, which emphasizes expert opinion as a metric to judge a
project’s quality and worth not unlike the expert restaurant critics. This system probably has reasonable internal
validity, at least within a journal, but may be lacking in external
validity. The major difference between
peer-review and restaurant critics though is that the main job of reviewers is
to be scientists and peer review is voluntary and consequently secondary. As such often not enough effort can be given
to scrutinize and tease apart submitted work.
Furthermore, only a few people make this decision before it is given a
seal of approval. Perhaps stemming from
the societal wave that is given prominence to Yelp, PubMed Commons and PubPeer
have been introduced as a post-publication review processes. Similar to Yelp, full comments can be posted
by anyone with an opinion, which allows for more review than a few select
reviewers. However, more similar to
Zagat, this a is a select group of people who have chosen to join and
contribute, and who are all scientists, who ideally meet a minimum of
specialized knowledge. Furthermore, as
is stated in an interview with the creators of PubPeer, scientists will tend to
hold themselves to a certain standard, with the comments having to be based
upon publically verifiable information and be substantive. The comments, similar to Zagat, are
curated/moderated by experts. Now the
sort of citation bias still exists where people choose to comment and as such
probably only comment when it is notable.
Nevertheless, as this person is electing to review they may be more
likely to allocate an appropriate amount of time to properly scrutinize the
work, as opposed to those who are simply assigned to review a project or
manuscript. Furthermore the only
qualification to review is being a part of the scientific community so it may so
happen that the review or comments may not be by the ideal or most appropriate
reviewer. This however may make the
review more generalizable. Additionally,
a huge strength of the scientific community is the advancement that comes from
integrating a variety of perspectives, which is a foundation of these systems.
The major drawback of these new review processes is that
they are post-publication and thus the information being reviewed is already
part of the literature. Perhaps we
should take another look to the restaurant review world to help us further
solve the problem of scientific literature curating and scrutiny. One solution would be to pay the journal
reviewers and buy out some of their time so they would be less likely to slack
on the reviewing process. Another
solution may be to have scores be assigned to published works on PubMed Commons
and PubPeer. These could be voluntary
reviews or they could be reviews by paid experts. The scoring would ideally be standardized not
unlike the Jadad scoring system for clinical trials. If anyone has other connections to draw between scientific literature review and restaurant review or disagrees with some of the connections I drew, I would love to hear about it in the comments.
After viewing Dan Ariely’s TED
presentation: The
Bugs in our Moral Code, I found myself astounded by the clear parallels between
Dan’s social psychology experiments and the current state at which the
scientific community finds itself. Briefly, an individual’s propensity to cheat
to a certain degree (defined by Dan as personal fudge factor) is controlled through
not only the associated risk, but also the perception of others - chiefly those
within our own community. This is a common modality that can easily sway the
minds of scientists if perceived as beneficial to either the funding of their
work or the legitimacy of their beliefs or ideas. This can manifest in
something as simple as the lack of a proper experimental control to that of
turning a blind eye to confounding data on a “make or break” grant submission.
While instances of direct data
fabrication are few and far between according to groups such as Retraction Watch and The
Scientist, the inundation of published experiments in highly regarded
journals found to be un-reproducible is only beginning to come to light. Editorials
on this subject first began to surface from names like Nature in 2012 with the
release of: Must
try harder. This article argues the arrival of an endemic of scientific “sloppiness”
citing the overwhelming number of novel cancer therapies that fail to reach
clinical trials due to inadequate pre-clinical data that cannot be reproduced. In recent times, nothing screamed out at me “non-reproducible” quite as much as
the story of Dr. Charles Vacanti at Brigham and Women’s Hospital in Boston MA. The
publication was first heralded as the greatest
advancement in stem cell technology of this century: STAP. Its retraction
and verdict of scientific misconduct later resulted in the destruction of the careers
of many highly regarded scientists as well as the suicide of one of the Japanese
co-authors. This instance could be owned up to the ever present “publish or perish”
mindset inherent of running a successful lab in today’s
funding environment, or simply the presence of one dishonest scientist with an
enlarged personal fudge factor. Regardless of the cause, these events demand a
proactive advance towards the dissemination of highly reproducible studies assessed
through strict guidelines imposed by the leading scientific journals. This tenet
is supported by Dan’s social experiments in which an honor code is introduced,
reducing the generalized cheating.
Other
editorials have argued that “Reproducibility
will not cure what ails science” stating that open access to data is the
only “cure”. With the advent of big data experiments, data mishandling becomes inherent to the nature of the experimental plan. Personally, I could not agree more with the push for more stringent
data reporting not only in terms of raw data calculations but also in the final
statistical analysis of such studies. PubMed Commons aims to provide a
secondary approach in which reproducibility and many other subjects can be
discussed in a forum based method inside of the scientific community in regards
to specific publications. At the end of the day, while the experimental integrity
lies at the hands of the scientists, the affirmation is at the sole discretion
of the peer review.
