PubMed Commons and PubPeer, the potential Zagat-Yelp hybrids of scientific literature

When deciding where to go to dinner many factors come into play: the cuisine, the location, the décor, the price, the circumstances or occasion for dining and the quality to name a few.  Furthermore the resources available to the patrons to allocate toward both the decision of where to dine and the act of dining itself carry considerable weight, as they can dictate the importance that must be placed on making a ‘correct’ decision.  On one end of the spectrum we may have a fiscally well off individual with an abundance of time trying to decide where to grab a midweek bite.  It would be reasonable to assume this decision carries little weight and the wrong decision can be made without a risk of significant consequence.  On the other end we may have a financially strapped person, with little free time trying to pick a restaurant for an anniversary dinner.  Here the resources that are being spent, being more limited, most likely carry more value and as such the decision probably carries more pressure to be correct.  Now let us say the person lives in a city with hundreds of restaurant options.  Given this person’s aforementioned lack of resources to devote to collecting all the data to make this decision from each source, let’s say the restaurant’s literature, they have to turn to tools to assist them, restaurant reviews.  Years ago, the trend was to trust expert opinion, such as restaurant critics, societal awards, and infamous guides such as Michelin.  However, these organizations also had limited resources and could only review a select number of establishments and usually with typically only one or two visits per restaurant.  What if the perfect restaurant for our individual restaurant was not reviewed? Then Zagat was launched, where diner’s opinions and reviews were collected and then reviews/ratings curated from this pool, thus allowing more restaurants to be reviewed and more samples taken per restaurant, albeit with the sacrifice of decreasing the specialized knowledge of the experts in drafting reviews.  Now let us operate with the assumption that there is an objective if not absolute quality being measured and make the further assumption that expert knowledge and experience lends itself to a more accurate measurement of said quality.  For this, let us assume experts actively try to avoid bias, such as recall bias, whereas the proletariat would not, presenting more subjective opinions and measuring the wrong metrics.  In order to offset the drawbacks of using crowd opinion, the reviews were curated by experts and furthermore the selection bias exists that it was only people who knew of Zagat and actively sought to contribute who submitted reviews.  Yet the aim of Zagat was still clear and the people that contributed ideally did so to help create a useful guide that aspired to be more toward the objective side on the objective-subjective spectrum.  Additionally, you had to review a minimum number of restaurants to submit to further select the pool.  However, still not every restaurant was reviewed.  Then Yelp came along, where any location was reviewed by anyone and full reviews were posted, giving everyone a wealth of data to use in their decision making process. Quality was further sacrificed as now almost all barriers to entry and expert opinion were eliminated.  Now a person who had never dined at a three star Michelin and had no frame of reference of quality could award a 5 star rating to a restaurant.  Let us go back to the individual trying to make a decision for the anniversary dinner.  If they turn to Yelp, they have a wider selection to choose from, and while the reviews lack internal validity, they are a wider sampling of people and may be more generalizable to a larger population and have more external validity. If they turn to Michelin, they have more internal validity, yet given the selection bias of it being only reviews of the expert population it may not have external validity and the population of restaurants to choose from is much smaller.  As a side note, Yelp has the caveat of considerable selection bias, one may even see it as citation bias, as reviewers elect to review, and are more likely to make the effort to review a notable experience, be it positive or negative while the non-crowd sourced guides reviewers are hired and instructed to review restaurants no mater how notable they were.  I feel it comes down to the individual at this point and if they feel their own opinions are more in line with restaurant critics or ‘yelpers.’

In science a similar problem is faced; there is a wealth of information, considerable pressure to make the correct decisions as to what to accept and what to reject and not enough resources to allocate to making these decisions using all available information.  To solve this issue systems of reviews and curating were put in place so ideally only the best quality work makes it into the collective scientific knowledge pool.  Currently, this is the peer-review system, which emphasizes expert opinion as a metric to judge a project’s quality and worth not unlike the expert restaurant critics.  This system probably has reasonable internal validity, at least within a journal, but may be lacking in external validity.  The major difference between peer-review and restaurant critics though is that the main job of reviewers is to be scientists and peer review is voluntary and consequently secondary.  As such often not enough effort can be given to scrutinize and tease apart submitted work.  Furthermore, only a few people make this decision before it is given a seal of approval.  Perhaps stemming from the societal wave that is given prominence to Yelp, PubMed Commons and PubPeer have been introduced as a post-publication review processes.  Similar to Yelp, full comments can be posted by anyone with an opinion, which allows for more review than a few select reviewers.  However, more similar to Zagat, this a is a select group of people who have chosen to join and contribute, and who are all scientists, who ideally meet a minimum of specialized knowledge.  Furthermore, as is stated in an interview with the creators of PubPeer, scientists will tend to hold themselves to a certain standard, with the comments having to be based upon publically verifiable information and be substantive.  The comments, similar to Zagat, are curated/moderated by experts.  Now the sort of citation bias still exists where people choose to comment and as such probably only comment when it is notable.  Nevertheless, as this person is electing to review they may be more likely to allocate an appropriate amount of time to properly scrutinize the work, as opposed to those who are simply assigned to review a project or manuscript.  Furthermore the only qualification to review is being a part of the scientific community so it may so happen that the review or comments may not be by the ideal or most appropriate reviewer.  This however may make the review more generalizable.  Additionally, a huge strength of the scientific community is the advancement that comes from integrating a variety of perspectives, which is a foundation of these systems. 

The major drawback of these new review processes is that they are post-publication and thus the information being reviewed is already part of the literature.  Perhaps we should take another look to the restaurant review world to help us further solve the problem of scientific literature curating and scrutiny.  One solution would be to pay the journal reviewers and buy out some of their time so they would be less likely to slack on the reviewing process.  Another solution may be to have scores be assigned to published works on PubMed Commons and PubPeer.  These could be voluntary reviews or they could be reviews by paid experts.  The scoring would ideally be standardized not unlike the Jadad scoring system for clinical trials.  If anyone has other connections to draw between scientific literature review and restaurant review or disagrees with some of the connections I drew, I would love to hear about it in the comments.