Grad School Rankings...What do they mean??

In light of the recent post regarding ranking competitiveness of the UAE, I started turning over the idea of rankings in my head.  We all value rankings, whether we admit it or not.  For one, rankings help us make decisions.  I'm sure a number of us peeked at the US News and World Report ranking biomedical research programs before selecting Emory as our home.  From personal experience, I found one of the prime emphases in grant writing class to be citing the number of F31 grants awarded to Emory GDBBS students (apparently we are currently ranked program number 2 in the US, and at one point last year, we were in first place). Oddly enough, despite our high ranking in terms of NRSAs, we are only ranked number 30 in terms of biological science graduate programs, according to the US News and World Report website. How could this be?  And what does statistics have to say about this?

How Emory Stacks Up...maybe.

Indeed, some rankings are based purely on objective, raw numbers, such as the NRSA statistic. The student either recieved F31 funding, or they did not. Others, such as the US News and World report rankings, and the UAE competitiveness rankings, are based on an amalgamation of a number of factors, including some that are subjective.  I did a bit of digging to figure out what the US News and World Report numbers are based on. Perusing their website left me with more questions than answers.  Any information given on how the rankings were determined is murky at best.  The most data I could find on how they rank describes their methodology for their undergraduate ranking system (the original). As one article from their own website investigated this topic correctly points out, "The host of intangibles that makes up the [college] experience can't be measured by a series of data points."  Factors such as reputation, selectivity and student retention are cited as some of the data points that determine ranking.  In terms of quantification, however, I did not find any clear answer.  The website cites a "Carnegie Classification" as the main method for determining rank. 

The Carnegie Classification was originally published in 1973, and subsequently updated in 1976, 1987, 1994, 2000, 2005, 2010, and 2015 to reflect changes among colleges and universities. This framework has been widely used in the study of higher education, both as a way to represent and control for institutional differences, and also in the design of research studies to ensure adequate representation of sampled institutions, students, or faculty.

As you can probably gather, the quantification involved in the Carnegie Classification is never really defined and left me wondering if there is some sort of conspiracy underlying these rankings.  Reading about these rankings has left me feeling what I imagine a nonscientist feels like when they try to understand scientific data from reading pop culture articles.  Still, we continue to value these rankings, even if we have no idea what they really mean.  Yet again, we revisit the theme that there is a strong need, not only for statistical literacy, but for statistical transparency. Statistical analysis needs to be clearly laid out so that the layperson can fully appreciate the true value of a ranking.

I also wonder about how quantitative and qualitative factors that are apparently used in the Carnegie Classification are combined together to determine one final ranking value.  As we have learned in class, continuous and categorical variables simply do not mix.  Yet here and most everywhere, we see them being combined. Maybe it is beyond my level of statistical comprehension, but I wonder if there is a way to correctly combine the two?

Determining the "competitiveness" ranking of the booming UAE

Rankings are a beautiful thing. They’re important in so many different arenas, from helping us figure out what to buy on Amazon to informing policy decisions of governments. Our discussion here will lean much more toward the latter case. I hope that the case study that follows sheds light on some of the ways statistics are used to determine rankings. Indeed, a commenter on a previous blog entry of mine expressed some interest in this subject.

In the last decade, the United Arab Emirates (UAE) economy has thrived and grown in what can only be described as extraordinary fashion (literally and figuratively). The Arabian Gulf country, thus, takes its "competitiveness" in the international arena quite seriously. Recently they established a Ministry of Happiness, but for many years the UAE's Federal Competitiveness and Statistics Authority (FCSA) has been working hard to inform the government on how to align the country's progression with a vision of "long term prosperity with a balance between productivity and quality of life for the nation."

Above is a figure generated by the FCSA. It displays the UAE's world rankings over the years as generated by a number of reports and indices that measure the strength of certain sectors of a society, such as trade (WEF-GETR), information technology (WEF-GITR), and travel and tourism (WEF-TTR). Some reports measure competitiveness directly, like the WEF-GCR (solid green dots on the graph), and it is this report I would like to focus on here. How can "competitiveness," being a concept, be statistically measured?

WEF-GCR, or World Economic Forum - Global Competitiveness Report, provides an abridged methodology, as well as a detailed methodology in the report's appendix, describing their approach to constructing their report for each country. They combine 114 indicators (measures of different concepts) that matter for productivity, grouping them into numerous categories that comprise the twelve pillars in the figure above. The pillars are distributed over a hierarchy of three sub-indices, in line with three main stages of development. For example, Pillar 1 "Institutions" comprises 21 indicators such as property rights, irregular payments and bribes, efficiency of legal framework in settling disputes, and protection of minority shareholders’ interests. Already we can see how complex the report gets; how is it that 114 indicators like these are quantitatively measured?

The report methodology states that its main source of measurements is from answers from the Executive Opinion Survey (EOS). The EOS is a survey that the WEF conducts annually, collecting information on a broad range of socio-economic issues. The key point is that the respondents to the survey comprise one class of individuals only: business executives. They answer questions such as:

In your country, how strong is the protection of property rights, including financial assets? [1 = extremely weak; 7 = extremely strong]. (Property rights)

In your country, how common is it for firms to make undocumented extra payments or bribes in connection with (a) imports and exports; (b) public utilities; (c) annual tax payments; (d) awarding of public contracts and licenses; (e) obtaining favorable judicial decisions? In each case, the answer ranges from 1 [very common] to 7 [never occurs]. (Irregular payments and bribes)

In your country, how efficient is the legal framework for private businesses in settling disputes? [1 = extremely inefficient; 7 = extremely efficient]. (Efficiency of legal framework in settling disputes)

In your country, to what extent are the interests of minority shareholders protected by the legal system? [1 = not protected at all; 7 = fully protected]. (Protection of minority shareholders' interests)

In the Technical Notes and Sources for the GCR, the WEF further states: "Indicators that are not derived from the Survey are sourced from international agencies and national authorities." And in the appendix, "The computation of the GCI is based on successive aggregations of scores from the indicator level (i.e., the most disaggregated level) all the way up to the overall GCI score. Unless noted otherwise, we use an arithmetic mean to aggregate individual indicators within a category."

So generally, a measurement is calculated for each indicator based the arithmetic mean of all the answers, a score is calculated for each category based on the arithmetic mean of all its indicators' measurements, a score is calculated for each pillar based on the arithmetic mean of all its categories' scores, and a score is calculated for each sub-index based on the arithmetic mean of all its pillars' scores. Categories and pillars are weighted based on how many others there are in the group (e.g. 4 categories in a pillar = each category is weighed at 25%, regardless of the number of indices it is comprised of). These are fixed weights; however, the weight put on each of the three sub-indices (basic requirements, efficiency enhancers, and innovation and sophistication factors) is not fixed:

This is really only a short summary explaining some of the main points of how the World Economic Forum conducts its Global Competitiveness Report. There are several other reports that may be explored, many taking a similar approach in terms of having weighted pillars, categories, etc. Is the approach we've seen here the perfect way to calculate "competitiveness?" What I would think is: how can something so complex be perfect? But it is an approach that no doubt takes advantage of the best survey and statistical methods we have developed. I should not fail to mention that the GCR has been rigorously tested for statistical validity, and it holds astonishingly well in that regard.

But people in the United Arab Emirates don't need to worry about all this. All they have to do is live in their villas, play with their pet lions, and drive their Ferraris on what the WEF-GCR says are the highest quality roads in the world.