NASHVILLE, June 19, 2008 – Adding a new twist to the sometimes raucous debate over the best way to “rank” business schools, an economist from the Vanderbilt Owen Graduate School of Management has come up with a novel approach to measure the “inherent quality of an institution.” The result of this experiment is part new ranking and part commentary on the shortcomings of all rankings…including his own admittedly “tongue-in-cheek” version.
Owen School professor Mike Shor – a noted expert in game theory – has developed a rogue methodology that tries to go beyond ranking a school’s performance based primarily on the quality of the students who choose to attend it. Instead, he has attempted to quantify the “market value” that each school contributes to its student body overall.
Using the widely accepted “predictive criteria” of starting salary and average GPA/GMAT scores from the top 50 schools in the 2009 US News & World Report ranking, Shor tried to eliminate the statistical problem known as the “self-selection bias,” or the reality that admitted students will almost always choose to go to a higher ranked school and possibly distorting the rankings outcome. Calculating expected data points for the “average” B-school within the US News Top 50, Shor re-ranked schools based on their performance above or below the average to identify which schools actually provide the greatest relative increase in expected earnings after graduation.
Shor’s unorthodox analysis was conducted blindly (and in the middle of the night), with the best and worst performers including (with Owen/Vanderbilt coming in at #6):
Top 5
Focusing on outputs versus inputs
According to Shor, focusing on the concept of market value is akin to the distinction between corporations who recruit the best talent versus those who actually do a better job of guiding that talent toward its potential. “My admittedly half-baked methodology tries to measure the efficacy of a school with its existing talent pool, so students have a clearer indication of which schools are more likely to increase their market value down the road,” he says.
The genesis of Shor’s new methodology idea came from another research project he was contemplating: how best to determine and measure “outcomes” from hospitals conducting bone marrow transplants. “The fact that some hospitals only accept cases with a high chance of success created the same sort of self-selection bias that afflict the B-school rankings and make apples-to-apples assessment challenging,” adds Shor.
He notes that existing rankings inherently reward schools that admit students with high credentials, but believes his approach can make such admissions behavior a negative. “If two schools have the same market outcome in terms of starting salary, the school that started with a lower-credentialed class should actually be rewarded for making the most increase in the market value of the student,” Shor posits. “It’s a direct indicator of the educational value that the school is providing to the students.”
In the end, Shor acknowledges that the methodology resulting from his academic tinkering is less than perfect, but does serve to highlight the shortcomings of B-school rankings overall. “There really is no magic formula for determining which school is better than another, and starting salary is only one possible statistical measure of outcomes,” he says. “The rankings are certainly here to stay, but my aim is to further the discussion over what constitutes an appropriate analysis when it comes to judging schools.
”Shor’s complete “re-ranking” of the Top 50 schools from US News using the publicly available data, along with his witty methodology, disclaimers and rankings commentary, is available on his blog.
