Presenting: our new college rankings.
The college admissions landscape is littered with college rankings. In 1983, US News first ranked American universities. Since then, rankings have been a fixture of the college world: they are produced by various businesses (US News, Princeton Review, Forbes, Atlantic Monthly), and heeded by students and colleges alike. To gain advantage, some universities have been alleged to manipulate their own rankings. And, while some of the factors used in the rankings are justifiable (alumni giving rate), some seem to be arbitrary (peer assessment surveys asking other colleges about your college’s ‘faculty dedication to teaching’). Each year, the methodology changes slightly, producing a slightly different list. In the end, the factors that are used to come up with the rankings seem arbitrary; the occasional change in the weighting of each factor, capricious. There is a need for a new approach.
Criteria for a ‘good’ college ranking system
- The system should be difficult to game; any ‘gaming’ of the system should actually benefit students. In contrast, consider the allegations that some schools tried to manipulate the US News rankings by encouraging more students to apply in order to decrease their acceptance rate.
- The factors measured should be relevant to students. In contrast, what Cornell’s dean thinks about the faculty dedication at the University of Texas may be irrelevant.
- The overall procedure for generating rankings should be stable from year to year. In other words, any change in the rankings between 2008 and 2009 should be explained by a substantive change in the underlying factors, not by an arbitrary change in how those factors are weighted.
The MyChances College Rankings
We have implemented the MyChances College Rankings based on revealed student preference. In this system, the college admissions process is treated like a chess tournament. The colleges play matches (which occur when 2 colleges admit the same student). In each match, there is a winner (the college that the student ends up attending) and a loser. The winner gains points; the loser forfeits them. When a high-ranked school beats a low ranked school, the high-ranked school gains few points, and the low-ranked school loses few points. If a low-ranked school beats a high-ranked opponent, it gains more points than if it beat an equally-matched opponent. After playing many games, the colleges that students prefer rise naturally to the top of the rankings.
Does the method of revealed student preference meet the 3 criteria outlined above? I believe it does.
Consider point #1 (gaming the system). Imagine that MIT wanted to beat out Harvard by trying hard to avoid admitting any students that they thought would be admitted to Harvard. They would end up succeeding in a model based on acceptance rate and yield (since their yield would likely increase), but their actual student body would be less qualified. In the revealed preference model, however, they would be less successful. They would not compete head-to-head with Harvard, so would ‘win’ more. But they would be winning against weaker ‘opponents’, earning fewer points for each victory.
For point #2 (relevance), the idea of revealed preference is that it aggregates the sum total of what matters to students – whatever those factors might be. It is likely that students behave rationally (by attending the school that they find most desirable). So long as other students share similar values, then revealed preference rankings will work well in explaining, and even guiding, their decisions.
For point #3 (stability), the tournament style system is simple and straightforward. It is responsive to changes in student preference over time. It does not rely on aggregations of various statistical factors, or college faculty survey results; nor does it depend upon arbitrary weighting of those factors.
The details of the procedure that we use to generate the rankings, and our use of chess-style Elo points, will be explained in a later post. For an academic treatment of a similar college ranking system, I recommend the working paper, “A Revealed Preference Ranking of U.S. Colleges and Universities,” 2005, by Christopher Avery, Mark Glickman, Caroline Hoxby, and Andrew Metrick (free link).
Tags: admissions, college, college admissions, college rankings, rankings, us news
[...] rankings system based on revealed preference of accepted students. Check out the article here: http://www.mychances.net/blog/2009/07/10/new-college-rankings/ [...]
There’s one minor flaw with your system. Lot’s of kids will choose a cheaper, lower-ranked college over an ivy simply because they can’t pay. Other times they choose the college because it is closer to home.
Jimbob,
That’s a good point, and brings up a few follow-up questions. What constitutes the most favored college? Should ranking all of the Ivies highly be the sine qua non of a valid ranking system? What constitutes gaming in a preference-based college ranking system? I will write another post soon that addresses these.
In the meantime, consider this: the lowest ranked Ivy is 14th in our 2009 results for National Universities. If other colleges are bribing students with scholarships to steal them away from the Ivies, it doesn’t appear to be working all that well.
[...] MyChances.net Data-driven college admission prediction « New College Rankings [...]
I think your ratings need work. For instance, I see SUNY Brockport way ahead of SUNY Geneseo, Stony Brook and Binghamton. No one would rank the schools this way. Additonally, Binghamton is relatively low on your list compared to Stony Brook.
I see a couple of UT schools ranked ahead of UT Austin, which is their flagship school and, I believe, the most difficult to get into.
I see Tufts rated well below Amherst. I think the schools are similar and Tufts is more difficult to get into.
These ratings based on student preferences could be skewed by students claiming they were accepted somewhere that they weren’t even accepted to. Also, students may attend a school closer to home for many reasons, preferring not to travel hundreds of miles further away from home, even if the “losing school” is a better school. Student preferences may also be based on where their friends/relatives are going or went.
I don’t think this rating system is an improvement at all.
Amherst has a lower acceptance rate than Tufts, and its accepted students have higher metrics across the board (SAT, etc). Though those are not used in the preference rank, it suggests that no matter which metric you use, Amherst ranks above Tufts.
The Texas issue is interesting. To a large extent, with more data, I doubt this would be a problem. However, there are theoretical reasons that UT Dallas could rank above UT Austin: UT Dallas has a huge grant program for National Merit Scholars. Additionally, perhaps UT Austin competes on a national stage, whereas the other schools compete on a regional stage. In this case, Austin would lose out frequently to the heavy hitters (HYP, etc). However, UT Dallas might not, since it would not be competing for the same students. This would be reflected in the rankings.
The SUNY schools seem to be misranked; I am not familiar enough with them to say why. I would look to the same 3 culprits: (a) not enough data; (b) competition on different stages (regional v national); and (c) lures (perhaps Brockport uses scholarships).
You mention that these rankings will be swayed by factors such as proximity to home. On average, these factors will cancel out each other. Some Texans might choose UT Austin over UC Berkeley due to proximity; some Californians might choose UC Berkeley over UT Austin for the same reason; etc. These will lead to some slight reduction in Elo points for the higher ranked school, but otherwise should have a minimal effect.
Finally, you talk about the possibility of a better school losing to a worse one. Can you explain how one should decide what the “better school” is? In my view, the better school for Student A might be the one with the best overall academics, regardless of cost or location. The best for Student B might be one with a great education department in an underserved community in the Midwest. The best school for you may not be the best for me. Our ranking system incorporates all of that data in aggregate. When appropriately paired to the source data, the tool can also peer into the preferences of particular subsets of students to get more personalized rankings. (That tool is currently unpublished.)
The ratings are still skewed. It seems the top 10 are in an argueably correct order, and then the system falls apart. I know of no other ranking that puts Notre Dame ahead of Cornell, Brown and a half a dozen others.
Are these raings by student preferences, as opposed to academic standards? That could explan it.
Correct; this is an revealed preference ranking. What “revealed preference” means is that we look at where students got in as the denominator, and where they decided to attend as the numerator. In theory, because college is such an expensive decision, students could be expected to make a rather thoughtful decision about where to attend after they find out where they get in, which is why this approach could be considered one of many reasonable ways to rank colleges.
The only other group to have done a revealed preference study – a set of professors from Yale, Harvard, and Stanford – also found Notre Dame rising surprisingly high above where they expected to find it in the rankings. One explanation offered for this is that Notre Dame recruits a particularly dedicated applicant pool that is willing to choose them over what are otherwise more notable institutions. Another explanation could well be that Notre Dame is undervalued as an institution.
Also, consider the different competition that Cornell and Notre Dame face: Cornell competes with the rest of the Ivies all day long (and loses about 80% of the time according to our data).
My theory on this situation is as follows (and some day I’ll look deeply into the data to see if this is actually the case): Presumably, many people want to get into an Ivy; they’re the same people who are aware the Cornell lacks the cachet of Yale or Harvard, so they’re less likely to attend Cornell if they get into a school with a “better name”. Notre Dame applicants are much less likely to be in the “I want to get into any Ivy” group of people. Notre Dame is more likely to be the big name school for them.