Presenting: our new college rankings.

The college admissions landscape is littered with college rankings. In 1983, US News first ranked American universities. Since then, rankings have been a fixture of the college world: they are produced by various businesses (US News, Princeton Review, Forbes, Atlantic Monthly), and heeded by students and colleges alike. To gain advantage, some universities have been alleged to manipulate their own rankings. And, while some of the factors used in the rankings are justifiable (alumni giving rate), some seem to be arbitrary (peer assessment surveys asking other colleges about your college’s ‘faculty dedication to teaching’). Each year, the methodology changes slightly, producing a slightly different list. In the end, the factors that are used to come up with the rankings seem arbitrary; the occasional change in the weighting of each factor, capricious. There is a need for a new approach.

Criteria for a ‘good’ college ranking system

1. The system should be difficult to game; any ‘gaming’ of the system should actually benefit students. In contrast, consider the allegations that some schools tried to manipulate the US News rankings by encouraging more students to apply in order to decrease their acceptance rate.
2. The factors measured should be relevant to students. In contrast, what Cornell’s dean thinks about the faculty dedication at the University of Texas may be irrelevant.
3. The overall procedure for generating rankings should be stable from year to year. In other words, any change in the rankings between 2008 and 2009 should be explained by a substantive change in the underlying factors, not by an arbitrary change in how those factors are weighted.

The MyChances College Rankings

We have implemented the MyChances College Rankings based on revealed student preference. In this system, the college admissions process is treated like a chess tournament. The colleges play matches (which occur when 2 colleges admit the same student). In each match, there is a winner (the college that the student ends up attending) and a loser. The winner gains points; the loser forfeits them. When a high-ranked school beats a low ranked school, the high-ranked school gains few points, and the low-ranked school loses few points. If a low-ranked school beats a high-ranked opponent, it gains more points than if it beat an equally-matched opponent. After playing many games, the colleges that students prefer rise naturally to the top of the rankings.

Does the method of revealed student preference meet the 3 criteria outlined above? I believe it does.

Consider point #1 (gaming the system). Imagine that MIT wanted to beat out Harvard by trying hard to avoid admitting any students that they thought would be admitted to Harvard. They would end up succeeding in a model based on acceptance rate and yield (since their yield would likely increase), but their actual student body would be less qualified. In the revealed preference model, however, they would be less successful. They would not compete head-to-head with Harvard, so would ‘win’ more. But they would be winning against weaker ‘opponents’, earning fewer points for each victory.

For point #2 (relevance), the idea of revealed preference is that it aggregates the sum total of what matters to students – whatever those factors might be. It is likely that students behave rationally (by attending the school that they find most desirable). So long as other students share similar values, then revealed preference rankings will work well in explaining, and even guiding, their decisions.

For point #3 (stability), the tournament style system is simple and straightforward. It is responsive to changes in student preference over time. It does not rely on aggregations of various statistical factors, or college faculty survey results; nor does it depend upon arbitrary weighting of those factors.

The details of the procedure that we use to generate the rankings, and our use of chess-style Elo points, will be explained in a later post. For an academic treatment of a similar college ranking system, I recommend the working paper, “A Revealed Preference Ranking of U.S. Colleges and Universities,” 2005, by Christopher Avery, Mark Glickman, Caroline Hoxby, and Andrew Metrick (free link).

On each college page you’ll find a “Personalized Analysis” tab. This now shows you about a dozen graphs, breaking the numbers down by accepted/rejected/applying. Your own score range is highlighted. (Note: you have to be logged in to see the graphs.)

I’ve attached an example for those of you who aren’t logged in.

Yesterday I updated the algorithm that spits out the probability (your chances of admission) for each person at each school. These new probabilities should be substantially more predictive than the old ones. (This change will not move <50% predictions above 50%, or vice versa. What has changed is only the scale; some rankings that were previously 56% might now be 75%, wholly dependent on which school we’re talking about.)

Previously, I had incorrectly converted between odds and probability. Now, I have changed the algorithm to correctly convert between the two. (Why this matters is idiosyncratic to how we process the data; suffice it to say that this matters.) In looking over the numbers at a couple of schools, this seems to have substantially increased the reliability of the predictions. It’s still not perfect, but it’s better.

To learn more about this, you can read some posts by ‘badass’ about this previous problem. 

As ‘badass’ noted, a good model should, more or less, work like this:
Say that there are 25 people who each have a 20% chance of getting in. If the model isn’t great, then maybe 15 of those (60%) will get in, or maybe 1 of those (4%) will get in. If the model is great, then 1 in 5 (20%), or about 5 of those people, should get in. The models aren’t perfect, so this won’t be exactly the case, but now we’re much closer to that at many colleges. 

(As an aside, you can see how a ‘perfect’ model will necessarily get predictions ‘wrong’. A perfect model that gives 10 people a 60% chance should be “wrong” on 4 of them: 4 of them should not get in. One that gives 10 people a 90% chance should be “wrong” on 1 of them. For the model-maker, these are the desired results.)

Now when you visit a school’s page, you’ll be able to click through to “Personalized Analysis”, which shows where you fall vs all admitted students (ever) and current applicants (for the next admissions cycle). Currently it is limited to GPA and SAT scores, but we plan to expand this to automatically show all of the categories that apply to you; i.e., if you listed an AP English Language score, we’ll graph that out for you.

Today the site has reached an important milestone: there are now 10,000 registered members. Thanks for your interest in this site and for your contributions to one another.

Some communities become more distant and less useful as their numbers grow. I think our community is quite the opposite; as more people join and contribute, we’ll all gain better insight into the daunting process of college admissions. And I’m working on a few tools right now to help keep the community interconnected, even as the site grows larger.

Until soon.