2006 NYT College Preference Matchups

In 2006, New York Times columnist David Leonhardt wrote an article that was ostensibly about Harvard ending its early admissions program. The meat of the article, though, was far more interesting, filled with a discussion of college rankings*. A survey of 3,200 high school seniors from 500 high schools was performed in 1999. The students were asked about which schools admitted them, and which college they ended up attending. From this information, a model was created and used to calculate how often students would choose one college over another. See the NYT Chart for the results.

That data is now 10 years old, so over the past several months, MyChances developed tools to perform a similar analysis. For 2009, we surveyed 1,200 students at 950 high schools. How do our results stack up?

2009 MyChances College Preference Matchups

Let’s look at the same colleges as in the Leonhardt article, focusing on three examples: Harvard vs Stanford, Georgetown vs Brown,  and Duke vs Princeton.

73% of students would have chosen Harvard over Stanford in 1999. This is completely unchanged in 2009.

Among students admitted to Georgetown and Brown, only 22% were estimated to go to Georgetown in 1999. Now, our evidence suggests that Georgetown has pulled even with Brown, winning 53% of the matchups.

Finally, 91% of students admitted to Princeton and Duke previously chose Princeton. Now, we estimate that 35% would choose Duke – a far more narrow advantage for Princeton than before.

Though it is interesting to make some of the same comparisons as were made in the 2006 New York Times article, you’re not limited to those schools using our tools. We had enough data to rank over 200 colleges in 2009; hopefully we’ll be able to expand to 300 in 2010, ultimately providing estimates for all 1,700 colleges in our database in the near future.

* = Paper is “A Revealed Preference Ranking of U.S. Colleges and Universities” by Avery, Glickman, Hoxby, and Metrick

(Note: College Rankings #3, an exposition of our methods, has not yet been published.)

In my previous post, I introduced the new college rankings system that we have implemented. In short, the system ranks colleges based on where their admitted students decide to attend. In this post, I will discuss some of the approaches that might be considered in creating a preference-based ranking. In the next post, I will discuss the preference ranking system that we have implemented.

2009 MyChances.net College Rankings

Yield isn’t enough

The goal of a preference-based ranking system is to capture people’s true preferences and represent them faithfully. To discover people’s preferences, a reasonable place to start might be a college’s yield. Yield is calculated as follows:

yield = (# of students attending) / (# of students accepted)

So how can we use yield to compare two schools? Suppose we match the University of Georgia (#70 on our list) against Pomona (#50 on our list). In this matchup, Georgia’s 55% yield actually beats out Pomona’s 39% yield. More of Georgia’s admitted students end up attending—so Georgia appears to be preferable to Pomona. But there is a problem here: we have no direct evidence that students, given the opportunity to attend either school, would choose Georgia. We simply don’t know what the students who were admitted to both schools would do.

In the abstract, there is another problem with this approach. Imagine that 100 students apply to both Georgia and Pomona. Suppose Pomona accepts 50 of them but Georgia accepts all 100. Now, suppose the 50 rejected from Pomona all decide to go to Georgia, giving it a 50% yield. Suppose, also, that 40 of Pomona’s accepted students also get into Harvard, Yale, or Princeton, and they all go off to those schools. This leaves Pomona with a 20% yield. Going by yield, it appears that Georgia is the preferred college by far—but in reality, all of the students admitted to both Pomona and Georgia who attended one of the two decide to go to Pomona. Yield, in this situation, gives us exactly the wrong answer about which school is preferred over the other!

Each student matters

What can we learn from the failure of yield as a measure of preference? Summary statistics simply don’t tell us enough. We need to drill down to the level of individual students. Only then can we build up a picture of their collective preferences. How can we do this? One approach might simply be to ask them what their preferences are. For example, we could survey a bunch of students applying to college, and ask them to order all of the schools they are considering, from most favorite to least favorite.

This is better; if those 100 people in our previous example honestly represented their preferences, we would probably see that the Pomona was preferred over Georgia. This is the intuitively correct result given our (fake) example. But even this approach isn’t perfect.

Talk is cheap; opinions, cheaper

One problem is that there is no cost associated with ranking a school #1 on your own personal list. Until you actually have to decide which college you are going to attend—and pay tuition to—for the next 4 years, your opinions have no teeth. Let’s say you rank UNC as your #1 school and Duke as your #8 (out of 8), because your Tar Heel family hates those Blue Devils. You apply, and get into both schools. Did I mention that you got a merit scholarship to Duke? All of a sudden, you find yourself attending your supposedly bottom-ranked school. You didn’t lie when you gave us your rankings, but you probably exaggerated how much you preferred UNC over Duke. Furthermore, you didn’t have all of the information that you used to make your decision—such as your merit scholarship—when you reported that Duke was your #8 school.

In general, asking people for their preferences leads to these additional problems:

• They may give feedback about colleges where their feedback is of questionable value. If someone with a 1.5 GPA says that they rank State U over Harvard, should that hurt Harvard—even though this person almost certainly wouldn’t be given the opportunity to attend there, anyways?
• They almost certainly give feedback that is based on imperfect information. At the moment where people are making their decision to attend one school out of several that they were admitted to, they have acquired as much information as they think they need to make this huge decision. Beforehand—and, in particular, before they have applied to and been admitted to colleges—their stated preferences may be much more labile.

Understanding these flaws helps flesh out a framework for a powerful-yet-simple preference-based college rankings system: one where students simply report where they were admitted and where they decided to attend. In my next post, I’ll get into some of the details of how to take this information and construct a ranked preference list. I’ll even demonstrate how this approach addresses a common criticism of the currently popular college rankings: that there is no way to truly distinguish between schools closely ranked (e.g., #3 vis-a-vis #5).

Today we’re letting everyone in on a sneak-preview of our latest tool: the college cross-admit preference tool. We think it’s a simple but powerful way to see which colleges are most favored by admitted college students.

To use it is simple: type in the names of two colleges that you want to compare (perhaps Florida and Florida State?). You’ll then see which fraction of site members prefers which school. Preference is determined by the relative fraction of members admitted to both schools who end up attending one or the other. For example, if 25% of students admitted to both College A and College B ultimately go to College B, we say they prefer College B over College A. When the results are statistically significant at the 95% level, you’ll see the results lit up in bright colors.

For the hardcore college admissions followers out there, this will remind you of this graphic from a 2006 NY Times article. One difference is that our list isn’t limited to 17 schools; as the data continues to become available, we’ll display this information for all 1700 schools that we track.

Requests? Feedback? Suggestions? Let us know.

I’m happy to introduce College Comparisons to MyChances.net. We’ve always had a list of ‘Peer Schools’ – schools that share many applicants with School X (whichever school you’re browsing). We’ve gone beyond that now, though. When you visit School X’s page, you’ll be able to see:

* Which schools students prefer more (e.g., if students get into both School X and School Y, they commonly attend School Y)
* Which schools students consider equally (e.g., if students get into both School X and School Y, they sometimes go to one and sometimes to the other)
* Which schools students prefer less (e.g., if students get into both School X and School Y, they commonly attend School X)

Hopefully this will help you broaden you college search and discover some new schools that you weren’t previously aware of. As always, good luck with the application process!