MyChances.net

Yesterday I updated the algorithm that spits out the probability (your chances of admission) for each person at each school. These new probabilities should be substantially more predictive than the old ones. (This change will not move <50% predictions above 50%, or vice versa. What has changed is only the scale; some rankings that were previously 56% might now be 75%, wholly dependent on which school we’re talking about.)

Previously, I had incorrectly converted between odds and probability. Now, I have changed the algorithm to correctly convert between the two. (Why this matters is idiosyncratic to how we process the data; suffice it to say that this matters.) In looking over the numbers at a couple of schools, this seems to have substantially increased the reliability of the predictions. It’s still not perfect, but it’s better.

To learn more about this, you can read some posts by ‘badass’ about this previous problem. 

As ‘badass’ noted, a good model should, more or less, work like this:
Say that there are 25 people who each have a 20% chance of getting in. If the model isn’t great, then maybe 15 of those (60%) will get in, or maybe 1 of those (4%) will get in. If the model is great, then 1 in 5 (20%), or about 5 of those people, should get in. The models aren’t perfect, so this won’t be exactly the case, but now we’re much closer to that at many colleges. 

(As an aside, you can see how a ‘perfect’ model will necessarily get predictions ‘wrong’. A perfect model that gives 10 people a 60% chance should be “wrong” on 4 of them: 4 of them should not get in. One that gives 10 people a 90% chance should be “wrong” on 1 of them. For the model-maker, these are the desired results.)