Simpson's Paradox

[badass]

http://en.wikipedia.org/wiki/Simpson's_paradox
(Look particularly at the kidney stone treatment example)

I think MyChances needs a new system for determining the accuracy of its assessments regarding chances of being accepted to colleges.
First, I'll discuss how this is an example of Simpson's Paradox. If we separate accuracy based on whether one is predicted to be accepted or rejected, we find for Princeton University (proclaimed 78% accuracy):
15/22 (68%) of those predicted to be accepted were truly accepted.
85/106 (80%) of those predicted to be rejected were truly rejected.
Meanwhile, for a slightly less competitive college such as Cornell University (proclaimed 73% accuracy) where more people are predicted to be accepted:
92/133 (69%) of those predicted to be accepted were truly accepted.
50/59 (85%) of those predicted to be rejected were truly rejected.

This table sums it up:

Predict	Accept	Reject	Total
Princeton	15/22 (68%)	85/106 (80%)	100/128 (78%)
Cornell	92/133 (69%)	50/59 (85%)	141/192 (73%)

Obviously Cornell is more accurately predicted by the model, but the statistics don't reflect this.

Considering that the model predicts that I'll be accepted to both schools, I know that the prediction will be considerably less accurate than 78% or 73%.

[badass]

Now, another problem with the prediction model is that it has a hard cutoff of 50% between reject and accept. Now this is disadvantageous for calculating accuracy. Whilst someone with a 100% prediction being rejected is surely inaccurate, someone with a 47% chance prediction being accepted is quite likely according to the model - 47% chance. Although it was more likely that the person be rejected, his acceptance is not unlikely at all and does not prove the prediction model wrong.

Now, let's say I'm given a 54% chance of admission to Princeton. If the prediction model is correct, I have a 54% chance of admission and a 46% chance of rejection. I want to see how accurate the model is for people who are given similar chances: I don't care much about people who were given 0% or 100% chances by the model. Let's say we create a range of predictions that we'll use to create our accuracy rating.
54% ± 10% : (44%,64%)
Now, if our prediction model is correct and there is a symmetrical distribution of predictions, 54% of the people in this range should be admitted. However, it turns out our range of 44% to 64% is heavily skewed to the lower numbers; I'd say the mean prediction in this range was about 47%, so we'd predict that 47% would be admitted. Let's find out what proportion were actually admitted:
21/73=29%
That's MUCH lower than predicted: the model is not very accurate. If you calculate the % difference between predicted and actual results:
(47-29)/29=62%
100%-62%=38% accuracy
NOTE: this accuracy rating has no statistical or scientific foundation. It's just a number that is opposite of percent difference, but it can provide a "rough" accuracy rating

[badass]

Now I'm going to try to define a method to calculate the accuracy of the prediction model for the entire set of applicants to a college.

We expect that (mean prediction of admission chances) * (number of applicants) will be the number of applicants who are accepted to the college. We'll now make a sampling distribution of means, assuming that such distribution will be roughly Student's T, to assess the accuracy.

Mean = mean of all the predictions of admissions chances
standard error = (standard deviation of all the predictions of admissions chances)/sqrt(number of applicants)

For example, using Princeton University:
mean=49.76%
standard error=14.62%/sqrt(128)=1.292%
However, the actual proportion of students accepted (of our sample) is 36/128=28.1%: much lower. What's the chance that our model is good although there is such a disparity?
(this gets statistical: http://en.wikipedia.org/wiki/Student's_t-test)
df=128-1=127
t=(0.281-0.4976)/0.01292=-16.8
This t-value gives us a P-value of 0, meaning that if the model was absolutely correct in assessing chances, there's no way we would possibly get the results we did (or more extreme).

Disclaimer: this stuff is weird for me statistically, and as a novice statistician, I am sure that what I am doing is not proper, but it may still be somewhat useful.

NOTE: This model is also flawed for another reason because it is based on the bulk numbers. Not that this is likely, but perhaps the prediction was completely off: people with 0% were accepted but with 100% weren't. The predicted proportion of acceptances could be similar as a total.

I'm going to look into using random variables next to see if that would be more useful.

[James]

How do you feel about Receive Operator Characteristic? At first blush, that's where I'd like to head in terms of giving a fair assessment of the predictive value of these models, but if you have another suggestion, please fire away.

Also, for purely abstract conceptualization of "47% chances vs 54% chances" or similar situations that you've brought up, convert them to odds and compare. Those two examples would be 1:1.9 odds and 1:2.2 odds (even if the models were completely accurate), a fairly slim difference regardless.

Regarding the arbitrariness with which 50% is used as a binary decision factor - it seems to be a slippery slope. If you decide not to call 50% the cutoff, where do you make the cutoff? 45% and 55%? Why not 40% and 60%? Your point certainly has merit, and one of the longstanding tasks at hand for us is to process the models using different cutoffs, including any and all that you and I have mentioned, to see if better accuracy could be had. I suspect that there are substantial improvements yet to be made, and we're on track to begin making these improvements in about a month.

If you'd like to keep talking here, please do. Otherwise, if you want to get in touch by email, I'll reply directly if you use the contact form.

Regards, James

[James]

For example, using Princeton University:
mean=49.76%
standard error=14.62%/sqrt(128)=1.292%
However, the actual proportion of students accepted (of our sample) is 36/128=28.1%: much lower.
...
df=128-1=127
t=(0.281-0.4976)/0.01292=-16.8

I fixed a formula with which I had been incorrectly calculating odds ratios.

Currently at Princeton:
mean = 21.06%
standard error = 1.97%
actual proportion accepted: 27.4%.

This is substantially more accurate than it was previously.