MyChances.net

On Sunday, Brent rolled out the first iteration of our new interface (for example, see how the college rankings now look). We were excited about the functionality, but subjectively the site seemed to load slowly for me, so I wanted to investigate. Specifically, I focused on browser ‘paint’ time, ignoring server-side processing delays (since the page gen time was a negligible 50 milliseconds).

Browser ‘painting’ is what happens after the page markup has been generated by the server and sent to the browser; based on the instructions within the markup, the browser ‘paints’ the canvas with items such as text and images. If javascript is on the page, the browser will execute the javascript and, if necessary, it will paint the canvas again according to the  javascript’s instructions. During the painting process, the page may subjectively feel sluggish or laggy (especially if the painting takes over 100ms, a commonly-accepted duration above which users tend to notice delays). At the end of all of this, the the canvas is totally painted and the user can interact with the page.

To troubleshoot our slow page painting, I made use of Google’s fantastic Chrome plugin ‘Speed Tracer’ . To conduct all of the following tests, I copied the HTML from a page on our site and loaded it locally to remove that variable from the equation. All other resources, such as ads, were loaded remotely. I pre-warmed any browser-cacheable objects by loading the page once prior to making any recordings.

Test 1: a normal page

First, I loaded a page as-is. The page is a user’s profile with four ads:  two Adsense ads (a skyscraper in the left nav, and a body ad in the middle of the page) and two custom ads that were remotely-loaded images (header and footer). How did it look?

(Click to load the full-size image.) Shown is the Speed Tracer screenshot. There are several ~500ms paint events. The page takes 3.5 seconds to finish rendering – that’s pretty painful. But which components are causing the slowdown — Adsense, our custom javascript, or something else? Let’s try to tease this out.

Test 2: a normal page with only 1 of 2 Adsense ads enabled

For this test, I removed the one Adsense ad from the page. The result? No change in overall speed (still ~3.5 seconds), although you can see that one brief rendering event is now gone.  Next, let’s get rid of all Adsense ads.

Test 3: a normal page without any Adsense ads

For this test, I removed both Adsense ads. The result? A huge change in rendering time! We dropped from 3.5 seconds down to 1 second. Instead of five 500ms paint events, we now only have one. This has a tremendous subjective benefit. Will removing the two non-Adsense ads improve our rendering time even more?

Test 4: a normal page with no ads

There is really no change here from the last test (still 1 second to render the whole page). The remotely-loaded, non-Adsense, image-only ads were not impacting our page rendering time. Can we fulfill Koch’s postulates by ‘infecting’ a plain-vanilla HTML page that has under 500 bytes of text with Adsense, reproducing the slowdown?

Test 5: a blank HTML canvas with one Adsense ad

Ouch. This is basically a blank page with some college-oriented words to trigger the right type of Adsense ad… and it takes 2.5 seconds to render.

Conclusion

We have a page that takes 3.5 seconds to render. 1 second of that time appears to be intrinsic to the page itself. 2.5 seconds of that time appears to be due to Adsense ads. Additional Adsense ads beyond the first one do not meaningfully worsen page rendering time.

Despite what strikes me as clear evidence (on one machine) of impressively slow rendering due to Adsense ads, I’m open to the possibility that there is another explanation and would love to hear feedback from people. Since Google has stated that they will be placing non-zero emphasis on page load time in the future, this feature of Adsense is somewhat concerning.

Notes

• Test rig: Intel Core i7 720, 8GB RAM, Win 7. Chrome version 6.0.458.1 dev. Speed Tracer version 0.17. Internet courtesy of Boston’s Cambridge Street Starbucks (1mbit download according to speedtest.net.)
• Adsense ads were always Flash (by chance). One could run another test with text-only ads, but image ads (and flash ads, which are conflated in the Adsense settings with image ads) seem to be the ‘standard’.
• HTML was loaded from local machine for all tests. All stylesheets and javascript were loaded from remote servers.

Quick update: each college now has its own threaded discussion board so you can comment – and easily keep track of who you’re talking to. The ‘discussion’ tab on each college page no longer links to old blog posts, but instead welcomes you into a running conversation with your peers. To see how this works, try joining the Harvard University discussion.

As always, feedback is welcome.

Tonight I added a couple of features to the scatterplots (scatter charts / scattergrams) that I introduced in my last post. The two new variables that you can now access are: acceptance status and prediction.

Let’s say you wanted to know how accurate our prediction engine is within one particular range of predictions. For example, you want to see what really happens when we claim someone has around 80-100% chances. Set the axes to Prediction, add a small amount of jitter, and you can now get a sense of how many miscategorizations we’ve made in that region. Take a look at Boston College to see an example.

You can now graph our predicted probability of acceptance on scatterplots.

Clearly we do a pretty good job at Boston College. For one, you can instantly see that the blue (accepted) applicants cluster on the right side with high predictions, while the red (rejected) cluster on the left. Additionally, look at the mean (average) lines. The mean prediction for accepted members was 83.3%, while that for rejected members was 35.6% – a very large difference.

Another way to visualize the data would be to set one axis to Accepted, and another to Prediction, so you get perfect separation between the blue (accepted) and red (rejected), and can perhaps more easily see how well our predictions separate the accepted from the rejected at any particular range of chances.

These updates were requested by Christian Romero; thanks, Christian.