Wikipedia has an article on a Javascript conformance test called sputnik from Google.
It has the results from various Desktop browsers on how standards-compliant they are. Now I know what you are thinking, probably #1 is Chrome because the test is from Google, followed by Opera, Safari, and Firefox; with IE trailing behind in a pitiful display of brokenness.
But that's not what Wikipedia shows. In fact, it shows IE as the most standards-compliant browser.
Microsoft has quite brilliant engineers and recently they've began to take their browser seriously again, but to say that Internet Explorer has the most standards compliant JS engine? This would be quite a departure from history.
I needed to verify these results, for my own sanity.
According to Wikipedia, the test has changed hands and is now hosted at test262.ecmascript.org. You can go there now, it's on a domain registered by Mozilla Corporation. So this doesn't look like some wonky unfair comparison that almost looks to be designed to make Microsoft look good. Let's get questions of legitimacy out of the way.
Here's the updated (2011-11-09) results. Each browser's data is compressed with bzip2.
| Browser | Fails | The photo | The data |
|---|---|---|---|
| IE 10 10.0.8102.0 | 6 | Screenshot | XML Results |
| Firefox 8.0 | 164 | Screenshot | XML Results |
| Firefox 9.0 (2011-11-08) | 164 | Screenshot | XML Results |
| IE 9 9.0.8112.16421 | 322 | Screenshot | XML Results |
| Chrome 16.0.912.32 beta-m | 415 | Screenshot | XML Results |
| Chrome 15.0.874.106 m | 416 | Screenshot | XML Results |
| Opera 11.52 | 3750 | Screenshot | XML Results |
| IE 8 8.0.6001.18702 | N/A[1] | Screenshot | N/A |
IE 10 did the best by far. Surprised? Me too. The next up was Firefox. Also surprised? Yeah, here too. Then IE 9 ... IE 9? The world is crazy sometimes. Chrome comes in right above Opera, which basically eats shit ... that's a new one. What has this world come to?
One of the important things to note is that failures seem to happen in batches. That is to say it's successful for a few hundred (or a few thousand) tests, then a bunch of tests fail consecutively. Almost makes me think of that math exam you once got where questions 2-8 depended on you getting question 1 right.
These tests also don't speak to how performant the browsers are on, well, anything. Their are more factors than compliance-to-standards when it comes to providing the best web experience. Regardless, these results certainly support the on-going notion that IE is finally back in the running.
Further Work
I just looked at the number of failed tests; not focusing on the content of what tests failed. I'm not familiar enough (and just plain not smart enough) with the internals of ECMAScript to know whether the results warrant further investigation or concern. I do know from my own experience that I've fallen victim to many more IE Javascript bugs then other browsers. And perhaps, that's the important thing; not whether a feature is implemented, or passes a smoke screen test, but whether the feature has problems. It would be great to hear what some of the browser developers have to say about these results.about:
Tests were done on Windows 8, XP, and Vista; to try to main consistency. If you want to contact the author, reply here below. The author is Chris McKenzie; a programmer dedicated to truth, no matter how crazy it gets. Check out his projects on github.
Opera 11.60 Beta (final release probably will be within few days) "pass" the test with "only" 11 fails. Opera 12 probably will be even better.
ReplyDeleteOpera 11.60 Beta results: http://i.imgur.com/b69Lx.png
ReplyDeleteWonderful! Do you have the xml too, just to be formal?
ReplyDeleteAlso if you could put the build information in the screenshot like I do with mine, that would be perfect. Otherwise, I'll do it by saturday
ReplyDeleteFirefox 10 nightlu also fails 164 tests.
ReplyDeleteOpera 11.60 fails 11 tests — 10 of these are due to UA sniffing in the test harness (sth.js — modify that so Opera goes down the same code-path as everything else and you have a single failure).
ReplyDelete@gsnedders: Wow. If you think it's a bug, let's try to find the authors of that test and see if we can get them to address it. That's really good!
ReplyDelete