The Browser Wars II

No more than four months ago, Opera released its version 10.50 of Opera desktop browser with a claim of being the fastest web browser on Earth. We, here at ThinkDigit, quickly decided to put that claim to test. The game became interesting with a tug-of-war between Google Chrome and Opera, and it was tough to determine a clear winner. If you’ve missed that round of The Browser Wars then, click here and enjoy the bloodshed!

Since then, a lot has changed, and the goal posts have been moved. This time the situation is a little different. We are watching Microsoft roaring with Internet Explorer 9, Mozilla reforming the Firefox browser, consecutive sixes from Opera, and a steady pace set by the already blazing Google Chrome – it’s really too much of a thrill to miss. Everyone is desperate to change the colour of the web with their own hue, taking byte size chunks off the browser pie.

With all the browsers delivering most of the needed features, the need for speed is taking a backseat — not that speed isn’t important but that almost all the browsers are fast enough. What we are expecting more out of a browser is the support for the new-age web standards, and less complicated browsing experience while browsing sites which have HTML5, CSS3 features within. There is no doubt that the craze of HTML5 is accelerating, and so it is for CSS3; JavaScript and next-gen DOM handling are also the mighty measures. But which browser is going to provide the most support for the upcoming web technologies and standards, and that too before all others? That is the billion dollar question, and the battle to conquer that throne has already begun. No one is even waiting for the HTML5 spec to finish!

Hence, while the browser war is steaming once again, we have geared up to determine the King of the web standards, and ability is the prime concern this time, not the speed alone.

We have selected a few tests, which primarily focus on HTML5, CSS3 and JavaScript supports and abilities, a few tests are taken from our previous attempt which meet the prerequisites. There will be Acid3 tests, Dromaeo tests and PeaceKeeper benchmarking suite as we had before, along with a few new tests i.e. CSS3, HTML5 tests and four selected test demos from IE9 testdrive site. These tests will be used for benchmarking, now let’s take a look at the browsers we will be testing out.

The Titans and Olympians of the battleground are as follows:

Same as before, our test rig is a regular laptop which is a very able one for web surfing, but no better than that. It uses an Intel Core Duo T2300 processor hosted on a compatible 945G chipset mainboard; it has 1x2GB DDR2 667MHz RAM and Intel GMA950 as graphics processor. We used Windows 7 as the operating system for the test rig. We intentionally didn’t include a high-end rig, to match the system specification with most of the cyber-dwellers.

 

Acid 3 Test Results:

Acid3 test is the latest addition to the Acid-Test suite. Since it was released in 2008, it has become the most popular JavaScript-DOM test out there. Not only for being fast and less resource demanding, but also because of giving a straight forward score out of hundred. Well, there may be debates about its fidelity and excellence but Acid3 test’s acceptance is wide-spread — no doubt on that. So, let’s take a look at the output we got from the tests we have run.

Yeah, we know – the figures are pretty same and most of us can recite them like a poem at this point. Apart from the final figures, we have added some of the little details about the tests performed, to shed more light on the performance (and to allow for a photo-finish possibility). Though you may already know, but in case you don’t — you can obtain those details by clicking on the ‘A’ of the ‘Acid3’ after the test is over.

As always, Opera, Safari and Chrome were the ones to get the perfect score (exactly identical score, even the warning counts, as you can see!) and IE8 continues holding its ground at a meagre 20! But apart from that, what catches our eyes is that Internet Explorer 9 has improved a lot. When we tried it last time, it scored 55, but this time, the platform preview 3 brings the best out of what IE has ever offered. It proves how serious Microsoft is about the browser market share. After the Windows 7 browser ballot phenomenon, Microsoft is giving one last fight to recover lost ground.

The scores for Firefox iterations are interesting! We already know that it’s not that good at Acid3 tests, and the scorecard of the Stable build shows us no hope, but the latest enhancements to the core of the browser is reflected through the result sheets of the Beta and the Nightly builds. Having 97 points each, they are not only close to perfection, but they generate less warnings than all other browsers. We also found that the Firefox builds were the fastest to complete the test. 

 

Dromaeo Test Suite Results:

These tests were also included in our hunt for the fastest browser – so, we already know much about them. But, for the new readers; a small recap: SunSpider, V8 and Dromaeo are the web browser JavaScript rendering test suites, from Apple, Google and Mozilla respectively. These tests are not concerned a bit about a browser’s rendering capability, but only the EcmaScript handling ability (which they test quite thoroughly).

As before, we chose Mozilla’s Dromaeo site which gives us all the tests under one roof, and the best part is that after running a successful test, we get a link that saves the test results with its finest details. Those URLs are provided below, linked with the total score.

First thing first, among all the browsers we have tested Internet Explorer 8 is the only which failed disqualified under this test. Not only is IE8 unable to perform some tasks and end the benchmark partially but it also couldn’t succeed to save the result for us to give you the links. So we took IE8 aside in these particular test sets.

As with their version increments, we can get a sense of the progress made by all these browsers; from the last time. Safari is the one with the least variation from its previous test result. It only managed to get a good margin in (its own) SunSpider test suite, but the other results are pretty much same as before. From version 10.50 to 10.60, the Carakan is furthermore polished to show a stable performance upgrade. The drastic drop of Dromaeo test result may be confusing, but we are sure now there was some problem in the test suite before (which gave Opera a score of nearly ten times than any other browser). The 10.70 Beta build is also a significantly high performer, to set our hope high for the final version to release.

Now, we will talk about a legend in the making. With a definite and undoubtedly high margin, Chrome checkmates Opera with the current releases. The previous results were neck-to-neck, but this time Google makes sure to dwarf ’em all — no matter whether it’s the Stable build or Dev — when it’s about JavaScript speed. Leading way ahead of all the browsers, Chrome is now giving itself competition! The performance difference between the Stable and the Dev build tells the story; while the Chrome v5 Stable build is admirable, the Chrome v6 Dev is awe-inspiring: 10-15% faster than that. Surely there is a NOS bottle hidden somewhere under the hood that has give Chrome’s V8 engine this extra boost!

We didn’t forget about Firefox, neither are we trying to hide its shame. Internet Explorer 9 manages to score a considerable figure, none of the Firefox builds show any hope even to touch it by any time soon. Even though the Beta and the Nightly builds show better performers than the Stable build, none of them are any match for rest of the lot. None of us have any clue exactly how much powerful JaegerMonkey is going to be, but Sorry Mozilla – until then, TraceMonkey is taking you nowhere.

 

FutureMark PeaceKeeper Test Result:

The PeaceKeeper test – although basically a browser speed test — also tests them for extensive JavaScript and DOM calculation payloads, and HTML5 canvas elements to render, along with a bunch of web 2.0 operations. There may be questions about the completeness of the test suite but PeaceKeeper, more or less, covers all we are trying to evaluate our browsers for.

When PeaceKeeper is run, not only do we get a URL, but the result is represented with a nice graphical bar chart, that can also be a comparison benchmark when used against a collective group of browsers. We have used this feature, and from here you can further explore the results online with further details. But for the article here, we will show you the snapshot of the score summery; let’s take a look —

No wonder Internet Explorer 8 came last, it is barely able to run half the tasks efficiently. But what amazed us is that, even after so much of build up and hype, IE9 has been able to defeat only its weaker sibling, and none other. It is true that IE9 is at only preview stage of development, but shall we see a dramatic improvement by the time it hits the market? Let’s hope for the best.

Firefox builds are the next ones in the line as we go up from the abysmal IE performances. The Firefox Beta and the Minefield builds can be considered to be the mediocre ones along with the Safari, which also has a comparable score, where as the Firefox Stable hangs with the IE brethren, with less than 2000 points.

Moving on to the next level, in 3000-4000 range, there are three browsers that forms the group – the stable build of Chrome, and the Opera siblings on top. But the only browser that could manage to go beyond the 4000 mark, is the Chrome Dev: the toughest player even in this trial.

 

CSS3 Capability Test Results:

CSS3 selectors test does exactly what it says: testing the CSS3 selectors; it calculates the results based on the browser’s success rate of handling the identified selectors from the test page simulations or a failure to do so.

Some selectors, such as :hover, :active, :focus, :selection etc. are kept out of this test, as they require active user interaction, and thus are not possible to judge with a simulated test environment. The test has been developed by Niels Leenheer, and was the winner of CSS3.info summer contest 2006 when it first released.

You may ask, why we did not include the CSS selectors test of the Dromaeo suite?

The first reason: the Dromaeo CSS Selectors test is a general CSS selector’s test and NOT a specialized test for CSS3, which is what we are focusing on here. Secondly, it tests for speed rather than compatibility. Thirdly, that test is too extensive and lengthy for a benchmark that mostly tests legacy CSS selectors.

Try the CSS3 selector test on your browser; go to http://tools.css3.info/selectors-test/test.html. After the test is over, click on a particular selector to see the details of the tests performed. The details pages shows the browser’s compatibility with each and every property that a selector may have. Being under development, the CSS3 standard is not a complete standard yet. But the test is always updated by the developer to match with the current W3C specifications.

Well, as is apparent, there’s nothing interesting here. All the browsers managed to score full marks. All but one: Internet Explorer 8. We have bashed it a lot already, so we’ll leave it for now – it did its best.

The support for all the selectors by all current browsers is indicative: that no matter how fast or slow they may be, the users will not see incorrect CSS rendering for using any of them. That also proves the intense hype for CSS3 even on the development side; a specification which is not even completely ready…

 

HTML5 Capability Test Results:

This test also comes from the same author, who designed the CSS3 test — Niels Leenheer. It can be found at http://html5test.com/. It analyses the browser’s HTML5 feature set, such as HTML5 canvas, audio, video, geo-location, WebGL, etc.

Even though it’s not flawless, this test is the only dependable one out there for HTML5 benchmarking. The author reports, that the suite has its limitations detecting supported codecs, or we may say the algorithm is not very strong when it comes to codec-MIME type analysis. The IE9 platform preview is supposed to score positive for supporting MPEG-4 and H.264 codecs, but the result comes negative. It’s buggy there no doubt – but it works fine analysing the other HTML5 aspects.

Initially the test was out of 160, but, as the HTML5 standard is getting richer day by day, and new specifications are being added, the test has also been updated. Thus, along with the add on tests, the maximum points a browser can score have reached 300. And bonus points are given to those which supports advanced features.

For a little deeper inquiry about each of the tests, you can follow the hyperlinks which leads you to a details page. There you will find details on that particular feature, as well as the status of the browser’s support.

Here we have the score-card:

The promises and claims about Internet Explorer 9 seem to be over-rated once again. Although it has beaten the score of IE8 (which should in turn be praised to be able to complete the HTML5 test at all); IE9 has a a long way to go before it can sit on the same table with the others.

Safari is the one which did extremely well in this particular test, even though it came second after Chrome Dev, but the time is not long past when it used to be the leading browser in this test. Now, with the Chrome builds catching up, and finally taking over, its glory has gone. And Chrome Dev did not only perform well but also scored more bonus points than all the other browsers.

Opera, on the other hand, performed moderately; while moving up in the graph gradually, the gradient is not too steep. Amazingly, both the Opera builds’ scores were the same, which indicates that not too much of the HTML5 feature set will be supported by this browser in the next release.

The performance of the Stable build may be shameful, but when it is about HTML5, lots of things are happening within the Firefox’s incubator. The latest beta and the Nightly builds did a great job, securing 189 points along with 9 bonus points which only makes it trail behind Chrome and Safari, but not by a large margin.

With deeper analysis, we find that while Firefox supported all other features of the HTML5 set well, it failed drastically in the ‘Forms’ elements test (it even failed the form validation). This was perhaps the only reason why Firefox lags Chrome and Safari, even though they are pretty weak at HTML document parsing, which Firefox does well.

 

Internet Explorer 9 TestDrive Results:

The IE9 platform preview’s test drive page consists of some technically useful and visually stunning HTML5 tests. We considered them to be a very nice addition to the test suite.

To let the browsers go through all the tests would have been a little too exhaustive for us, rather than the browsers, and for you too… So we decided to pick a select few from the tests available. The final picks were: the FishIE Tank, Asteroid Belt, Canvas Zoom (DeepZoom) and the IE Logo tests. Before we check the results, it will be better to know a little about the tests individually.

The FishIE Tank:

This one is an aquarium simulation with HTML5 canvas elements. In front of a static background, animated fishes are generated from a static image-strip sprite in the canvas; basic collision logic is also taken care of. The number of fishes in the tank determines the performance of the browser, as increasing the number slows down the rendering.

We found that 100 fishes gave the browsers a stable workload. The realtime frame rate was shown at the time of testing; we noted that value over certain period of time to take the average.

Asteroid Belt:

Asteroid Belt is also pretty much like the FishIE Tank, but it’s more of a web app than a simulator. The motion of the main asteroid depends on the user interaction, making it potentially equivalent of an online space arcade game (without being that much of fun).

The processing load depends upon the satellite orbiting the asteroid: we found the default value [4] to be just fine. This one also gives the result in frames per sec. (fps) that we have recorded for the benchmarking.

Canvas Zoom:

Among all the test demos, the Canvas Zoom uses most of all the aspects that we required to be tested. It uses HTML5 and CSS3 for the canvas element, scaling and smooth transition as well as AJAX for real-time updates of page content.

Representing an online map application (Google Maps, Wikimapia etc.), Canvas Zoom uses a wide angle image which can be magnified to get the finest detail on the image. Although only the fps value is shown for the benchmarking, we also noted how smoothly a browser was able to zoom in and zoom out of the image.

IE Logo:

The IE Logo test is for some special CSS3 features. It basically uses the CSS3 border-radius property to form the IE logo, and JavaScript, to create the animation from rectangle div elements. Cool, eh?

This test is not very resource heavy but when ran across different browsers, we came to understand that providing support and providing quality are not the very same thing. We rated the browser’s performance out of 5, according to the quality of the IE Logo generated, where 5 being the best (exact) and 0 being the worst (fail) possible rendering of the logo.

It may be that all the browsers did a very good job in our CSS3 test, giving support (recognizing) to all the CSS3 properties those were fed, but the test did not indicate how good the ‘support’ will get rendered on the screen. Well, this test shed some light onto that aspect:

 

We will start with two notes: Including IE8 in this test was a mistake as it is too primitive to handle HTML5. Also remember that as Microsoft is hosting these test demos, there might be a bias towards IE9.

In FishIE Tank, IE9 scores a 25fps, but among the other browsers we can see Opera top with 6fps and Firefox trailing just. This test made the Webkit browsers bleed; even though Chrome still managed to render 2-3 frames per second, Safari was completely stuck producing one frame after a few seconds.

The Asteroid Belt once again has IE9 in the lead, along with Opera and Firefox (not the stable build) taking the second place with a some-what ‘playable’ frame rate. The WebKit browsers stuttered once again, while Firefox stable’s performance was intermediate.

We were amazed to find that Opera was the second browser which could not properly run a test, beside IE8. In the Canvas Zoom test, the Opera browsers totally failed to render the main zooming canvas element. Whereas IE9 was soaring at more than 35 fps with Firefox trailing; the performance of both of the Safari and Chrome Dev was not that good. Here we should also mention one thing: even with a higher frame rate, Firefox’s transitions were not that smooth compared to Safari and Chrome (i.e. when the page update through AJAX is occurring, Firefox waits for the zoomed image to be loaded, but WebKit zooms in the previous low-res image, until the actual higher resolution image has been loaded).

The last test was, as we said before, a experiment on the quality vs. quantity issue. We have seen Firefox doing drastically bad in JavaScript ‘speed’ tests, and it did equally good to support the CSS3 properties. But when we allowed all the browsers to render the IE Logo from the rectangles, only Firefox (and IE9) could manage to do exactly what was expected out of them. Opera, Safari may be fast, and Chrome faster still at JavaScript rendering, but what is the purpose of just being fast, at the cost of quality of the output?

 

The Verdict:

Three standards, ten browsers, eleven benchmark suites and a plethora of tests later… It was a heck of a war indeed! We learned a lot about the core of every major browser, and the field(s) of expertise for each of them. Among all the tests we our browsers have been through, the IE demo tests seem a little biased. The scoresheet that they produced is eager to prove Internet Explorer 9 as the mightiest among the browsers. We will check if these tests were ‘specially’ designed for IE9 or not. Until then, we will not take the data too seriously for our inference.

We can see how serious Microsoft is to use IE9 to recover the gradually decreasing usage share of their browser. These tests and the ones conducted months before show how ancient the current Internet Explorer core is. No wonder, organizations are using it as the easiest firewall till date, to limit the online activities of their employees. They don’t really have any particular field to polish IE9 for, as all the stings of the Trident need a complete revamp! So far, Microsoft has done a great job in making IE relevant again but they have a long way to go still…

Performance wise, the second-worse are Mozilla’s Gecko-based browsers. Surprisingly, they are next in the global browser market share too, after IE (and are having a hard time maintaining the captured percentage). In our fastest browser hunt, the Mozilla browsers bled badly and we speculated enhancement in near future. Although some progress is perceivable, it’s not much, leaving it behind many competitors. While it comes to quality and support for standards, however, the Firefox browsers have shown good improvement: recognizing and handling most of the HTML5 and CSS3 features, the newer builds also offer experimental features such as GPU accelerated WebGL rendering. As promised by Mozilla, we expect to see the Fox claim the web again; being faster than ever, while also providing the quality – that it already does.
Safari would be the browser, which is less talked about in a geeky environment, and mostly used only by Apple users. But as it seems, this WebKit browser has a lot of stamina within it. Safari proved itself a powerful tool against HTML5 (though we didn’t include them in this article, the latest nightly builds are scoring more than 230 in the HTML5 test – that’s higher than any other browser). It also scored honourable marks in the JavaScript tests; it smoothly passed the CSS3 trial, as well as the Acid3 test. Not supporting the free audio/video codec may prove being a major con, but the support for MPEG-4, h.264 and AAC cancels some pain out.

Our next competitor is Opera. It has been around for a long time, and is older than any other browser, apart from Internet Explorer. It has faced and witnessed many torments on the web. When first released, Opera was all about speed (ca-chow!), bringing down the fame of Internet Explorer and giving the Netscape a tough fight. A long time has passed, competitors have changed, Opera remained, (always as THE ‘supporting character’ in the play, crying out for the lead role) and is still being named when browsing speed is the topic of discussion. And of now — HTML5, CSS3, JavaScript and DOM — no matter what is thrown at this browser, Opera handles them all. The Carakan engine is serving it well, and with the version updates we are being introduced with a steadily more powerful Opera browser each time. The Presto layout engine is growing by mighty day-by-day, so much so that even the Opera Mobile browser is able to perform at most of these desktop browser tests pretty modestly (it scores 99/100 in the Acid3 tests, for example). In comparison with all the browsers, Opera holds its ground to be in a leading position; the only problem — that Google Chrome is rushing past and Opera has to be content being second best.

Google Chrome: with less than two years of age, we may call it an infant, but it’s by no means an amateur. Chrome reminds us a lot about the David and Goliath story, shooting down the big players one by one. Taking the browser usage share into account, it has already taken over Opera and Safari, and is breathing down Firefox’s neck – and for good reasons too. The V8 engine is getting breathtakingly fast with each iteration, and is beating any browser at any level, from almost every perspective. In HTML5 support context, Chrome supports the most number of audio/video codecs till date, as well as excelling in other features, such as geo-location, web application, canvas, etc. As for its support for CSS3 standard, Chrome does its best not to miss out any aspect of it; however questions can be raised from the perspective of render-quality. But as the standard itself is still shaping up, we can’t really blame Chrome just now; especially considering the pace at which it is moving: Google will probably wipe off the imperfections by the time CSS3 matures.

Having all the scores in hand we can finally draw the inference drawn. As preview builds should not be judged on par with the stable builds — they typically contain bugs and unstable experimental features, such is their nature — they will not be eligible to claim the crown. We will take the scores of the Beta, Dev and Nightly builds as only an indication for the future of these browsers. Taking the performance data of only the stable builds into consideration, it is clear as daylight that Google Chrome brings the best to the web. It has proved itself in one test after another — no matter what test was thrown at it, the output was at least the same as others, if not better. We declare Google Chrome the champion; let’s continue with the award ceremony, all rise for the web anthem…