No more than four months ago, Opera released its version 10.50 of Opera desktop browser with a claim of being the fastest web browser on Earth. We, here at ThinkDigit, quickly decided to put that claim to test. The game became interesting with a tug-of-war between Google Chrome and Opera, and it was tough to determine a clear winner. If you've missed that round of The Browser Wars then, click here and enjoy the bloodshed!
Since then, a lot has changed, and the goal posts have been moved. This time the situation is a little different. We are watching Microsoft roaring with Internet Explorer 9, Mozilla reforming the Firefox browser, consecutive sixes from Opera, and a steady pace set by the already blazing Google Chrome - it's really too much of a thrill to miss. Everyone is desperate to change the colour of the web with their own hue, taking byte size chunks off the browser pie.
Hence, while the browser war is steaming once again, we have geared up to determine the King of the web standards, and ability is the prime concern this time, not the speed alone.
The Titans and Olympians of the battleground are as follows:
But, all of those were bugfix releases for unstable dev and nightly versions,
would have little or no impact on browsers' performances.
Same as before, our test rig is a regular laptop which is a very able one for web surfing, but no better than that. It uses an Intel Core Duo T2300 processor hosted on a compatible 945G chipset mainboard; it has 1x2GB DDR2 667MHz RAM and Intel GMA950 as graphics processor. We used Windows 7 as the operating system for the test rig. We intentionally didn't include a high-end rig, to match the system specification with most of the cyber-dwellers.
Acid 3 Test Results:
Because, to summon the resources, the first attempt loses some time downloading the test contents.
Yeah, we know - the figures are pretty same and most of us can recite them like a poem at this point. Apart from the final figures, we have added some of the little details about the tests performed, to shed more light on the performance (and to allow for a photo-finish possibility). Though you may already know, but in case you don't -- you can obtain those details by clicking on the 'A' of the 'Acid3' after the test is over.
As always, Opera, Safari and Chrome were the ones to get the perfect score (exactly identical score, even the warning counts, as you can see!) and IE8 continues holding its ground at a meagre 20! But apart from that, what catches our eyes is that Internet Explorer 9 has improved a lot. When we tried it last time, it scored 55, but this time, the platform preview 3 brings the best out of what IE has ever offered. It proves how serious Microsoft is about the browser market share. After the Windows 7 browser ballot phenomenon, Microsoft is giving one last fight to recover lost ground.
The scores for Firefox iterations are interesting! We already know that it's not that good at Acid3 tests, and the scorecard of the Stable build shows us no hope, but the latest enhancements to the core of the browser is reflected through the result sheets of the Beta and the Nightly builds. Having 97 points each, they are not only close to perfection, but they generate less warnings than all other browsers. We also found that the Firefox builds were the fastest to complete the test.
Dromaeo Test Suite Results:
As before, we chose Mozilla's Dromaeo site which gives us all the tests under one roof, and the best part is that after running a successful test, we get a link that saves the test results with its finest details. Those URLs are provided below, linked with the total score.
First thing first, among all the browsers we have tested Internet Explorer 8 is the only which
failed disqualified under this test. Not only is IE8 unable to perform some tasks and end the benchmark partially but it also couldn't succeed to save the result for us to give you the links. So we took IE8 aside in these particular test sets.
As with their version increments, we can get a sense of the progress made by all these browsers; from the last time. Safari is the one with the least variation from its previous test result. It only managed to get a good margin in (its own) SunSpider test suite, but the other results are pretty much same as before. From version 10.50 to 10.60, the Carakan is furthermore polished to show a stable performance upgrade. The drastic drop of Dromaeo test result may be confusing, but we are sure now there was some problem in the test suite before (which gave Opera a score of nearly ten times than any other browser). The 10.70 Beta build is also a significantly high performer, to set our hope high for the final version to release.
We didn't forget about Firefox, neither are we trying to hide its shame. Internet Explorer 9 manages to score a considerable figure, none of the Firefox builds show any hope even to touch it by any time soon. Even though the Beta and the Nightly builds show better performers than the Stable build, none of them are any match for rest of the lot. None of us have any clue exactly how much powerful JaegerMonkey is going to be, but Sorry Mozilla - until then, TraceMonkey is taking you nowhere.
FutureMark PeaceKeeper Test Result:
When PeaceKeeper is run, not only do we get a URL, but the result is represented with a nice graphical bar chart, that can also be a comparison benchmark when used against a collective group of browsers. We have used this feature, and from here you can further explore the results online with further details. But for the article here, we will show you the snapshot of the score summery; let's take a look --
No wonder Internet Explorer 8 came last, it is barely able to run half the tasks efficiently. But what amazed us is that, even after so much of build up and hype, IE9 has been able to defeat only its weaker sibling, and none other. It is true that IE9 is at only preview stage of development, but shall we see a dramatic improvement by the time it hits the market? Let's hope for the best.
Firefox builds are the next ones in the line as we go up from the abysmal IE performances. The Firefox Beta and the Minefield builds can be considered to be the mediocre ones along with the Safari, which also has a comparable score, where as the Firefox Stable hangs with the IE brethren, with less than 2000 points.
Moving on to the next level, in 3000-4000 range, there are three browsers that forms the group - the stable build of Chrome, and the Opera siblings on top. But the only browser that could manage to go beyond the 4000 mark, is the Chrome Dev: the toughest player even in this trial.
CSS3 Capability Test Results:
CSS3 selectors test does exactly what it says: testing the CSS3 selectors; it calculates the results based on the browser's success rate of handling the identified selectors from the test page simulations or a failure to do so.
Some selectors, such as :hover, :active, :focus, :selection etc. are kept out of this test, as they require active user interaction, and thus are not possible to judge with a simulated test environment. The test has been developed by Niels Leenheer, and was the winner of CSS3.info summer contest 2006 when it first released.
You may ask, why we did not include the CSS selectors test of the Dromaeo suite?
The first reason: the Dromaeo CSS Selectors test is a general CSS selector's test and NOT a specialized test for CSS3, which is what we are focusing on here. Secondly, it tests for speed rather than compatibility. Thirdly, that test is too extensive and lengthy for a benchmark that mostly tests legacy CSS selectors.
Try the CSS3 selector test on your browser; go to https://tools.css3.info/selectors-test/test.html. After the test is over, click on a particular selector to see the details of the tests performed. The details pages shows the browser's compatibility with each and every property that a selector may have. Being under development, the CSS3 standard is not a complete standard yet. But the test is always updated by the developer to match with the current W3C specifications.
Well, as is apparent, there's nothing interesting here. All the browsers managed to score full marks. All but one: Internet Explorer 8. We have bashed it a lot already, so we'll leave it for now - it did its best.
The support for all the selectors by all current browsers is indicative: that no matter how fast or slow they may be, the users will not see incorrect CSS rendering for using any of them. That also proves the intense hype for CSS3 even on the development side; a specification which is not even completely ready...
HTML5 Capability Test Results:
This test also comes from the same author, who designed the CSS3 test -- Niels Leenheer. It can be found at https://html5test.com/. It analyses the browser's HTML5 feature set, such as HTML5 canvas, audio, video, geo-location, WebGL, etc.
Even though it's not flawless, this test is the only dependable one out there for HTML5 benchmarking. The author reports, that the suite has its limitations detecting supported codecs, or we may say the algorithm is not very strong when it comes to codec-MIME type analysis. The IE9 platform preview is supposed to score positive for supporting MPEG-4 and H.264 codecs, but the result comes negative. It's buggy there no doubt - but it works fine analysing the other HTML5 aspects.
Initially the test was out of 160, but, as the HTML5 standard is getting richer day by day, and new specifications are being added, the test has also been updated. Thus, along with the add on tests, the maximum points a browser can score have reached 300. And bonus points are given to those which supports advanced features.
For a little deeper inquiry about each of the tests, you can follow the hyperlinks which leads you to a details page. There you will find details on that particular feature, as well as the status of the browser's support.
Here we have the score-card:
The promises and claims about Internet Explorer 9 seem to be over-rated once again. Although it has beaten the score of IE8 (which should in turn be praised to be able to complete the HTML5 test at all); IE9 has a a long way to go before it can sit on the same table with the others.
Safari is the one which did extremely well in this particular test, even though it came second after Chrome Dev, but the time is not long past when it used to be the leading browser in this test. Now, with the Chrome builds catching up, and finally taking over, its glory has gone. And Chrome Dev did not only perform well but also scored more bonus points than all the other browsers.
Opera, on the other hand, performed moderately; while moving up in the graph gradually, the gradient is not too steep. Amazingly, both the Opera builds' scores were the same, which indicates that not too much of the HTML5 feature set will be supported by this browser in the next release.
The performance of the Stable build may be shameful, but when it is about HTML5, lots of things are happening within the Firefox's incubator. The latest beta and the Nightly builds did a great job, securing 189 points along with 9 bonus points which only makes it trail behind Chrome and Safari, but not by a large margin.
With deeper analysis, we find that while Firefox supported all other features of the HTML5 set well, it failed drastically in the 'Forms' elements test (it even failed the form validation). This was perhaps the only reason why Firefox lags Chrome and Safari, even though they are pretty weak at HTML document parsing, which Firefox does well.
Internet Explorer 9 TestDrive Results:
The IE9 platform preview's test drive page consists of some technically useful and visually stunning HTML5 tests. We considered them to be a very nice addition to the test suite.
To let the browsers go through all the tests would have been a little too exhaustive for us, rather than the browsers, and for you too... So we decided to pick a select few from the tests available. The final picks were: the FishIE Tank, Asteroid Belt, Canvas Zoom (DeepZoom) and the IE Logo tests. Before we check the results, it will be better to know a little about the tests individually.
This one is an aquarium simulation with HTML5 canvas elements. In front of a static background, animated fishes are generated from a static image-strip sprite in the canvas; basic collision logic is also taken care of. The number of fishes in the tank determines the performance of the browser, as increasing the number slows down the rendering.
We found that 100 fishes gave the browsers a stable workload. The realtime frame rate was shown at the time of testing; we noted that value over certain period of time to take the average.
Asteroid Belt is also pretty much like the FishIE Tank, but it's more of a web app than a simulator. The motion of the main asteroid depends on the user interaction, making it potentially equivalent of an online space arcade game (without being that much of fun).
The processing load depends upon the satellite orbiting the asteroid: we found the default value  to be just fine. This one also gives the result in frames per sec. (fps) that we have recorded for the benchmarking.
Among all the test demos, the Canvas Zoom uses most of all the aspects that we required to be tested. It uses HTML5 and CSS3 for the canvas element, scaling and smooth transition as well as AJAX for real-time updates of page content.
Representing an online map application (Google Maps, Wikimapia etc.), Canvas Zoom uses a wide angle image which can be magnified to get the finest detail on the image. Although only the fps value is shown for the benchmarking, we also noted how smoothly a browser was able to zoom in and zoom out of the image.
This test is not very resource heavy but when ran across different browsers, we came to understand that providing support and providing quality are not the very same thing. We rated the browser's performance out of 5, according to the quality of the IE Logo generated, where 5 being the best (exact) and 0 being the worst (fail) possible rendering of the logo.
It may be that all the browsers did a very good job in our CSS3 test, giving support (recognizing) to all the CSS3 properties those were fed, but the test did not indicate how good the 'support' will get rendered on the screen. Well, this test shed some light onto that aspect:
|FishIE Tank [fps]||X||25||0||5||6||3||2||3||4||4|
|Asteroid Belt [fps]||X||31||0||13||14||1||1||7||10||12|
|Canvas Zoom [fps]||X||35||10-15||X||X||7-15||11-14||8-15||12-20||12-20|
|IE Logo [quality/5]||1||5||2||3||3||2||2||5||5||5|
We will start with two notes: Including IE8 in this test was a mistake as it is too primitive to handle HTML5. Also remember that as Microsoft is hosting these test demos, there might be a bias towards IE9.
In FishIE Tank, IE9 scores a 25fps, but among the other browsers we can see Opera top with 6fps and Firefox trailing just. This test made the Webkit browsers bleed; even though Chrome still managed to render 2-3 frames per second, Safari was completely stuck producing one frame after a few seconds.
The Asteroid Belt once again has IE9 in the lead, along with Opera and Firefox (not the stable build) taking the second place with a some-what 'playable' frame rate. The WebKit browsers stuttered once again, while Firefox stable's performance was intermediate.
We were amazed to find that Opera was the second browser which could not properly run a test, beside IE8. In the Canvas Zoom test, the Opera browsers totally failed to render the main zooming canvas element. Whereas IE9 was soaring at more than 35 fps with Firefox trailing; the performance of both of the Safari and Chrome Dev was not that good. Here we should also mention one thing: even with a higher frame rate, Firefox's transitions were not that smooth compared to Safari and Chrome (i.e. when the page update through AJAX is occurring, Firefox waits for the zoomed image to be loaded, but WebKit zooms in the previous low-res image, until the actual higher resolution image has been loaded).
Three standards, ten browsers, eleven benchmark suites and a plethora of tests later... It was a heck of a war indeed! We learned a lot about the core of every major browser, and the field(s) of expertise for each of them. Among all the tests
we our browsers have been through, the IE demo tests seem a little biased. The scoresheet that they produced is eager to prove Internet Explorer 9 as the mightiest among the browsers. We will check if these tests were 'specially' designed for IE9 or not. Until then, we will not take the data too seriously for our inference.
We can see how serious Microsoft is to use IE9 to recover the gradually decreasing usage share of their browser. These tests and the ones conducted months before show how ancient the current Internet Explorer core is. No wonder, organizations are using it as the easiest firewall till date, to limit the online activities of their employees. They don't really have any particular field to polish IE9 for, as all the stings of the Trident need a complete revamp! So far, Microsoft has done a great job in making IE relevant again but they have a long way to go still...
Performance wise, the second-worse are Mozilla's Gecko-based browsers. Surprisingly, they are next in the global browser market share too, after IE (and are having a hard time maintaining the captured percentage). In our fastest browser hunt, the Mozilla browsers bled badly and we speculated enhancement in near future. Although some progress is perceivable, it's not much, leaving it behind many competitors. While it comes to quality and support for standards, however, the Firefox browsers have shown good improvement: recognizing and handling most of the HTML5 and CSS3 features, the newer builds also offer experimental features such as GPU accelerated WebGL rendering. As promised by Mozilla, we expect to see the Fox claim the web again; being faster than ever, while also providing the quality - that it already does.
Google Chrome: with less than two years of age, we may call it an infant, but it's by no means an amateur. Chrome reminds us a lot about the David and Goliath story, shooting down the big players one by one. Taking the browser usage share into account, it has already taken over Opera and Safari, and is breathing down Firefox's neck - and for good reasons too. The V8 engine is getting breathtakingly fast with each iteration, and is beating any browser at any level, from almost every perspective. In HTML5 support context, Chrome supports the most number of audio/video codecs till date, as well as excelling in other features, such as geo-location, web application, canvas, etc. As for its support for CSS3 standard, Chrome does its best not to miss out any aspect of it; however questions can be raised from the perspective of render-quality. But as the standard itself is still shaping up, we can't really blame Chrome just now; especially considering the pace at which it is moving: Google will probably wipe off the imperfections by the time CSS3 matures.
Having all the scores in hand we can finally draw the inference drawn. As preview builds should not be judged on par with the stable builds -- they typically contain bugs and unstable experimental features, such is their nature -- they will not be eligible to claim the crown. We will take the scores of the Beta, Dev and Nightly builds as only an indication for the future of these browsers. Taking the performance data of only the stable builds into consideration, it is clear as daylight that Google Chrome brings the best to the web. It has proved itself in one test after another -- no matter what test was thrown at it, the output was at least the same as others, if not better. We declare Google Chrome the champion; let's continue with the award ceremony, all rise for the web anthem...