Jun 26, 2009

Speed Test: iPhone 3GS is faster than Apple claims

Apple has made claims that the iPhone OS 3.0 yields significant performance gains on the 3G model, and that the new 3GS can accomplish the same tasks up to twice as fast as its predecessors. Anecdotally, the new 3GS definitely “feels” faster under certain conditions. But how do Apple’s devices and OS versions really compare to one another? And perhaps of even greater interest, how does the latest hardware from Cupertino compare to smart phones recently released from other vendors?

“Objavectaweb-C” OS?
One of the key challenges in conducting an objective evaluation of software performance across devices that utilize different operating systems lies in accounting for the fundamental differences in the various OSs. While the iPhone 3G and 3GS could potentially run the same app on the same Objective-C-based operating system (making direct comparisons relatively straightforward), Android apps are Java-based, and the Palm Pre runs the entirely new Web OS. Given these divergent OS implementations, is there anything that come close to a standard unit of measure for judging performance of this growing breed of “superphones?”

Finding Common Ground
The common thread between these three OS’s is JavaScript execution in WebKit—the open source project that, in varying degrees, powers web browsing technology for these three disparate operating systems. With the exception of certain browser plugins (e.g., Flash), web rendering technology installed on today’s premiere mobile devices makes almost all—and in some cases even more—features of their ubiquitous desktop web browser counterparts available. Therefore, given the global commonality of JavaScript and WebKit-based web browsers, it becomes possible to compare the performance of these “pocket computers that make phone calls” to the performance of desktop machines.

The Yardstick
The WebKit Open Source Project provides a JavaScript test Suite dubbed SunSpider. According to the description on the SunSpider home page, “this benchmark tests the core JavaScript language only, not the DOM or other browser APIs. It is designed to compare different versions of the same browser, and different browsers to each other.” We at Medialets have found it to be one of the best attempts to measure real world JavaScript performance in a balanced and statistically sound way.

Medialets ran the SunSpider test suite in the following environments:
  1. Safari 4.0.1 on a 2.0 GHz Intel Core 2 Duo White MacBook.
    The MacBook results were used as a baseline for relative comparisons.
  2. Mobile Safari on the iPhone 3G with iPhone OS v2.2.1
  3. Mobile Safari on the iPhone 3G with iPhone OS v3.0
  4. Mobile Safari on the iPhone 3GS with iPhone OS v3.0
  5. The “Browser” app on the T-Mobile G1 with Android OS v1.5 (Cupcake)
  6. The “Web” app on the Palm Pre with Web OS v1.0.2
Each device was fully restored and rebooted immediately before running the test suite. Every attempt was made to assure that no atypical background tasks were executing while the tests were running. The SunSpider tests automatically run five times sequentially and the mean average from all five tests are reported. Network speed and latency have no effect on the results of the test.

Disclaimer: Before considering the results of the tests, it is important to note that each OS likely has certain advantages and features that probably make it inherently well suited for some tasks more than others. The main purpose of these comparisons is merely to compare JavaScript performance within each environment. It should not be misconstrued as indicative of which device or OS is inherently “better” than any other.
The results of the iPhone-based tests alone are rather astonishing and seem to indicate that many of Apple’s claims about the performance gains of their 3.0 OS and the iPhone 3GS may hold some water. Using OS 3.0 on the same iPhone 3G yields nearly 3X the JavaScript performance in Mobile Safari vs. using iPhone OS 2.2.1. The iPhone 3GS ups the ante by another factor of 3, bringing JavaScript performance on the iPhone 3GS to just 12X that of a full-powered desktop machine that has well over four times the raw processing muscle alone. The T-Mobile G1 running the “Cupcake” version of the Android OS completed the test suite in about 91 seconds. This makes it about a third faster than the iPhone 3G running Apple’s previous OS (2.2.1). The Palm Pre came storming out of the gate with speeds that closely rival the iPhone 3G running Apple’s latest iPhone OS.

Do any of these numbers really indicate which phone might be the best choice for a given individual? Absolutely not. At Medialets we use all of these devices, and love each one for many reasons. The fact that these tests can even be performed across this many device/OS combinations is a testament to how far mobile technology has come in such a relatively short time. We are looking forward to seeing an even greater variety of advanced mobile devices and OS revisions enter the market and we’ll keep you posted as we test more devices in our lab. Subscribe to our feed, leave a comment below, or reach out to us directly if you have any questions.


No comments:

Boorkmark & Share

Bookmark Options