Database Develop

ROM Comparison Analysis

Many of you guys are good at programming and can whip up cabs in your sleep--I'm not so good at C++, etc but here's my contribution, hope it helps someone.
Analysis of Wizard ROMs
Objective: Try to determine if there is performance improvements in newer ROMs. Also to see if the 12->8 MB paging size has a measureable effect in performance.
Summary of Results:
There doesn't appear to be much of a measurable difference in performance among these ROMs, with the exception of the last two ROMs tested. For the rest of the pack, the hardware responds about the same with a little noise. Summitter's 2.17 ROM appears a little above average, but this could be just a testing glitch, and its not significantly above the others. This appears to indicate that the upgrade to decrease the paging pool has no effect (to this benchmark).
I'm concerned about the consistency of results with the last two. They were performed under slightly different test conditions (at home, vs at work for the others), and there's no reason ShogunMark's at least (if not both) shouldn't be closer to the others in performance. I plan to rerun these tomorrow and see how they compare.
Method: I flashed a decent cross-section of the ROMs out there--don't feel insulted if yours isn't included. In fact, I'd be happy to add yours to the mix--just ask! For each ROM listed, I flashed then immediately recorded stats from Settings>System> About, Device Info, and Memory. Then added PC Pro Labs Pocket PC Benchmark and embedded vb runtime (required for Benchmark), removed USB cable, soft-reset and ran benchmark software once to obtain last few columns of results.
Assumptions: The biggest assumption is that this is decent benchmarking software. While it was written for WM2003, its probably not a big stretch to assess WM2005. I konw Spb has a benchmark program, but it ran for about 1.5 hours on the first ROM. Too long for me, guess I'm just impatient! Other big assumption is that higher values are better, although this appears inversely releated for the file read/write benchmarks compared to the kb/sec measurements. This may also be true for the others, butsince they're all about the same it really doesn't matter.
Additional Observations: At the very least, this was an interesting exercise to record some comparable data regarding the different ROMs. ROMs have evolved over time, and sometimes there are questions about which ROM contains what, etc. This might lead to another idea where we keep track of the ROMs in some type of registry to track consistent information about each.
Analysis Improvements: This could be improved by using more current benchmark software. Spb Benchmark is a decent candidate, but takes a while to run. Also, the analysis could be improved with many benchmark runs per ROM instead of just one run. This would average out the "noise" betweens runs and might give a better indication of slight performance trends.
Finally, the method used above will be repeated under similar conditions for the last two ROMs to see if the provided results are valid.
Terms:
BM - Benchmark
GDI - graphics display test
CPU - central processing unit
kb/s - kilobytes per second (used for read/write tests).

wow, its like a report card for roms, lol and good work by the way...
however, is it possible to lay out the results in a better format?

nevermind.. i noticed this new damn board wont let you upload an excel file, i did it myself.. thanks again.. and i agree it is odd that the last 2 are lower than everyone else's but still close to each other.. i would vote computer as well

Good job! This pretty much confirms what I've been thinking for a long time...there's no real difference in performance. If you're going to add or alter the list, I'd like to also see the stock T-Mobile 2.26 rom thrown in there as well. Maybe if people see that "it seems faster" doesn't mean anything and they're all pretty much the same, they might think twice about risking their expensive toy for a reason they probably can't define.

zip it up and then upload it...

lvlolvlo said:
zip it up and then upload it...
Click to expand...
Click to collapse
Good idea! Zip includes the Excel file of results and run report for each of 8 runs performed so far with PC Pro benchmark.
Didn't get to updating this with my work computer today, but I will in the next few days.

Measuring units?
hi, jorge_culv, least numbers mean quicker?

Good question, and I honestly don't know the answer. There's no documentation with this benchmark, even on the hosting company's website. Other benchmarks I've used had higher numbers as better. There is one clue if you look closely at the file read and write tests for each ROM. It appears the lower benchmark scores match up with the faster read and write speeds, so for those it appears lower is better, not sure if that also applies to the CPU and graphics tests.
Also, I'm real hesitent about the last 2 runs--don't read too much into those until I can do more follow up testing (hopefully in the next few days).

markgamber said:
Good job! This pretty much confirms what I've been thinking for a long time...there's no real difference in performance. If you're going to add or alter the list, I'd like to also see the stock T-Mobile 2.26 rom thrown in there as well. Maybe if people see that "it seems faster" doesn't mean anything and they're all pretty much the same, they might think twice about risking their expensive toy for a reason they probably can't define.
Click to expand...
Click to collapse
Mark, I'll try to throw in a T-Mo ROM as well. I'm not sure if this test really proves there's no difference in performance. I might be splitting hairs, but maybe it only proves there's no difference in CPU math, graphics or file read/write--in other words, the ROM upgrades affect performance in ways not measured by this benchmark. The factors measured by this benchmark sure seem more hardware related and maybe not influenced by the ROM at all. I was really hoping for a difference between the newer "30MB" roms, as that may affect read/write speed through the smaller page file, but the verdict is still out...

True, but it might be interesting to take a look at. I don't mean to come down on people creating these roms but personally, if there's a performance difference between any of them and stock 2.26, it's so small that I've never noticed it. When you compare a minute performance difference to the boost of overclocking the cpu, you wonder why you ever bothered risking the phone in the first place. At least I wondered why and, unable to come up with an answer other than "because I can" and not really wanting to throw away $350, I stopped putting new roms on the phone and won't do it again without a damn good reason.

Vladimyr said:
hi, jorge_culv, least numbers mean quicker?
Click to expand...
Click to collapse
Lower the number the better except for File read/write. I'm intrigued as to what gives the bottom two their boost in the benchmarks.

so what do you guys suggest as the best rom with both speed / batteries / stability?

More testing
I have 3 ROM version by this moment:
Qtek 2.18
T-mobile 2.26
AKU 3.2 on T-mobile 2.26
And i make test of CPU load on every ROM update by Rhino Stat.
On Qtek 2.18 and AKU 3.2 on T-mobile 2.26 CPU has constantly load 8-9% in stand-by mode.
And T-mobile 2.26 ROM has 0-1% CPU load in stand-by mode.

comparisons rom

scuse me for my english.
it always speaks of the Rom more or less good and I am the first to say this is good that it is better to say 'no' then maybe this was better, etc.
I mounted all around Rom, and virtually all those that come on xda is fitted and then removed to prove me other good and seemed wrapped wrapped, always the same, less good. certainly the effect placebo time ago but there is actually the best one, only that it would close two phones on which to mount two different rom and make the various comparisons, but unless you're rich and crazy if not slightly.
So I ask you, and especially to the great forum, is not that you have programs of measurements dificil video from running on coreplayer to try on two rom then to measure the data, measurements for opening files word, excel and so heavy compared with stopwatch hand, programs that measure the gps fix, etc? if all we could do the same tests, using the same criterion to we will be and accurate comparisons between the Rom.
How about that work and do that? will be

Sorry, but your post is really inapprehensible, even after reading it two times I don't know what you are talking about.

I excuse your English, and i understand. But most good rom / bad rom comparison aren't as simple as that. They all use same OS, Mostly same driver set ect. so hard statistics on frame rate in core player would be mostly identical same for GPS ect. A few Roms do advise improved performance and if so usually DO show stats such as RAM use, Frame rate improvement. but mostly what is judged when someone says good rom/bad rom is the applications integrated - too many or too few, Which apps. Theme integration, How many bugs there are or lack of, whether fixes for know bugs are up to date. Also the Roms personality or theme. I agree its really hard to compare Roms going back and forth with only opinion to judge by. but that is what it mostly comes down to. To one person Barebones 1.5 would Rock "nice and lite" to another Davidueck v4.5 would be the ultimate filled with apps. Most people are in the middle and so say i like it "But . . ."
Best is decide if you want a Lite rom very few apps small storage space. or a heavy rom with many apps and customizations pre-installed then you compare versions... like Rom A has googlemaps 2.2...Rom B has Googlemaps 3.0.0.3 hmmmmm "will i use googlemaps?"
hope this helps you.
And umm Johnpatcher . . . My mother once told me something maybe good for you.. you're "Senior" and all but still... My mother told me... "If you can't think of something Civil to say, then just keep you mouth shut." If you can't understand his English and no idea what he means.. why reply? why waste his/my/your time? He apologized in advanced for his English, his point is a valid idea but uninformed. If it bother you just ignore him. Rather than taking a cheap shot

Performance

Okay, another question that makes me feel kinda dumb.
How can I improve my N1's performance?
Specifically, how in the world do I get my nexus even close to the N1's I see on Linpack's "Top Devices"?
I'm running diet_2.6.33.2.hybrid_avs_925mv_OC with cyanogen 5.0.6. I can get 7.778 MFLOPS. Honestly, I don’t even know what that means. All I know is that I want more. Or do I? Does it even make that much of a difference in everyday usage?
I've noticed that in the JIT thread under development they talk about getting up to 17 MFLOPS, but no one really explains what JIT is or does. Just that it's unstable and that if your not sure what it is, you don't need it.
I can benchmarkPi in 2790 - how in the hell do I get it lower? Under the rom description of some of the top 50 they mention JIT and experimental kernels, but there's not much room for more than just the generic names of them. Where do I get these magical roms and kernels? Do I even want them? Are they so unstable that they can only be used to gain geek cred on some benchmark lists and once they hit the "high score" they explode in flames? Do I need safety goggles before I flash the update.zip?
Seriously though, I know I'm not the only one with these questions - so any info would help. And do me a favor, please do not respond to say that if I don't know - I don’t need to know. At one point, you didn't know either. No one was born with this knowledge.
I know there is a very bright person on here that is willing to take a couple of minutes to shed some light on us simpletons. I'd like to thank you in advance. Thanks.

Even if you get higher linpack scores, you will barely notice any difference in your devices performance... That's why I tend to ignore it. If its possible to get faster than Cyanogen ROMs (not in terms of linpack) I don't know how different things would be.. im curious about how quick Froyo is though.
Oh and have you searched, Jit has been covered in more places than just xda developers, several times it has been discussed I believe.. while using it, my Linpack scores did get into the double digits though - but this (not even that noticeable actually) speed increase was not worth the random reboots, and such. When Jit is stable though, it will be quite nice.. hopefully Froyo has it.

Linpack number did not make any differences in overall usage . Now, I am using my own UV kernel and setcpu max freq to 500MHz.. only... but, I still able to get 26.8 fps in Neocore's 3D benchmark. Playing games, 3D or not, still smooth and fluid.
I also waiting for Froyo and hope we can get even efficient and fast OS.
TQ.

[Q] Mflops?

I had a Samsung Moment and most of the rom descriptions included the Mflop scores. I sold it and got a Hero and I've noticed pretty much no one lists or talks about Mflop scores? Is there a reason for this, is it not important since the Hero is so slow anyways?
The fastest score I've seen was with the new AospMod rom 5.6. Do any of the 2.1 roms compare?

reckoner13 said:
I had a Samsung Moment and most of the rom descriptions included the Mflop scores. I sold it and got a Hero and I've noticed pretty much no one lists or talks about Mflop scores? Is there a reason for this, is it not important since the Hero is so slow anyways?
The fastest score I've seen was with the new AospMod rom 5.6. Do any of the 2.1 roms compare?
Click to expand...
Click to collapse
I think it's because no one here really cares about it. Mostly because it doesn't matter. Most of us on the hero boards are more concerned with important stuff like stability, and coming from a moment and calling the hero slow is a case of the pot calling the kettle black. ;-)

reckoner13 said:
I had a Samsung Moment and most of the rom descriptions included the Mflop scores. I sold it and got a Hero and I've noticed pretty much no one lists or talks about Mflop scores? Is there a reason for this, is it not important since the Hero is so slow anyways?
The fastest score I've seen was with the new AospMod rom 5.6. Do any of the 2.1 roms compare?
Click to expand...
Click to collapse
MFlops is an arbitrary cpu scoring system much like Bogomips for Linux and you see it posted on almost every roms OP. The problem is that its affected by tons of stuff and just because aospbot posts a 5.6 doesn't mean thats what you'll get, you might be running a ton of widgets and facebook sync and other stuff.
CM6 also got a 5.6 or so and I've seen 2.1 sense rom also post a high scores like this. You should use it yourself and determine what is a good score on your phone under normal conditions and then use that as a bases for scoring other roms on YOUR phone.
Mflops are usually calculated by how many Floating Point operations it can calculate per second (usually pi to some exponent), while Mips is how many Fixed point operations it can calculate a second.
So pretty much ignore it unless your running it on your phone.

user7618 said:
I think it's because no one here really cares about it. Mostly because it doesn't matter. Most of us on the hero boards are more concerned with important stuff like stability, and coming from a moment and calling the hero slow is a case of the pot calling the kettle black. ;-)
Click to expand...
Click to collapse
Come on the moment has an 800mhz processor that can be overclocked and the same amount of RAM as the Hero (288). It's faster, but that doesn't mean it's better. I prefer the Hero.

reckoner13 said:
I had a Samsung Moment and most of the rom descriptions included the Mflop scores. I sold it and got a Hero and I've noticed pretty much no one lists or talks about Mflop scores? Is there a reason for this, is it not important since the Hero is so slow anyways?
The fastest score I've seen was with the new AospMod rom 5.6. Do any of the 2.1 roms compare?
Click to expand...
Click to collapse
Mflops is still used, but it is no longer a primary measure of a phone's capabilities. Most people use Quadrant to determine a phone's performance, as that incorporates 2D and 3D graphic performance in addition to calculations. But I've heard people use BenchmarkPi, CPU Benchmark, etc. It all is really quite irrelevant if your phone is fast enough and performs well enough for you; posting scores of any type is really just a digital pissing contest.

Performance ROM+Kernel

Hello. I would like to ask whitch ROM + Kernel gives best performance in games? Becouse now I barely get 20k points in Antutu... and fps is jumping crazy in 3d test. Does this has anything to do with throttling?

First of all I would like to say that benchmarking apps are a bunch of BULL! Don't rely on them. Complain only if you find lags/stutters in real gaming. But yes, throttling does effect it.
As far as ROMs are concerned, such threads are not allowed here. Head over to the android development section and see which suits you the best.

"What's the best" ROM/Kernel threads are not allowed. Please just try them for yourself.
Closed