I'm creating this thread based on what i read here: http://briefmobile.com/cyanogen-demonstrates-quadrants-flaws
I'd really like to know what lagfix gives me the best results and i lost my faith in quadrant. Are there any decent alternatives? Most benchmarks i know are only good for testing cpu performance...
Its not fair to say that quadrant sucks, and particularly I don't like that the article seems to be implying that the lagfixes that a lot of us are running are simply artificially inflating the SGS's score. The SGS has the best harware on the market crippled by bad firmware, and tbh I think its the lag fix that has allowed the hardware to be reflected in quadrant scores.
Anyway, quadrant doesn't suck. Its not perfect, but it does help you determine for example, how different ROMs on the same device compare for performance, as well as showing off things like GPUs that often don't get seen.
However it just runs a series of tests and weights the results accordingly. Particularly, the I/O seems to be given a lot of weight and so can be nobbled to increase the score. However, as anyone who has experienced excessive lag on their SGS can attest, it also works the other way. Without the lag fix, the SGS is a very pedestrian device, as the ~1000 quadrant scores tell you, while with the fix it FLIES.
Actually, some people are saying they notice no difference with the lag fixes it seems. And I've also seen forums full of people saying they can hear the difference between WASAPI/Exclusive audio and shared audio in Windows. Amazingly, out of the dozens of people trying to destroy me and demand it was implemented, none of them were willing to blind test (they were all just willing to say "it's obvious"). How many people here running the lagfix, have run any kind of blind test, or test with predictable steps to demonstrate a difference? Please, raise your hand people, because there is a huge difference between "it seems lag is gone" and "Lag is verified gone".
Quadrant is a guide, it doesn't test interactivity, and it only tests a few operations. It's similar to how a browser can pass ACID3, but have terrible compliance to new standards. Because people became so convinced of benchmarks accuracy, Nvidia and ATI started optimising for benchmarks.
Normal SSD's are also blazingly fast on benchmarks initially, but if they don't have TRIM, their performance drops significantly. That's another example of something benchmarks don't accurately test (because the testing is incomplete).
Use it to get a general idea of how the phone performs (although, specs might be more useful in some cases), but you probably need profiling and a predictable list of steps to diagnose the lag exactly. Run benchmarks which test according to the types of applications you are planning to run (if you play 3D games for instance, use a 3D game benchmark). But don't rely on them exclusively to tell you how well a device performs, because only running the applications and testing them yourselves can tell you.
LostAlone said:
Its not fair to say that quadrant sucks, and particularly I don't like that the article seems to be implying that the lagfixes that a lot of us are running are simply artificially inflating the SGS's score. The SGS has the best harware on the market crippled by bad firmware, and tbh I think its the lag fix that has allowed the hardware to be reflected in quadrant scores.
Anyway, quadrant doesn't suck. Its not perfect, but it does help you determine for example, how different ROMs on the same device compare for performance, as well as showing off things like GPUs that often don't get seen.
However it just runs a series of tests and weights the results accordingly. Particularly, the I/O seems to be given a lot of weight and so can be nobbled to increase the score. However, as anyone who has experienced excessive lag on their SGS can attest, it also works the other way. Without the lag fix, the SGS is a very pedestrian device, as the ~1000 quadrant scores tell you, while with the fix it FLIES.
Click to expand...
Click to collapse
But do you really think that one-click-lagix or whatever is that much faster than mimocan's one (like 2800 vs 1800)? I don't think so...
andrewluecke said:
How many people here running the lagfix, have run any kind of blind test, or test with predictable steps to demonstrate a difference? Please, raise your hand people, because there is a huge difference between "it seems lag is gone" and "Lag is verified gone".
Click to expand...
Click to collapse
*raisehand*
Predictable steps to demonstrate a difference:
1. Use 2 SGS, firmware of your choice
2. One with, say, 1 Click Lag Fix - the other one stock
3. Now for the test, perform simultaneously:
3.1. Open Market and go to "Downloads", update an app, close.
3.2. Open Contacts, quickly switch to favourites, dial Number. Drop call. Exit.
3.3. Open Messaging, conversation of your choice, quickly switch the writing language, write, send, exit.
3.4. bla bla goanforeva
Now if you don't notice an improvement there and in many other cases you most certainly have done something wrong.
If you are saying people are just being placebo-ed you are implying that everybody else besides you is unable to tell a real difference. Think about it, maybe it's the other way around.
When I got my device it came with the Asian JG4 firmware I think and I used it for 48hrs as a point of reference for this so called "lag" I updated my firmware to the latest we have and found zero difference in terms of performance ( same 2 day usage ) decided to flash my device with JM5 and used it w/o a lag fix for 2 days, notice a very slight improvment on how fast the inbox/contacts can open and then did a fresh flash with lagfix 2.x installed I didnt NOT notice any performance difference but I am always open and welcome to any lagfix that is stable,safe and fast ( in that order ) right now I have JG8 installed w/o lag fix and its very snappy, I still want to install a lag fix for this even if i dont see/feel any performance difference at all.. thats just me.. Now im wondering w/c lag fix is the most stable and safe.. stability/safty > speed
EDIT:
Also I believe it could be possible that certain versions of lagfixes work better with certain versions of firmware also, its all about getting the perfect combination.. altho my question still stands on w/c is the safest lagfix heheheh
Well actually i gave up on benchmarks.
To me the whole user experience is more important. If apps open instantly and the phone runs smooth then it's fine with me. I did notice some firmwares are better then other ones.
Right now i'm running JM1 (rev 3) with CFLagFix1.80 installed which makes the phone very stable and running smoothly so right now i'll stick to this firmware until a proper GPS fix is released or until Froyo is released.
Phandroid said:
*raisehand*
Predictable steps to demonstrate a difference:
1. Use 2 SGS, firmware of your choice
2. One with, say, 1 Click Lag Fix - the other one stock
3. Now for the test, perform simultaneously:
3.1. Open Market and go to "Downloads", update an app, close.
3.2. Open Contacts, quickly switch to favourites, dial Number. Drop call. Exit.
3.3. Open Messaging, conversation of your choice, quickly switch the writing language, write, send, exit.
3.4. bla bla goanforeva
Now if you don't notice an improvement there and in many other cases you most certainly have done something wrong.
If you are saying people are just being placebo-ed you are implying that everybody else besides you is unable to tell a real difference. Think about it, maybe it's the other way around.
Click to expand...
Click to collapse
Lol I think you didn't understand my question... I'm sure that these lag fixes all work, no doubt about that. But which of them is faster? For that i can't rely on quadrant i'm sure...
Related
Many of you guys are good at programming and can whip up cabs in your sleep--I'm not so good at C++, etc but here's my contribution, hope it helps someone.
Analysis of Wizard ROMs
Objective: Try to determine if there is performance improvements in newer ROMs. Also to see if the 12->8 MB paging size has a measureable effect in performance.
Summary of Results:
There doesn't appear to be much of a measurable difference in performance among these ROMs, with the exception of the last two ROMs tested. For the rest of the pack, the hardware responds about the same with a little noise. Summitter's 2.17 ROM appears a little above average, but this could be just a testing glitch, and its not significantly above the others. This appears to indicate that the upgrade to decrease the paging pool has no effect (to this benchmark).
I'm concerned about the consistency of results with the last two. They were performed under slightly different test conditions (at home, vs at work for the others), and there's no reason ShogunMark's at least (if not both) shouldn't be closer to the others in performance. I plan to rerun these tomorrow and see how they compare.
Method: I flashed a decent cross-section of the ROMs out there--don't feel insulted if yours isn't included. In fact, I'd be happy to add yours to the mix--just ask! For each ROM listed, I flashed then immediately recorded stats from Settings>System> About, Device Info, and Memory. Then added PC Pro Labs Pocket PC Benchmark and embedded vb runtime (required for Benchmark), removed USB cable, soft-reset and ran benchmark software once to obtain last few columns of results.
Assumptions: The biggest assumption is that this is decent benchmarking software. While it was written for WM2003, its probably not a big stretch to assess WM2005. I konw Spb has a benchmark program, but it ran for about 1.5 hours on the first ROM. Too long for me, guess I'm just impatient! Other big assumption is that higher values are better, although this appears inversely releated for the file read/write benchmarks compared to the kb/sec measurements. This may also be true for the others, butsince they're all about the same it really doesn't matter.
Additional Observations: At the very least, this was an interesting exercise to record some comparable data regarding the different ROMs. ROMs have evolved over time, and sometimes there are questions about which ROM contains what, etc. This might lead to another idea where we keep track of the ROMs in some type of registry to track consistent information about each.
Analysis Improvements: This could be improved by using more current benchmark software. Spb Benchmark is a decent candidate, but takes a while to run. Also, the analysis could be improved with many benchmark runs per ROM instead of just one run. This would average out the "noise" betweens runs and might give a better indication of slight performance trends.
Finally, the method used above will be repeated under similar conditions for the last two ROMs to see if the provided results are valid.
Terms:
BM - Benchmark
GDI - graphics display test
CPU - central processing unit
kb/s - kilobytes per second (used for read/write tests).
wow, its like a report card for roms, lol and good work by the way...
however, is it possible to lay out the results in a better format?
nevermind.. i noticed this new damn board wont let you upload an excel file, i did it myself.. thanks again.. and i agree it is odd that the last 2 are lower than everyone else's but still close to each other.. i would vote computer as well
Good job! This pretty much confirms what I've been thinking for a long time...there's no real difference in performance. If you're going to add or alter the list, I'd like to also see the stock T-Mobile 2.26 rom thrown in there as well. Maybe if people see that "it seems faster" doesn't mean anything and they're all pretty much the same, they might think twice about risking their expensive toy for a reason they probably can't define.
zip it up and then upload it...
lvlolvlo said:
zip it up and then upload it...
Click to expand...
Click to collapse
Good idea! Zip includes the Excel file of results and run report for each of 8 runs performed so far with PC Pro benchmark.
Didn't get to updating this with my work computer today, but I will in the next few days.
Measuring units?
hi, jorge_culv, least numbers mean quicker?
Good question, and I honestly don't know the answer. There's no documentation with this benchmark, even on the hosting company's website. Other benchmarks I've used had higher numbers as better. There is one clue if you look closely at the file read and write tests for each ROM. It appears the lower benchmark scores match up with the faster read and write speeds, so for those it appears lower is better, not sure if that also applies to the CPU and graphics tests.
Also, I'm real hesitent about the last 2 runs--don't read too much into those until I can do more follow up testing (hopefully in the next few days).
markgamber said:
Good job! This pretty much confirms what I've been thinking for a long time...there's no real difference in performance. If you're going to add or alter the list, I'd like to also see the stock T-Mobile 2.26 rom thrown in there as well. Maybe if people see that "it seems faster" doesn't mean anything and they're all pretty much the same, they might think twice about risking their expensive toy for a reason they probably can't define.
Click to expand...
Click to collapse
Mark, I'll try to throw in a T-Mo ROM as well. I'm not sure if this test really proves there's no difference in performance. I might be splitting hairs, but maybe it only proves there's no difference in CPU math, graphics or file read/write--in other words, the ROM upgrades affect performance in ways not measured by this benchmark. The factors measured by this benchmark sure seem more hardware related and maybe not influenced by the ROM at all. I was really hoping for a difference between the newer "30MB" roms, as that may affect read/write speed through the smaller page file, but the verdict is still out...
True, but it might be interesting to take a look at. I don't mean to come down on people creating these roms but personally, if there's a performance difference between any of them and stock 2.26, it's so small that I've never noticed it. When you compare a minute performance difference to the boost of overclocking the cpu, you wonder why you ever bothered risking the phone in the first place. At least I wondered why and, unable to come up with an answer other than "because I can" and not really wanting to throw away $350, I stopped putting new roms on the phone and won't do it again without a damn good reason.
Vladimyr said:
hi, jorge_culv, least numbers mean quicker?
Click to expand...
Click to collapse
Lower the number the better except for File read/write. I'm intrigued as to what gives the bottom two their boost in the benchmarks.
so what do you guys suggest as the best rom with both speed / batteries / stability?
More testing
I have 3 ROM version by this moment:
Qtek 2.18
T-mobile 2.26
AKU 3.2 on T-mobile 2.26
And i make test of CPU load on every ROM update by Rhino Stat.
On Qtek 2.18 and AKU 3.2 on T-mobile 2.26 CPU has constantly load 8-9% in stand-by mode.
And T-mobile 2.26 ROM has 0-1% CPU load in stand-by mode.
Okay, another question that makes me feel kinda dumb.
How can I improve my N1's performance?
Specifically, how in the world do I get my nexus even close to the N1's I see on Linpack's "Top Devices"?
I'm running diet_2.6.33.2.hybrid_avs_925mv_OC with cyanogen 5.0.6. I can get 7.778 MFLOPS. Honestly, I don’t even know what that means. All I know is that I want more. Or do I? Does it even make that much of a difference in everyday usage?
I've noticed that in the JIT thread under development they talk about getting up to 17 MFLOPS, but no one really explains what JIT is or does. Just that it's unstable and that if your not sure what it is, you don't need it.
I can benchmarkPi in 2790 - how in the hell do I get it lower? Under the rom description of some of the top 50 they mention JIT and experimental kernels, but there's not much room for more than just the generic names of them. Where do I get these magical roms and kernels? Do I even want them? Are they so unstable that they can only be used to gain geek cred on some benchmark lists and once they hit the "high score" they explode in flames? Do I need safety goggles before I flash the update.zip?
Seriously though, I know I'm not the only one with these questions - so any info would help. And do me a favor, please do not respond to say that if I don't know - I don’t need to know. At one point, you didn't know either. No one was born with this knowledge.
I know there is a very bright person on here that is willing to take a couple of minutes to shed some light on us simpletons. I'd like to thank you in advance. Thanks.
Even if you get higher linpack scores, you will barely notice any difference in your devices performance... That's why I tend to ignore it. If its possible to get faster than Cyanogen ROMs (not in terms of linpack) I don't know how different things would be.. im curious about how quick Froyo is though.
Oh and have you searched, Jit has been covered in more places than just xda developers, several times it has been discussed I believe.. while using it, my Linpack scores did get into the double digits though - but this (not even that noticeable actually) speed increase was not worth the random reboots, and such. When Jit is stable though, it will be quite nice.. hopefully Froyo has it.
Linpack number did not make any differences in overall usage . Now, I am using my own UV kernel and setcpu max freq to 500MHz.. only... but, I still able to get 26.8 fps in Neocore's 3D benchmark. Playing games, 3D or not, still smooth and fluid.
I also waiting for Froyo and hope we can get even efficient and fast OS.
TQ.
Hi everyone,
I have question. My result in Quadrant Standard is something about 800. A lot of people have higher result that's mine. There is any method to do SGS faster and get better results or where is problem in my phone ??
if you want higher quadrant scores you need to do a lagfix. the market has one called one click lag fix but it roots your phone. imo quadrant scores arent indicative of real world phone speed. if it feels fast, it is fast. go to the development section and read up on why this pone lags and all the different lag fixes available. but basically samsung uses a slow file system which causes 'lag' but i guess its all relative to what you are used to.
Everything is ok now. Thanks for your help
Hi guys,
I've just run a Quad Standard test on my N1 running CM7 RC2 and am getting a score of 1199, which is lower than even the stock 2.2. I've tried with SetCPU ON & OFF, but got pretty much the same score?
A while back, I've flashed the RODS MIUI ROM & the WM Kernel (1152MHz) and ran the test only to get a score of 1192.
Wonder what could be wrong??
It's a synthetic benchmark and therefore meaningless. Does the phone run ok, does it feel like there's a problem?
The only thing Quadrant is good at measuring is how well a ROM is optimised for Quadrant
The phone runs absolutely fine. Its not like i have any issues with it, but just wanted to compare how it performs with different ROM's.
karthik247 said:
The phone runs absolutely fine. Its not like i have any issues with it, but just wanted to compare how it performs with different ROM's.
Click to expand...
Click to collapse
Best to break out the stopwatch and perform some real world tests. They're the only ones that matter
The only major difference i noticed after switching to MIUI from CM7 is that booting is super fast. Besides that, everything else felt pretty much the same.
OFF TOPIC
Does anyone know how to get rid of the power widget on the notification bar?
The option is in the CM settings which can be found in the usual settings
I noticed this too when i ran the benchmark, but simply dismissed it because its not the final version of the rom, and just a RC
DirkGently1 said:
The option is in the CM settings which can be found in the usual settings
Click to expand...
Click to collapse
I meant in the MIUI ROM.
Also wanted to know which kernel is the better among Pershoot and WM?
After using the phone with MIUI for a couple of hours now, figured out two issues.
1. When im getting a call, the trackball starts flashing and only after a good 2-3secs does the ringer go off.
2. And while playing music videos, the audio-video sync doesnt seem to be perfect. Its as if the phone is lagging a wee bit. Could this be cuz i only got around 40mb of free space on the device??
Quadrant scores, IMO, mean nothing. They give you a numerical value on the specs and capabilities thattthe phone can do under certain situations but they don't give you any real world numbers.
Yes the scores on my 2.3.3 N1 were "low" as. Well but how do you explain it being just as fast (no exaggeration here) as a stick Atrix? I had both and took the Atrix back.
The Atrix has amazing specs and its hardware is awesome, but as an overall feel I didn't like it, the whole experience didn't feel fluid to me.
All in all, quadrant scores are good for testing stock phones and comparing specs to each other, but you should by no means use that to judge the quality or value of a phone.
That would be like buying a car off of eBay by only comparing specs and stuff written on paper; you ddon't have a feel for the car.
Sent from my Nexus One using XDA Premium App
Hi, so after cm rooting i noticed there is also an ext4 upgrade...is this worth it and will I have data lose during the change? Im on 2.3.3 and am wondering if it is really worth it! Ed
are you kidding me?
besides that this post should not even be in this forum (try Q & A) there are hundreds of other posts and threads discussing this topic if you only use your -(0-0)- !
Yes and no
It may have some advantages but as for what I'm not sure but as for nay major advantages I would say no. I'm still using EXT2 and my benchmarkes are still 1900+. I've tried almost all the other lagfixes and found no real advantage. The only thing I hate about custom kernels with lagfix is the secondary samsung splash screens and custom splash screen. They kill the post time.
I would say it isn't as worth it as what people say. First thing most people do after converting their filesystem is run quadrant. It's possible that improved quadrant scores don't translate to real world performance though.
Even worse, running quadrant actually engages the placebo effect so you walk in with a more positive impression. Meanwhile, I don't recall ever seeing anyone from the EXT4/EXT2 i9000 community running blind tests, and neither development community has actually shown any evidence formal testing has been performed. All the arguments seem to be based around quadrant and PC testing. If methods like this were applied to clinical testing, every drug would pass
Honestly, give both a try, but do it blindly.
Why Quadrant may be wrong
This is just a bit of background why Quadrant's scores may not reflect real life performance. Until we check the actual ratio's of Quadrant, and compare with actual usage ratio's though, we can't identify how "real" it's scores are.
Consider a benchmark which produces 1 final score. It may be calculated by:
[MAX TIME - Time to read 1000mb] + [Max time - time to write 1000MB]. In this case, both scores contribute to 50% of the final score, which can be worth 2x MAX Time.
Scenario 1: Time to read/write is both the same
Scenario 2: Read time is 1% shorter than Scenario 1, but write time is 1% longer. Both will have the same score in Quadrant..
Scenario 3: Read time is 5% longer than Scenario 1, but write time is 50% shorter. Scenario 3 will get the best score
Which one is ACTUALLY faster though. The benchmark-toting individuals will claim Scenario 3 is faster, because of the score. HOWEVER, that may be incorrect. Consider the following:
If a user reads 100x more data than they write:
1) Clearly, faster read scores are more important.
2) The BEST filesystem will be Scenario 2, despite being equal last.
3) Scenario 1 will be mid place
4) The scenario with the best score, will actually have the WORST performance.
5) A drop in 1% read performance would need a HUGE increase in write performance to actually be faster.
Until we have an idea of how accurate Quadrant REALLY is, run your own tests, and do so without knowing which filesystem is running. High quadrant scores may boost your e-penis size, but as you can see, it is theoretically possible for the scores which are produced to score slower performing filesystems more highly than faster ones. Disappointingly though, a decreasing number of users/developers at XDA these days are actually interested in the truth, and simply in not being wrong.
Even worse, the community for some reason seems VERY anti-RFS, and wont give it a chance regardless. It might be a LOT better than it used to be. Either way, it seems to be good enough for me.
Ignore the theatrics and run a blind test. That's the only way to determine what is ACTUALLY faster.
monkeytennis said:
Hi, so after cm rooting i noticed there is also an ext4 upgrade...is this worth it and will I have data lose during the change? Im on 2.3.3 and am wondering if it is really worth it! Ed
Click to expand...
Click to collapse
I guess you mean CF root right? Will just answer the data loose question. No you shouldnt. But do a backup before. Its fast and easy (if you are on CF root that is)
If its worth depends on you? You experience any lag on rfs? Phone slow?
ramrod54 said:
It may have some advantages but as for what I'm not sure but as for nay major advantages I would say no. I'm still using EXT2 and my benchmarkes are still 1900+. I've tried almost all the other lagfixes and found no real advantage. The only thing I hate about custom kernels with lagfix is the secondary samsung splash screens and custom splash screen. They kill the post time.
Click to expand...
Click to collapse
@ramrod54 , where did you get ext2 support on JVK? And what the?
What samsung splash screens and custom splash? What rom and kernel are you on? And what lagfixes? And we both know quadrant score doesnt matter does it ?
Yes, it's worth it. Some things (Android Market, Gmail) works really MUCH faster then on rfs.
Unrealwolf said:
Yes, it's worth it. Some things (Android Market, Gmail) works really MUCH faster then on rfs.
Click to expand...
Click to collapse
Hrm, I've never noticed a difference. Market is slow on any room, hell market.android.com is slow too. Gmail always worked fast for me, on any rom.
Personally, I don't see or feel the point in using anything but rfs, but I suppose if someone has half a bazillion apps installed, then maybe an alternative filesystem might be better.
I say try it. If you notice a difference, good for you. If you don't, then stick with rfs.
What about battery performance ? from what I have read, battery performance is also better with RFS.
I always use ex4, its not as needed now on gingerbread but I just prefer the file system...it is better than RFS...but RFS has improved a great deal so you may not noticed that much difference, the rom may become a little smoother..You wont get data lose because of ex4, maybe the way the kernel is built...dont forget that 2.3.3 gingerbread is still beta and without the source code for the kernel you cant expect great things yet, although chainfire has done some amazing work and now we can change the file system using his app....works really well.
Also regarding battery, the difference in performance is such a small margin that its not even an issue.
What alot of people aren't aware of is that the Nexus S for example uses ex4 file system as default straight out of the box
Its not just a lagfix for the galaxy s, its a very good file system too...
Just a side note on Quadrant, ex4, ex2 will trick the app...if you buy the pro version, you will see how much the file system stretches on the bar...Quadrant is more for fun....or HTC
Think it´s worth cause rfs slow down your system after a while