Database Develop

Quadrant sucks, howto properly bench our devices?

I'm creating this thread based on what i read here: http://briefmobile.com/cyanogen-demonstrates-quadrants-flaws
I'd really like to know what lagfix gives me the best results and i lost my faith in quadrant. Are there any decent alternatives? Most benchmarks i know are only good for testing cpu performance...

Its not fair to say that quadrant sucks, and particularly I don't like that the article seems to be implying that the lagfixes that a lot of us are running are simply artificially inflating the SGS's score. The SGS has the best harware on the market crippled by bad firmware, and tbh I think its the lag fix that has allowed the hardware to be reflected in quadrant scores.
Anyway, quadrant doesn't suck. Its not perfect, but it does help you determine for example, how different ROMs on the same device compare for performance, as well as showing off things like GPUs that often don't get seen.
However it just runs a series of tests and weights the results accordingly. Particularly, the I/O seems to be given a lot of weight and so can be nobbled to increase the score. However, as anyone who has experienced excessive lag on their SGS can attest, it also works the other way. Without the lag fix, the SGS is a very pedestrian device, as the ~1000 quadrant scores tell you, while with the fix it FLIES.

Actually, some people are saying they notice no difference with the lag fixes it seems. And I've also seen forums full of people saying they can hear the difference between WASAPI/Exclusive audio and shared audio in Windows. Amazingly, out of the dozens of people trying to destroy me and demand it was implemented, none of them were willing to blind test (they were all just willing to say "it's obvious"). How many people here running the lagfix, have run any kind of blind test, or test with predictable steps to demonstrate a difference? Please, raise your hand people, because there is a huge difference between "it seems lag is gone" and "Lag is verified gone".
Quadrant is a guide, it doesn't test interactivity, and it only tests a few operations. It's similar to how a browser can pass ACID3, but have terrible compliance to new standards. Because people became so convinced of benchmarks accuracy, Nvidia and ATI started optimising for benchmarks.
Normal SSD's are also blazingly fast on benchmarks initially, but if they don't have TRIM, their performance drops significantly. That's another example of something benchmarks don't accurately test (because the testing is incomplete).
Use it to get a general idea of how the phone performs (although, specs might be more useful in some cases), but you probably need profiling and a predictable list of steps to diagnose the lag exactly. Run benchmarks which test according to the types of applications you are planning to run (if you play 3D games for instance, use a 3D game benchmark). But don't rely on them exclusively to tell you how well a device performs, because only running the applications and testing them yourselves can tell you.

LostAlone said:
Its not fair to say that quadrant sucks, and particularly I don't like that the article seems to be implying that the lagfixes that a lot of us are running are simply artificially inflating the SGS's score. The SGS has the best harware on the market crippled by bad firmware, and tbh I think its the lag fix that has allowed the hardware to be reflected in quadrant scores.
Anyway, quadrant doesn't suck. Its not perfect, but it does help you determine for example, how different ROMs on the same device compare for performance, as well as showing off things like GPUs that often don't get seen.
However it just runs a series of tests and weights the results accordingly. Particularly, the I/O seems to be given a lot of weight and so can be nobbled to increase the score. However, as anyone who has experienced excessive lag on their SGS can attest, it also works the other way. Without the lag fix, the SGS is a very pedestrian device, as the ~1000 quadrant scores tell you, while with the fix it FLIES.
Click to expand...
Click to collapse
But do you really think that one-click-lagix or whatever is that much faster than mimocan's one (like 2800 vs 1800)? I don't think so...

andrewluecke said:
How many people here running the lagfix, have run any kind of blind test, or test with predictable steps to demonstrate a difference? Please, raise your hand people, because there is a huge difference between "it seems lag is gone" and "Lag is verified gone".
Click to expand...
Click to collapse
*raisehand*
Predictable steps to demonstrate a difference:
1. Use 2 SGS, firmware of your choice
2. One with, say, 1 Click Lag Fix - the other one stock
3. Now for the test, perform simultaneously:
3.1. Open Market and go to "Downloads", update an app, close.
3.2. Open Contacts, quickly switch to favourites, dial Number. Drop call. Exit.
3.3. Open Messaging, conversation of your choice, quickly switch the writing language, write, send, exit.
3.4. bla bla goanforeva
Now if you don't notice an improvement there and in many other cases you most certainly have done something wrong.
If you are saying people are just being placebo-ed you are implying that everybody else besides you is unable to tell a real difference. Think about it, maybe it's the other way around.

When I got my device it came with the Asian JG4 firmware I think and I used it for 48hrs as a point of reference for this so called "lag" I updated my firmware to the latest we have and found zero difference in terms of performance ( same 2 day usage ) decided to flash my device with JM5 and used it w/o a lag fix for 2 days, notice a very slight improvment on how fast the inbox/contacts can open and then did a fresh flash with lagfix 2.x installed I didnt NOT notice any performance difference but I am always open and welcome to any lagfix that is stable,safe and fast ( in that order ) right now I have JG8 installed w/o lag fix and its very snappy, I still want to install a lag fix for this even if i dont see/feel any performance difference at all.. thats just me.. Now im wondering w/c lag fix is the most stable and safe.. stability/safty > speed
EDIT:
Also I believe it could be possible that certain versions of lagfixes work better with certain versions of firmware also, its all about getting the perfect combination.. altho my question still stands on w/c is the safest lagfix heheheh

Well actually i gave up on benchmarks.
To me the whole user experience is more important. If apps open instantly and the phone runs smooth then it's fine with me. I did notice some firmwares are better then other ones.
Right now i'm running JM1 (rev 3) with CFLagFix1.80 installed which makes the phone very stable and running smoothly so right now i'll stick to this firmware until a proper GPS fix is released or until Froyo is released.

Phandroid said:
*raisehand*
Predictable steps to demonstrate a difference:
1. Use 2 SGS, firmware of your choice
2. One with, say, 1 Click Lag Fix - the other one stock
3. Now for the test, perform simultaneously:
3.1. Open Market and go to "Downloads", update an app, close.
3.2. Open Contacts, quickly switch to favourites, dial Number. Drop call. Exit.
3.3. Open Messaging, conversation of your choice, quickly switch the writing language, write, send, exit.
3.4. bla bla goanforeva
Now if you don't notice an improvement there and in many other cases you most certainly have done something wrong.
If you are saying people are just being placebo-ed you are implying that everybody else besides you is unable to tell a real difference. Think about it, maybe it's the other way around.
Click to expand...
Click to collapse
Lol I think you didn't understand my question... I'm sure that these lag fixes all work, no doubt about that. But which of them is faster? For that i can't rely on quadrant i'm sure...

Quadrant Standard test

Hi everyone,
I have question. My result in Quadrant Standard is something about 800. A lot of people have higher result that's mine. There is any method to do SGS faster and get better results or where is problem in my phone ??

if you want higher quadrant scores you need to do a lagfix. the market has one called one click lag fix but it roots your phone. imo quadrant scores arent indicative of real world phone speed. if it feels fast, it is fast. go to the development section and read up on why this pone lags and all the different lag fixes available. but basically samsung uses a slow file system which causes 'lag' but i guess its all relative to what you are used to.

Everything is ok now. Thanks for your help

[Q] What's up with the performance of froyo on Galaxy S?

Hi Ppl,
I got the official froyo release for Galaxy S India yesterday. Good Good... But wait, is it worth it? I don't know..
Before I upgraded to froyo I had applied the lag fix on 2.1 and the quadrant score showed somewhere at 1700-1800 which is good. Quadrant score on stock showed around 800-900.
Quadrant score on 2.2 stock now shows 925-950. This really pissed me off. I ran another test which is linpack and I got 13 Mflops. I do not understand what's wrong with the froyo update. I read somewhere that 2.2 is not compatible with humming bird processor. Per my understanding 2.2 upgrade should boost the CPU to the next level and the performance should be really good than 2.1.
I was comparing dell streak with galaxy s. I know if I should or not but I just wanted to look at the scores on both. Dell says Streak has a 1 GHz processor and so do Galaxy S. But the linpack score on streak showed 30 Mflops which is way better than Galaxy S which is at 13 Mflops. I really do not understand what the problem here is.
Is there no difference b/w 2.1 and 2.2 on Galaxy S? If yes, 2.2 is a joke I would say.
I am concerned because I still found issues with the froyo update. Its still laggy. I didnt notice much difference b/w 2.1 and 2.2 really
Can anyone help me out with this please?
Thanks,
Ravi

I wouldn't worry too much about quadrant scores, but if your phone is lagging then you probably need to reapply a lag fix. Did you update via Kies? I updated mine a while ago using odin (never had much joy with Kies) and my phone seemed pretty good without a lag fix, I did do a factory reset though before installing. Mines now running spikey's oc rom and oclf and is scoring around 2300 on quadrant, but more importantly the phone is ultra responsive with zero lag

Do a factory reset and reapply a lagfix, you'll get 1600ish quadrant scores again.
Sent from my GT-I9000 using XDA App

Thanks for the reply, guys. I had read somewhere that the score would be 1600-1800 on stock froyo? Is that not right? Do I still need to apply a lag fix?
And yes. I updated using Kies. I did a factory reset of the phone before and after I update froyo.
Which lag fix do u suggest? Voodoo or OCLF? I guess OCLF uses EXT2 and Voodoo uses EXT4 which is a bit faster than the former right?

who cares about the score, it is all BS
what matter most, is how smooth it runs for you, and when you switch from app to app, and have stuff multitasking in the background

AllGamer said:
who cares about the score, it is all BS
what matter most, is how smooth it runs for you, and when you switch from app to app, and have stuff multitasking in the background
Click to expand...
Click to collapse
That FACT should be sticky on this forum.

ok.. I am facing a new problem now. I do not know if this is the right thread or not.
After the froyo official update I noticed I could download apps from the market except for the installation process. Some apps get stuck after a complete download and never gets installed
IS there a fix for this?

Low Quadrant Standard Test Score

Hi guys,
I've just run a Quad Standard test on my N1 running CM7 RC2 and am getting a score of 1199, which is lower than even the stock 2.2. I've tried with SetCPU ON & OFF, but got pretty much the same score?
A while back, I've flashed the RODS MIUI ROM & the WM Kernel (1152MHz) and ran the test only to get a score of 1192.
Wonder what could be wrong??

It's a synthetic benchmark and therefore meaningless. Does the phone run ok, does it feel like there's a problem?
The only thing Quadrant is good at measuring is how well a ROM is optimised for Quadrant

The phone runs absolutely fine. Its not like i have any issues with it, but just wanted to compare how it performs with different ROM's.

karthik247 said:
The phone runs absolutely fine. Its not like i have any issues with it, but just wanted to compare how it performs with different ROM's.
Click to expand...
Click to collapse
Best to break out the stopwatch and perform some real world tests. They're the only ones that matter

The only major difference i noticed after switching to MIUI from CM7 is that booting is super fast. Besides that, everything else felt pretty much the same.
OFF TOPIC
Does anyone know how to get rid of the power widget on the notification bar?

The option is in the CM settings which can be found in the usual settings

I noticed this too when i ran the benchmark, but simply dismissed it because its not the final version of the rom, and just a RC

DirkGently1 said:
The option is in the CM settings which can be found in the usual settings
Click to expand...
Click to collapse
I meant in the MIUI ROM.
Also wanted to know which kernel is the better among Pershoot and WM?
After using the phone with MIUI for a couple of hours now, figured out two issues.
1. When im getting a call, the trackball starts flashing and only after a good 2-3secs does the ringer go off.
2. And while playing music videos, the audio-video sync doesnt seem to be perfect. Its as if the phone is lagging a wee bit. Could this be cuz i only got around 40mb of free space on the device??

Quadrant scores, IMO, mean nothing. They give you a numerical value on the specs and capabilities thattthe phone can do under certain situations but they don't give you any real world numbers.
Yes the scores on my 2.3.3 N1 were "low" as. Well but how do you explain it being just as fast (no exaggeration here) as a stick Atrix? I had both and took the Atrix back.
The Atrix has amazing specs and its hardware is awesome, but as an overall feel I didn't like it, the whole experience didn't feel fluid to me.
All in all, quadrant scores are good for testing stock phones and comparing specs to each other, but you should by no means use that to judge the quality or value of a phone.
That would be like buying a car off of eBay by only comparing specs and stuff written on paper; you ddon't have a feel for the car.
Sent from my Nexus One using XDA Premium App

GingerDX Quadrant benchmark

finally gone from stock rom to gingerdx 014 YAY!
Ran a quadrant advanced benchmark and noticed I dont get anything displayed when it goes through the first two 3D benchmarks aside from the FPS and frame numbers at the bottom. (the corridor one with stairs and I forget what the 2nd one is).. Then I get the DNA strand 3rd bench test which shows up fine and it displays a result of 1124 ???
Is it missing out the display of these for a reason?! Hopefully not to artificially bump up the quadrant score?!? surely not?
Anyone else noticed this?? Or is my gingerdx install a bit fubar?!
Installed from stock 2.1 using CWM after doing a full wipe and dalvik wipe..

Black screen in 3d tests except DNA is normal on all Gingerbread roms(like GDX).
But that bug does not affect on scores.
Don't worry.
PS:
I'm not sure but, The newest version of quadrant from the market can work normal without bug.
Scores will be the same.

Thanks for that.. I was only joking about the scores.. I will download the latest version and try that. I thought I may have done something wrong and wondered why mine wasnt displaying the tests

lancemark said:
PS:
I'm not sure but, The newest version of quadrant from the market can work normal without bug.
Scores will be the same.
Click to expand...
Click to collapse
Right, 3D work with quadrant 1,17 but the score is smaller! I use the advanced version.
With odl quadrand I get a score about 1100. With new quadrant I only get around 950

blue4you said:
Right, 3D work with quadrant 1,17 but the score is smaller! I use the advanced version.
With odl quadrand I get a score about 1100. With new quadrant I only get around 950
Click to expand...
Click to collapse
I've noticed the same thing. I get 1100 in the old one, 892 in the new.