Database Develop

UPX'ed applications packed on ROM?

Hi all,
I don't know if 'The Devs' already use it, but UPX (http://upx.sourceforge.net) is free (open source) and can pack binaries so that they take up a smaller space. they get a small overhead to unpack, so they may end up taking a little bit more memory. But in the case of the Blue Angel, it looks like memory is not going to be an issue, right? even though, it looks like the memory overhead isn't that much.
Disadvantages:
* longer load time because of uncompresion
* more memory consumption
Advantages:
* Exes and DLLs take up less space
The UPX devs themselves explicitly say compressing DLL may cause way more consumption, because less resources would be shared. compressing exes does not look that troublesome.
I realized compressing the exes contained in ROM could save up a bit more of space, which would give the devs more space to let other stuff in.
I have experimented with installed apps, as I can't copy/paste the system exes, because (gues what?) they reside in ROM. but it looks like the compresion can be significant.
Maybe I am being silly and the devs already know about it, but I haven't read anywhere about this being used, so I felt like suggesting.
kudos to Helmi and Bepe for the new breakthrough!

afaik they are allready using upx in their roms.

POLL: move PowerPoint/Word/Excel .exe to EXTROM installer?

EDIT: Duh, forgive me, I have put the wrong KB sizes above. correct values are below.
PRO: if you don't use, have never used and will never use one or more of these apps the ROMs can pack more add-on software which won't have to be installed or take up storage space.
CON: If you use them it gets a little bit less convenient, as you have to install it after a hard reset.
NOTE: I don't know how much the system needs those dlls, but I guess they are not deeply embedded in it and might not even need to be installed, unless ActiveSync desperately needs them to convert files back and forth. Which does not make sense to me, at least images, when there is any conversion process, are processed in the PC, as it is much more powerful...
Footprint:
Office dlls:
office.dll: +63KB
officeres.96.dll: +95 KB
officeres.dll: +96 KB
Pocket PowerPoint:
ppt.exe: +2.277 KB
Pocket Excel:
pxl.exe: +852 KB
pxlfile.dll: +34KB
pxl2xls.dll: +47KB
xls2pxl.dll: +78KB
SubTotal: 1.011 KB
Pocket Word:
pwod.exe:+227 KB
Total: 3.769 KB
There could be more Office-related dlls scattered around, but I don't know how to identify them...

I say pack all those files into a self installer CAB and it can be up to the user to include it or not. In theory if we could ever agree on a nice stable base rom to build an online ROM kitchen like the XDA had/has and make everything else optional.

An online ROM kitchen would be sweet!

theloon said:
An online ROM kitchen would be sweet!
Click to expand...
Click to collapse
we'll see on the next couple of days... maybe we'll post it on the forum... BA WM5 rom kitchen... I sure Black6spdZ, xplode, thingonaspring, Midget and others will make it better :wink:
I just need to arange and fix a few stuff, since most people more prefer bepe's ways, for me it just gettin a little complicated to manage it work with bepe's way...lol...

I voted to keep it in the ROM, but only because I'm 99.99% sure that moving them will cause all sorts of problems. Any app launcher or today plugin that integrates with the office apps will break. They'll have hardcoded the path to the /windows directory, I'd put money on that. You might be able to replace the /windows exes with small launchers that point to the storage card I suppose.
I think the compression of the apps holds a lot of potential. Is there a compression tool that decompresses to the RAMdisk and runs from there? That would be the best of both worlds; the file is compressed in the ROM but only needs to be decompressed once on each RAM reset.

fraser said:
I voted to keep it in the ROM, but only because I'm 99.99% sure that moving them will cause all sorts of problems. Any app launcher or today plugin that integrates with the office apps will break. They'll have hardcoded the path to the /windows directory, I'd put money on that. You might be able to replace the /windows exes with small launchers that point to the storage card I suppose.
I think the compression of the apps holds a lot of potential. Is there a compression tool that decompresses to the RAMdisk and runs from there? That would be the best of both worlds; the file is compressed in the ROM but only needs to be decompressed once on each RAM reset.
Click to expand...
Click to collapse
We already have UPXed version of Office apps and dlls working, I believe Helmi will integrate them in the next version.
The 3.769 KB file footprint for the files above becomes around 1.450 KB, that's around 2.2MB of space saving.
For those who don't know:
UPX is a compression method meant to have extremely fast decompression speeds and very low resource comsumption. If I understood correctly, the ARM assembler-optimized UPX decompressor which is added to our dlls and exes is 224 bytes.
From the website, a memcpy on a Pentium 233 (dunno the OS) happened at 60MB/s, while the UPX decompression of the same data was 13MB/s.

fraser said:
I voted to keep it in the ROM, but only because I'm 99.99% sure that moving them will cause all sorts of problems.
Click to expand...
Click to collapse
This is VERY true. Accessing the files would require a launcher as fraser says, and/or accessing any UPX'd resources in the DLL, or EXE without loading it in will cause cataclysmic failures in the calling application. The dangers are VERY high.
Brazilian Joe said:
I think the compression of the apps holds a lot of potential. Is there a compression tool that decompresses to the RAMdisk and runs from there? That would be the best of both worlds; the file is compressed in the ROM but only needs to be decompressed once on each RAM reset.
Click to expand...
Click to collapse
In theory you could do this with the RamDisk now, and use a RAR/ZIP/7Zip style archive on a SD Card (or in ROM). Simply run (on boot) an unarchive of the contents to RAM, and away you go. But in theory this isn't any better (and in reality is worse than) just installing the applications to the RAMDisk in the first place, and/or keeping a second "SD Card" just for the hard-reset cases when you want to re-install everything.
Brazilian Joe said:
We already have UPXed version of Office apps and dlls working, I believe Helmi will integrate them in the next version.
Click to expand...
Click to collapse
Just take care of PIM Managers, and other applications which may make direct usage of these applications. Also, a UPX'd DLL will require longer/slower load times due to decompression overhead as well as the IMGFS decompression requirements itself (even though the IMGFS portion will be 1:1 - i.e. uncompressed).
Brazilian Joe said:
The 3.769 KB file footprint for the files above becomes around 1.450 KB, that's around 2.2MB of space saving.
Click to expand...
Click to collapse
It will unfortunately be MUCH less than that. Try building the ROM with the UPX'd applications and check the free sectors, and then compare that to a ROM with them replaced. The difference in free sectors * 512 bytes per sector is the REAL compression ratio achieved.
My guess is that UPX is about 40-60% better than the IMGFS, so this would mean a gain of around 1.1Mb in the ROM, and not 2.2Mb. But 1.1Mb of useful space is DEFINATELY nothing to be sneezed at! ;-)

Keep in ROM ...
Oh yeah, and I voted to keep the MS Apps in ROM.
Powerpoint could/should be compressed, and possibly Word/Excel if they're still 100% safe (they were in my testing in TuMa v1.3). Compressing any of the core OS though - ICK. I'm not in favour of that.
Save as much as we can, without going crazy ... and keep the Core OS, the Core OS. All the features we want to add should be exactly that - features!

Very Important Note To Increase The Wm5 Spped

I have noticed in all the roms coocked by Bepe Rom Kitchen there is some kind of Problem that decrease the speed of wm5 , it is the rgu and dsm files in the rom under windows folder these are the registry fles that we can exclude them from the rom after they will be saved in the rom these ara about 100 files with 1 or 2 kb size but this decrease the speed of windows folder , the way to exclude them safely coock the rom as usual first start createOS.exe it will generate the TEMP Folder then start the createROM.bat it will build the boot hive and the stopes and ask you to press any button to continue this is the time to delete the rgu and dsm files go to search and find all the rgu and dsm files and be sure to search the hidden files too because most of these files is hidden and when you finish o back to create ROM.bat and finish the rom Building and the first thing you will notice is the speed of opening the windows folder .
Note: you can exclude all the unnessery files that we dont need like the help files *.tm and the note files *.pwi and the word files *.dot etc...
Ahmedkom

That's quite interesting. I'd noted the slow performance in opening "/windows" in Total Commander and I'd often wondered if this impacted the phone. There are way too many files in there, but it was an issue on WM2003 as well and has it's root in desktop Windows. Every time the filesystem has to retrieve a file from there, it has to lookup it's information in a table. While there are ways of writing fast lookups, reducing the input set is often far better. As "/Windows" is quite possibly the busiest directory on the phone, any boost here has to be worthwhile.

Thanks ahmedkom for bringing up this interesting subject, for although there is a tremendous speed gained by mamaich's virtual page pool the Hima beats BA 3:1 when it comes to openning the /Windows dir.
This is true even if the Hima ROM compared to was prepared using Bepe's ROM cook tools. But, then again as you mentioned the BA .dsm & .rgu files are a lot more in numbers compard to the the Hima ROM.
Would you mind sharing how much speed improvement in number of wait cursor 360 degree turns was gained by eliminating those reg & .dsm files? My almost fully loaded BA displays /Windows dir & files after 4 wait cursor turns as against 1.5 turns of a similaryly loaded Hima (using Resco Explorer as File Mgr.).
Cheers!

sure, imagine how fast it's going to be if you delete all files from /windows !!
Honestly, when a 400Mhz CPU is not enough to display a listing of 100 files, something is veeeery wrong. The only reason i put up with with this, is that there's no alternatives with the big screen and gsm phone.
Anyhow, I have latest Helmi build and in calendar just going from day to day, back and forth within the same week is slow noticeably slow. It's obviously re-reading big chunks of data from flash every time, no matter how many times you go between Thur and Fri.
So, given that calendar app is so obtuse, it would be nice to have a file / disk cache on OS level -- which i believe Helmi ROM already has but somehow it aint working...

yeah i just realised Helmi's RAMdisk is just a partition for temp files in ram...

UPX - Compression of exe for better speed - Base for cooker

Hi all,
After reading some news on UPX, a great tools that can compress AMR file, I have decide to investigate this.
So I download the version with UCL include (that my ubuntu for that )
I have try several compression and replace over the rom file just to see the difference.
And of course, it's really better when it work on apps.
They compress by ration of 50% to 70% apps, so Opera for example run really more faster now with that, also Album is really more faster and of course the famous AudioManager_Eng !!!
It load music (even with 4G of music with lot's of different artist and album) in around 1 or 2 sec, before they take nearly 10s to show the playlist !
So technical they pack the exe, optimize it for size and speed.
I want to introduce a list of successfull tools that work really better with this tools.
Thanks to that, I really hope cooker will use it to cook already compressed file into they rom.
Ok, Second post will be all apps that work well
and Thirst post all one that is useless to do it because size is more important after or program stop working and has some bugs.
Let's go
Edit :
Seems NRV compression better, so I will test it more and report

Second post for working apps
Here list of working apps :
I use UPX-UCL with --best option : upx --best *.exe (on linux, should work on windows and other)
See the site : http://upx.sourceforge.net/
Opera : 2.7M (compressed size), 6.3 (unpack) : really nice feeling, page appear more faster, running apps is shorter, wonderfull !
AlbumSearcher (for thumbmail)
AudioManager : awesome the new speed, it open list in 1 or 2 seconds for 4G of music !
AdobeReaderLE
Camera
GoogleMap (already use it in fact, the GoogleMap_Compressed use UPX)
HTCAlbum : speed is great ! so as other tools but wow !
RSSHub : nearly the same, hard to compare
StreamingPlayer : run faster, great
Kinoma : already use it but not with --best option (so I have unpack and repack with best, take 1M instead of 1,7, run faster )
WkTask : hard to now, but work well !

List of non working apps
Here the list of non working or useless compression apps :
CorePlayer : work but take more place (1,7 instead of 1,6), so useless to use it
TotalCommander : .NET not support yes, so any other .NET apps it's useless to for moment
DiamondBackgroundChanger : don't work
Manilla : don't work, TF3D won't work after, so don't compress it

So I'm waiting for brave people to help me grow up this list, this free gpl tool is really nice to be known !
don't forget, use it with UPX --best option

http://mobile-sg.com/software/?p=UPX4PPC&platform=ppc
is more easy

[REF][TUT] Improve program launch times - UPX Squeezer BATCH
Why start a new discussion? Sir B. allready covered that one
http://forum.xda-developers.com/showthread.php?t=481880

kwbr said:
Why start a new discussion? Sir B. allready covered that one
http://forum.xda-developers.com/showthread.php?t=481880
Click to expand...
Click to collapse
Oh sorry, I see, thanks for all links
I do that because it seems no cooker use it ... I was thinking that's was a new think.
my fault to lake of searching
let's continue on other thread

Oh, I've try UPX --brute mode with NRV compression library
For opera : 2,0 m instead of 6,7 !!!
So far better than --best and ucl
Work really well by the way

hi bro thanks for this work, when u compreessed opera u might see that ur program memory might be higher than normal, if you can cha=eck this for me, it happen in previous roms i try the compression on opera

The compression won't make the actual program speed any faster (like opera loading pages), that's just placebo effect. Why? Because when the compressed program is loaded, it will be uncompressed to the ram, and will run and act excatly like the uncompressed program.
"Running a compressed executable essentially unpacks the original executable code, then transfers control to it. The effect is the same as if the original uncompressed executable had been run, so compressed and uncompressed executables are indistinguishable to the casual user."
http://en.wikipedia.org/wiki/Executable_compression
However, the initial program loading will be noticeable faster for bigger exe files, naturally because of the slow flash memory.

duttythroy said:
hi bro thanks for this work, when u compreessed opera u might see that ur program memory might be higher than normal, if you can cha=eck this for me, it happen in previous roms i try the compression on opera
Click to expand...
Click to collapse
Hi dutty
Opera take 1,9 m compressed other upx --brute mode !
Ok, when I launch, with or without compression, take 45k ok memory
So, upx tools can accelerate their launch time,
and compress all dll could also speed up global windows, because launch time of them will be shorter so, running tools with be faster
inside tools I think they load external dll, and so if their launch time is shorter, app could be faster
it's a theory

Vinski- said:
The compression won't make the actual program speed any faster (like opera loading pages), that's just placebo effect. Why? Because when the compressed program is loaded, it will be uncompressed to the ram, and will run and act excatly like the uncompressed program.
"Running a compressed executable essentially unpacks the original executable code, then transfers control to it. The effect is the same as if the original uncompressed executable had been run, so compressed and uncompressed executables are indistinguishable to the casual user."
http://en.wikipedia.org/wiki/Executable_compression
However, the initial program loading will be noticeable faster for bigger exe files, naturally because of the slow flash memory.
Click to expand...
Click to collapse
Ok I'm partially agree
any tools will take less time to launch, thanks to their smaller size.
Memory could be nearly the same, of course
but as I say before, if we compress all exe and all dll (that can be compressed), I really think apps could be faster, why ?
Imagine, tools need 50 dll to be launch, not necessary at launch time but during certain function. So, you clic on a function, start loading lots of dll to do it, if global launch time of them is shorter, app will be faster.
Ok for opera and render engine, but for other case ...
what do you think on it ?

I agree with u, if some function needs disc access for loading dll's, compression would help. Definatetly its worth compressing some bigger files, like opera, I have done that. But for smaller files ~100kb this might not bring any real advantage, bc I believe this also brings overhead for the cpu. It needs to unpack these files on the fly.

Great Chefs please consider adding RAMDISK to your custom ROM

since blackstone has sufficient ram, it would be highly efficient to install a ramdisk and relocation all tmp file to the ramdisk. At least it will make web browsing and other related applications to start a lot faster. We can also use it to store some tmp files. A 8M ramdisk is good enough for general use and I believe a lot of program can make benefit from it. Thanks.

Uhm, Since the device uses ALL flash memory, wouldn't using a RAMdisk be pointless?

snootch said:
Uhm, Since the device uses ALL flash memory, wouldn't using a RAMdisk be pointless?
Click to expand...
Click to collapse
No, the RAM is a lot faster than the flash memory ...
I had the same idea a while ago, but figured out that it is quite useless. Using a ramdisk for browsers will make it faster on the one hand, but on the other hand you loose the possibility to cache content, so you'll have to download all data again and again, which means that you have to pay for it again and again .
Beside the browsers I don't see any usefull application, which uses temporary files (and the temporary files are slowing down the application).
So, what exactly do you have in mind?

johnpatcher said:
No, the RAM is a lot faster than the flash memory ...
I had the same idea a while ago, but figured out that it is quite useless. Using a ramdisk for browsers will make it faster on the one hand, but on the other hand you loose the possibility to cache content, so you'll have to download all data again and again, which means that you have to pay for it again and again .
Beside the browsers I don't see any usefull application, which uses temporary files (and the temporary files are slowing down the application).
So, what exactly do you have in mind?
Click to expand...
Click to collapse
I think a ramdisk can be used for a lot more than just browsers. It can be used as a cache for any program that needs to cache commonly used data (such as the theme files in G-Alarm) If it could be used as a cache for the .NET CF VM, that would be even better as it takes ~5-10 seconds for the VM to start up when you start an application built using .NET CF. P.S.: Were you thinking of having the chefs include this program? http://www.amv007.narod.ru/en/index.html

for the best performances, i would suggest to put the swapfile onto the ramdisk....do you agree?

This is something i was first asking when i joined. onoklog early roms for blackstone did have it but now ha stopped. it is good for dump tempory files and does have a feature in it that supports caching files so certain dumps are not lost on reset. Would be happy to see this intergrated into new roms.

install it by yourself.not veryone need it

mcdull said:
since blackstone has sufficient ram, it would be highly efficient to install a ramdisk and relocation all tmp file to the ramdisk. At least it will make web browsing and other related applications to start a lot faster. We can also use it to store some tmp files. A 8M ramdisk is good enough for general use and I believe a lot of program can make benefit from it. Thanks.
Click to expand...
Click to collapse
is there really that much difference in the access time between main storage and memory ?
for disks its milli seconds vs nano seconds so ram disks make sense

I can't tell how fast the RAM is .. but the nand is extremely slow...

web browsing does benefit noticably when u have opera, etc. cache writing to RAMdisk instead of "main" (nand) memory. also works great as a place for temp. storage (ie. all those files that usually get written to "Volatile" folder). that's about it though...
I don't think it needs to be cooked in either - self-installation is easy enough anyways