Related
So, a little history...
I'm on my 2nd Captivate (unrelated issue). The boot problems didn't arise until I restored my data onto it. The first time I tried restoring system data along with my apps (which I figured was safe since I was going from 2.1 stock to the same) but that had lots of issues. So I did a factory reset, and started over. This time I just restored apps and their data, and a few specific pieces of system data (contacts, wifi APs, etc). That worked better. But later I got too aggressive with what system apps I "froze" (using Titanium Backup), and it got into boot loops I couldn't break out of. So I did another factory reset.
This time, first I carefully froze only apps I was to confirm were safe to freeze. I didn't proceed restoring my apps until I was done freezing and ensured it was booting fine. I then restored all my apps (and their data). The problem is now that it takes forever to boot... in fact, it'll go into a boot loop if I just leave it be. It seems the only way I can gain access is to try and unlock it before it's done booting, clear any "Force close" errors (sometimes takes a few tries) and given enough attempts, I can get in. But the boot takes an unbelievable amount of time, and even with my original apps is many times longer than before on my previous Captivate. Once I fuss my way in, it seems mostly fine, but something is obviously wrong and I want to get it straightened out.
Aside from doing another factory reset, and reinstalling all my apps (which takes like a day without troubleshooting after each, since batch restores in TB don't seem to work well on the Captivate so I have to do them one by one), I'm hoping there's a way to troubleshoot it in its current state and try to fix the problem surgically versus erasing and starting over. I looked at the logcat logs but got in over my head... there are so many errors and warnings and I don't know what's normal and what isn't... too many to know where to begin with searching Google.
So... advice? What tools are available? Even the logcat doesn't seem to kick in until the boot is mostly done, so I'm not sure if it can catch the problem while it's happening. One frustrating thing about Android is that is seems to have no "safe mode" or other diagnostic boot or full logging where you can methodically look at what's happening and experiment with the config. If this was a Windows, Linux or FreeBSD box I'd be in my element and able to get to the bottom of this, but on Android I feel even more crippled, locked-out and helpless than even on Windows. Urgh.
There's got to be a better way to troubleshoot and fix than endless random factory resets. This is something us anti-Windows people scold PC makers for, with all their use of "Restore CDs" for every minor and trivial software issue.
Thanks!
I can't help but think you are still disabling some essential system apps. Either that or one of your apps is causing major problems. Please list what you have frozen in tibu. Btw, the batch function works fine and is what most people on here use.
Also, what is force closing after you restore your apps.
newter55 said:
I can't help but think you are still disabling some essential system apps. Either that or one of your apps is causing major problems. Please list what you have frozen in tibu. Btw, the batch function works fine and is what most people on here use.
Also, what is force closing after you restore your apps.
Click to expand...
Click to collapse
First of all, thanks for taking a stab at this.
My frozen apps are:
* AllShare
* AT&T FamilyMap
* AT&T Hot Spots
* AT&T Maps
* AT&T Music
* AT&T Navigator
* AT&T Radio
* Daily Briefing
* Days
* Instant Messaging
* Media Hub
* Mini Diary
* Mobile Banking
* Mobile Video
* MobiTV
* Where
* Write and Go
* YPmobile
I also tried removing my MicroSD card, as well as switching back to Touchwiz (from LauncherPro). Didn't help.
Yeah the batch problem in TB is very frustrating. I'm using the pay/donate version so I'm missing out on a feature I paid for. I've been exchanging emails with Joel (the author) and we haven't figured it out yet. Batch backups work fine. Batch uninstalls also work fine (update: phone just spontaneously rebooted after about 50 or so uninstalls in a batch). It's the batch restores that seem to choke it. It's not corrupt backup files... a verify runs fine, and I can individually restore the same handful of apps one by one that will choke and hang/reboot the Captivate if attempted to restore in a batch.
The FC error I get is on boot-up, as I try to unlock the screen prior to the boot finishing. I often see "Process system is not responding".
I have aLogcat installed, if that's any use. A few questions about that:
- What's the best logging level to view on? In other words, do I care about "Warnings"?
- What errors are common, harmless, and safe to ignore?
Currently I've tried uninstalling everything down to just a few core apps. Certainly boots fine now, but I get plenty of warnings and errors in logcat.
Are u restoring just the user installed apps+data, or system apps too? Or restoring system stuff like contacts data, accounts prefs, etc?
diablo009 said:
Are u restoring just the user installed apps+data, or system apps too? Or restoring system stuff like contacts data, accounts prefs, etc?
Click to expand...
Click to collapse
I did not do a "restore system data" or any batch/bulk option in TB that restored all system data. As I recall, the only system data I restored a-la-carte (by selecting the individual item from the TB list) were:
Accounts
Bluetooth pairints
Bookmarks
Calendar
Contacts
Wi-Fi Access Points
These were all items in green in TB. I don't believe I restored anything else. Possibilities I suppose are wallpaper settings, "Country, Launguage, Time Zone"... but I definitely would not have restored anything not green.
Are these items safe? Is there any system data definitely not safe to restore? I have to wonder though, if "system data" is unsafe to even restore to the same stock OS version... why back it up at all?
I'm not a long distance from doing yet another factory reset I suppose, if it must come to that. But I'd love a way a bit more analytical/exacting to try and troubleshoot this other than "reinstall one app, reboot, see what happens" as that will take me a week to get back to where I was. I also suspect it's not just one single app that would suddenly show a huge difference after installing, but instead might be the cumulative errors from several apps and knowing how to identify that and clean them up would be useful.
Using adb logcat you can view what is occurring while the phone is booting and possibly see where it is hanging or what is causing the slow boot times. I have seen problems from restoring data such as accounts and contacts with titanium backup but does not seem that it should be an issue when using the same system though I have very little experience with the stock firmware. I know it is not an answer to your question but it seems that you are wanting to remove all the att/Samsung BS so why not flash a rom that does this as well as much more?
Sent from my ADR6300 using XDA App
Thanks for the tip about adb. I've actually not needed adb for anything yet so I've never set it up or used it. I wasn't aware that the service would be active early enough on the phone's boot process to allow it to log boot logs... nor was I even aware it could do this. I'll definitely check out how to set this up... however, if you have a free second and can point me in the right direction (FAQ, instructions, etc) it'd certainly be appreciated otherwise I'll search around and try to find it.
I could probably find other ways to restore contacts and could set my accounts up again manually but I really doubt that's the cause and the other ways are sort of a pain and imperfect. Since neither of us is really convinced that'd be it I won't bother yet until/unless you really suspect it.
I wondered how long it'd take before someone would suggest a custom ROM, this being XDA and all. Short version is I'm not really sold on the concept, as they are all based on the buggy beta leaked ROM, or 2.2 ROMs from other devices that have been hacked up to sort of work as well as possible on the Captivate. All seem to have issues... enough that I'm not really left feeling confident about them. Seems every release unleashes new issues despite addressing old ones, and all seem to have at least a handful of gremlin items that just don't work quite right. Don't have a warm fuzzy feeling, and I still feel like Samsung is going to release an official 2.2 for the Captivate within the next month or so, so I'm interested to see what comes of that. If nothing else, it'll give a better baseline for custom 2.2 ROMs. Then there's the 2.3 being worked on... now that might be interesting.
I don't really think my issue here is related to me running 2.1.
On my phone so it is a pita to search and add a link for you but search for android sdk and you will find what you need to get adb up and running.
And as far as the rom issue goes..it is your phone and I respect your concerns I just had to ask
I would think its media hub that slows it down. It will search your SD cards on every boot. I would start there first.
smokestack76 said:
I would think its media hub that slows it down. It will search your SD cards on every boot. I would start there first.
Click to expand...
Click to collapse
That's one of the apps I've frozen though.
For me the longest process while booting is the stupid media scanner upon bootup. Takes FOREVER for the phone to finally "boot" all the way up.
Been looking for a way to disable it (not really lol) and only have it scan manually to see if the boot time will improve. I'd start searching there.
Also - from what I've read nothing you did should have affected the phone. BUT - if your using Google for your Calendar and Contacts.. and they all get synced up to Google? Why bother doing the restore for those? After you sign up with the Market they get pulled back down to your phone automagically
It's definitely more than just the media scanner. I watch that. When everything is loaded up, it actually reboots in a loop unless intercept the FC. The media scanner will rerun over and over each time... far more than the standard two times.
And I use Google Calendar for my events, but I keep my contacts locally on my phone.
So what is the FC again?
Yep.. my media scanner will run at least 3 times before it stops checking everything.
Very frustrating that the software does this EVERY time I boot back into my phone - you'd think a programmer would put a check to see if it had run before or make it user configurable to scan when you want it to.
avgjoegeek said:
So what is the FC again?
Click to expand...
Click to collapse
See my second post (reply #3):
"Process system is not responding"
LOL sorry not enough coffee and a lil' guy that decided to wake up at 4a.m. = not a good mix.
Well.. did the ol' wise search of Google and came up with:
Might be a permissions issue. Easiest way to fix it is to run ROM Manager and have it fix permisions.
Run the command yourself in ADB:
Code:
Open terminal and
>su
$mount -a
$fix_permissions -r
The -r is optional, but necessary if you find orphaned apps (the app not found please reinstall message)
And reboot. That may help.
And it was also stated that you might have an errant widget/application causing the issue as well. That will be fun trying to figure out what it is.
And.. from my non-dev/non-professional experience - I have checked the logs on my phone and do see a large number of warnings on the phone - but never hindered performance.
So.. 99% of the time you can probably ignore them.
But back to the FC issue - I would try doing a restore of your apps/data again (I read where you have it back down to the "core) and then run the permission script or have ROM Manager do it for you and see if it returns.
Just an update that I think the "fix permissions" thing solved most (but perhaps not all) of my issues. Thanks so much for the tip. I've been reinstalling apps in batches and it's much better, although I see it getting bogged-down bit by bit and I can't pin down what or why.
Thing is, the apps I'm installing in these later rounds/batches shouldn't be resident all the time, shouldn't be auto-loading, and don't come up in things like Startup Cleaner or Advanced Task Killer. Nor do the various process monitors I've tried seem to have the granularity/ability to catch them while they're happening.
So things are better, but I still have some issues without a suitable means to diagnose. I don't get why just having more apps installed, but not running, should affect boot time so much. Hmm...
An inability to troubleshoot certainly rains on my love-affair with Android... not that I'm jumping to another platform anytime soon, but I really want this to work well (as well as be a good salesman to friends and family who often turn to me to show off quality technology).
So a friend of mine updated his T-Mobile UK Samsung Galaxy S to the official 2.2.1 JPY firmware using Odin a few weeks ago. This went well, and is the same firmware that I've had on my unbranded SGS since December.
However, he has an interesting problem. Some of his apps (Handcent, Dolphin HD, plus a few others) seem to forget all of their settings every time the app is exited. For example, every time he goes back into Dolphin it enters the setup wizard as it it's being ran for the first time. This doesn't happen on every app, and it doesn't happen every time.
I noticed there are a couple of other threads on here with people reporting similar issues on custom ROMs (with no solution), but this is a stock ROM from samsungfirmware.com and certainly doesn't happen on my SGS, so it's not the ROM to blame.
Anyone know of a fix?
This question posted in the last few days .
jje
As I said in my post "I noticed there are a couple of other threads on here with people reporting similar issues ... (with no solution)". I wasn't being lazy and just starting another thread without doing some research first. I've read the other threads and they do not contain a solution that will work for him.
I'm assuming you're referring specifically to this thread? http://forum.xda-developers.com/showthread.php?t=992335 In my friend's case, rooting is not an option, so this solution won't work for him.
However, based on that it looks like a factory reset may fix it. If I don't come back people reading this can assume it worked.
Hello
I experimented the same issue.
Looking at some log traces event it reported some permissions errors on intestinal filesystem like database location access write errors. This can explain setting persistence trouble.
After various workaround, bypass or others kind of fallback I did a full reinstallation of filesystem including a wipe data of the system. Take care to backup application and sensible data SMS, photo...
Since all is working fine and no more problems with settings
I don't know if it's helping you.
This is just my personal experience on this same problem.
Perhaps more simple solution is possible
Regards
Gilles
Thanks. So far so good after a Factory Reset. If it were my phone I'd be looking at Logcat, etc, too, but it's a bit hard to do remotely.
It's odd how some apps saved their settings fine, but some don't.
A factory reset did fix it... for a while. But then other apps started suffering from the problem.
So I've been researching this a bit more today. Here's what I've discovered:
The problem is unique to Samsung Galaxy S phones, and appears to be only ones running 2.2.1 (but maybe 2.2. too?).
Samsung have renamed /data/data to /dbdata/databases, and shared preferences for applications are persisting in this folder even if the application is uninstalled. If you install the app again, the app becomes a different user from the one that owns the shared preferences.
For example - If you install some random application and you look at its process running with the ps command, it will show as "app_XX", where XX is some number. For our example here let's say it shows as "app_55". When it then saves its settings in /dbdata/databases the folder the settings are saved in will have owner (shown by ls -l) as "app_55" too. That's fine, and normal.
The seed of the problem is sewn when you uninstall the app. Those shared preferences are not removed. Even using the "Clear data" option before uninstalling doesn't seem to help. If you then install the app again and look at the output of ps you'll see the new app is "app_56". The shared preferences are still there from last time with owner "app_55", hence the permissions error when it tries to save its settings, as user app_56 cannot modify user app_55's files.
A factory reset obviously cures it as this, at least temporarily, as it wipes the data in /dbdata/databases.
If you're rooted, you can go into /dbdata/databases and delete the relevent folder for the application that's having the problem. It will be recreated with the correct owner next time the app saves its settings.
I didn't try chown to change the owner to the correct one, but maybe that's a way to keep the old settings and correct the problem.
Bingo. No wonder my titanium backup is not remembering the preference settings.
Hey,
Has anyone ever looked into how the Whatsapp Backup is stored? It can be used after reinstalling the app, but not after hard resetting/flashing/etc. So it should be stored on the phone, but not the normal isolated storage of the app since I can reinstall it and use the backup then.
Is there special access necessary or could other apps also access (and modify/export) this backup? I'm asking this because I hope to find a way to make a backup that really is useful, I almost only write via whatsapp because SMS cost way more than internet here. Thanks!
why do you think whatsapp isnt useing isolated storage? shure it is, but you still cant access it - unless you have a interopt unlocked device
the reason you can restore, is that the app id stays the same.. you could try to deploy an older (uncrypted) whatsapp xap to your phone, if the app id still stays the same (and im not shure it does! actually i think it doesnt..) you would gain access... again, thats total theory! i havent tryed this, and it has propably the potential to mess up your backup...
maybe GoodDayToDie has some more info about the app-id thing.. i'll also do a quick research
tfBullet said:
why do you think whatsapp isnt useing isolated storage? shure it is, but you still cant access it - unless you have a interopt unlocked device
the reason you can restore, is that the app id stays the same.. you could try to deploy an older (uncrypted) whatsapp xap to your phone, if the app id still stays the same (and im not shure it does! actually i think it doesnt..) you would gain access... again, thats total theory! i havent tryed this, and it has propably the potential to mess up your backup...
maybe GoodDayToDie has some more info about the app-id thing.. i'll also do a quick research
Click to expand...
Click to collapse
Thanks for the reply! I din't know files in the isolated storage keep being there after you uninstall the app..
I tried deploying an old Version (1.4) of Whatsapp and it replaced the current one, so it should use the same app id. I didn't find anything in the isolated storage, but the backup is still there when I reinstall it from the store. I'll try launching the deployed app first now.
have you checked out the IsolatedStorageSettings?
Let me have a quick look where this thing is on my harddrive... when i find it, i'll be able to tell you where / how it saves the backup
tfBullet said:
have you checked out the IsolatedStorageSettings?
Let me have a quick look where this thing is on my harddrive... when i find it, i'll be able to tell you where / how it saves the backup
Click to expand...
Click to collapse
Nope, I have no real clue how to do that, the only thing I can is deploying apps and watching their isolated storage thanks!!
they used "messages.sdf" & "contacts.sdf" before, but then at some version they started to migrate this files into a database.. not shure where it gets stored, or if you can access it with simply browsing the IsoStoreage... ill make a quick test project to test out how or if we could access it..
EDIT: actually i was talking **** the .sdf files are already databases, and the data still resides there... and forgot to mention: even if you couldn't see the database files, you should see the user-picture thumbnails that reside in "cphotos/" + some-sha1-hashed-userinfo...
IsoStore is cleared when an app is uninstalled. So far as I know, this is instant, as part of the app removal process, although I suppose I haven't actually checked that. However, apps can (and many do) implement a backup situation to cover this use case by using a unique identifier that survives a re-install. There are several places such IDs can come from. Since the one you have survives app installs but *not* OS reflashing (even though you presumably sign on with the same Live ID afterward), I'm guessing it's a value that uniquely identifies your OS install and is randomly generated the first time the OS boots. Re-flashing counts as a new install, I guess.
I'd have to investigate further to be sure. There could be other mechanics at play, such as the OS keeping the data around for a short time in case you re-install the app, or the app storing its data in some other (off-phone) location. It's not storing it in some special folder within the phone, though; there's nowhere else it could!
Backup history with Whatapp on Android. Then check the backup file on:
/sdcard/WhatsApp/Databases/msgstore.db.crypt
Or
/data/data/com.whatsapp/databases/msgstore.db and wa.db (root)
Coweri said:
Backup history with Whatapp on Android. Then check the backup file on:
/sdcard/WhatsApp/Databases/msgstore.db.crypt
Or
/data/data/com.whatsapp/databases/msgstore.db and wa.db (root)
Click to expand...
Click to collapse
Sorry, but this is Windows Phone, not android..
@GoodDayToDie so, there is no simple way like deploying an app with the same ID and trying to access the backup with it?
Since the data would have been deleted when the old app was removed (and since you can't sideload an app with the same Product ID as an existing Store app), no, that won't work (well, it didn't in WP7; I guess you could try again here; some things are somewhat less secure now than before).
th0mas96 said:
GoodDayToDie so, there is no simple way like deploying an app with the same ID and trying to access the backup with it?
Click to expand...
Click to collapse
Wait for a interop-unlock... thats the way to go in this case.
Until then, you can send your conversations to yourself by mail (option form the context menue)
GoodDayToDie said:
IsoStore is cleared when an app is uninstalled. So far as I know, this is instant, as part of the app removal process, although I suppose I haven't actually checked that. However, apps can (and many do) implement a backup situation to cover this use case by using a unique identifier that survives a re-install. There are several places such IDs can come from. Since the one you have survives app installs but *not* OS reflashing (even though you presumably sign on with the same Live ID afterward), I'm guessing it's a value that uniquely identifies your OS install and is randomly generated the first time the OS boots. Re-flashing counts as a new install, I guess.
I'd have to investigate further to be sure. There could be other mechanics at play, such as the OS keeping the data around for a short time in case you re-install the app, or the app storing its data in some other (off-phone) location. It's not storing it in some special folder within the phone, though; there's nowhere else it could!
Click to expand...
Click to collapse
GoodDayToDie, any news about this Whatsapp backup feature in Windows Phone? Is it possible to utilize this feature as an "ordinary" developer?
I would have to reverse engineer the app to see how its backup feature works. The most likely explanation - that it's storing the backup "in the cloud" using the device ID (which resets when you do a hard reset, I think) - is easily possible for any app so long as you provide the storage space...
GoodDayToDie said:
I would have to reverse engineer the app to see how its backup feature works. The most likely explanation - that it's storing the backup "in the cloud" using the device ID (which resets when you do a hard reset, I think) - is easily possible for any app so long as you provide the storage space...
Click to expand...
Click to collapse
That could be an explanation. But then the Whatsapp developers could easily offer a full backup, as on other platforms, linked to the phone number or something. Then you could restore the messages even after a phone exchange. But who knows what's in their heads ...
GoodDayToDie said:
I would have to reverse engineer the app to see how its backup feature works. The most likely explanation - that it's storing the backup "in the cloud" using the device ID (which resets when you do a hard reset, I think) - is easily possible for any app so long as you provide the storage space...
Click to expand...
Click to collapse
Nope, it doesn't seem to be online.. it backups pretty big chats extremely fast with 2 bars of GPRS, so it can't be via internet.. it even backups without connection at all. That's the weird thing.. how are apps even allowed to store files that keep being there after an uninstall?
There are a couple really sneaky ways you could do that; one that comes to mind is creating a fake "image" or "ringtone" or similar, serializing the data to it, and then looking for it the "first" time the app is run after installing. However, I'm definitely more curious now. There are folders which apps can request permissions to write to, but usually that's a trick limited to "second-party" apps (OEMs, etc.) and prohibited for third parties.
I've already looked into the code, as far as i can tell there is no online backup feature. it just stores the conversations in a database.
And to answer yout question schluff: no there is absolutley no way the usual developera to utilize this.
btw: @GoodDayToDie could you provide us the newest WhatsApp XAP (2.11.312.0)?
When I get the chance to extract it of my computer, yeah. It's really hard to get full FS access working these days, so I'm looking into other ways to access the install folders and storage of other Apps.
here you go
edit: I've changed nothing, so it's the whole install folder in this zip file
Thanks for the extraction! Obviously, I can't do anything with it
However, if anyone is interested in this too, in the following versions it seems to backup to the SD card, at least @Nazwzil8 reported so at twitter: https://twitter.com/Nawzil8/status/410486248156172288 he reported a lot about whatsapp, he seems a legit beta tester.
I don't remember having this problem before, it only happens sometimes and with different results, it's very random.
Scenario A) When I open Xposed, it starts checking for updates and notifies me of every new update, if I go to the "Dowloads" activityand press the reload button, nothing new adds. <-- normal scenario, it always was like this before 2.6.1.
Scenario B) When I open Xposed, it starts checking for updates but doesn't find any new update (and doesn't give out any error, at least on a toast), when I enter the "Downloads" activity, and press the reload button, new updates appear..
Scenario C) When I open Xposed, nothing happens. It doesn't load ANY update or tried to get new ones, even if I had some updates pending to install before, it won't show anything until I enter the "Downloads" activity. There the updates to-do are like normal and pressing reload gets new ones.
Also, I noticed that from 2.6.1 is when I started choosing specific update configuration for some of the modules only (global is STABLE and for some apps EXPERIMENTAL now).
This doesn't only happens to modules, today I had an scenario B when after entering the "Downloads" activity and pressing reload, I got notified of the new Xposed experimental1 update..
I didn't do the "all modules disabled" test because since this is very random and I rarely find updates for my mods, It would mean having xposed disabled maybe for hours/days which sucks
Updates from the repository are only refreshed every 24 hours. The downloaded repo.xml is kept in the cache and still needs to be loaded every time the app is started - unless the app was still in the memory.
So the different situations can be explained like this:
A) The last refresh was more than a day ago, so the installer first downloads the new XML, then loads it.
B) The last refresh was less than a day ago, so the installer just loads the cached XML. When you press the refresh button, the 24 hour limit is ignored, so you see new updates.
C) The app might still have been running in the background, so no refresh was necessary. Or maybe you have cleared the cache, so nothing was downloaded and the XML to load was empty.
It isn't really worth looking into this in more detail as 2.7 experimental1 handles downloads quite differently.
rovo89 said:
Updates from the repository are only refreshed every 24 hours. The downloaded repo.xml is kept in the cache and still needs to be loaded every time the app is started - unless the app was still in the memory.
So the different situations can be explained like this:
A) The last refresh was more than a day ago, so the installer first downloads the new XML, then loads it.
B) The last refresh was less than a day ago, so the installer just loads the cached XML. When you press the refresh button, the 24 hour limit is ignored, so you see new updates.
C) The app might still have been running in the background, so no refresh was necessary. Or maybe you have cleared the cache, so nothing was downloaded and the XML to load was empty.
It isn't really worth looking into this in more detail as 2.7 experimental1 handles downloads quite differently.
Click to expand...
Click to collapse
That explains a lot. However, doesn't fully explains the C scenario, since on it, the main activity doesn't show all the "pending to install but already notified (in green letters) updates", it doesn't show anything, not even any module or download on any of the activities (forgot to mention this), hitting the reload button fixes everything tho.
The question is.. why does the system works like this? Why it has to be that "manual"? (I spend the entire day pressing the reload button lol). Why it couldn't just check for updates everytime the app was loaded / everytime the main activity was opened / or even without the app opened, in the background. Maybe bandwith? not sure otherwise
If the system is changing in the next version (and for now in experimental), yeah we'd be losing time checking there. How does the new system works? does it do anything of the ideas I said just above? I'd love to test it, but im not moving from STABLE releases of xposed, it is way important for me :good:
Thanks a lot for your fast reply!!
RusherDude said:
That explains a lot. However, doesn't fully explains the C scenario, since on it, the main activity doesn't show all the "pending to install but already notified (in green letters) updates", it doesn't show anything, not even any module or download on any of the activities (forgot to mention this), hitting the reload button fixes everything tho.
Click to expand...
Click to collapse
That's really strange, not intended and I never had that situation. Do you have an app that constantly clears the cache? Loading the XML of course happens independently from the 24 hour limit, so the only possible explanation is that the cached file was deleted.
RusherDude said:
The question is.. why does the system works like this? Why it has to be that "manual"? (I spend the entire day pressing the reload button lol).
Click to expand...
Click to collapse
As mentioned, THIS is definitely not working as intended. I suspect something is deleting the file in the background on your system, as I never heard of such issues before.
RusherDude said:
Why it couldn't just check for updates everytime the app was loaded / everytime the main activity was opened / or even without the app opened, in the background. Maybe bandwith? not sure otherwise
Click to expand...
Click to collapse
Sure it's bandwidth. The repository index has grown to about 350 kB and is always growing. The Xposed Installer 2.6.1 has been downloaded more than 1.25M times, 2.4.1 even reached more than 2.5M downloads. That surely includes duplicate downloads, bots, users who decided to abandon Xposed etc. But still, the repository index has been downloaded 4.5M times in May, generating more than 1.1 TB of bandwidth. Now imagine what would happen if the installer checked for updates twice daily, or even in the background (which would also generate bandwidth for the many users who don't open the app daily).
Apart from that, traffic caused for the user, waiting times until the file is downloaded etc.
With the database approach and partial updates, I might be able to lower the update rate a bit, but I don't really see a necessity for that. There are about 10-20 updates per day, which affect usally none, sometimes one or two of your installed modules. And even when there are updates, there is hardly any reason that you have to update right now, instead of half a day later.
RusherDude said:
If the system is changing in the next version (and for now in experimental), yeah we'd be losing time checking there. How does the new system works? does it do anything of the ideas I said just above? I'd love to test it, but im not moving from STABLE releases of xposed, it is way important for me :good:
Click to expand...
Click to collapse
Just check the thread, I explained it there. There are no changes in the framework, only in the downloader, so it won't affect your modules.
rovo89 said:
That's really strange, not intended and I never had that situation. Do you have an app that constantly clears the cache? Loading the XML of course happens independently from the 24 hour limit, so the only possible explanation is that the cached file was deleted.
As mentioned, THIS is definitely not working as intended. I suspect something is deleting the file in the background on your system, as I never heard of such issues before.
Click to expand...
Click to collapse
Well, now that I remember the times that happened I was cleaning some so it may be the reason.
rovo89 said:
Sure it's bandwidth. The repository index has grown to about 350 kB and is always growing. The Xposed Installer 2.6.1 has been downloaded more than 1.25M times, 2.4.1 even reached more than 2.5M downloads. That surely includes duplicate downloads, bots, users who decided to abandon Xposed etc. But still, the repository index has been downloaded 4.5M times in May, generating more than 1.1 TB of bandwidth. Now imagine what would happen if the installer checked for updates twice daily, or even in the background (which would also generate bandwidth for the many users who don't open the app daily).
Apart from that, traffic caused for the user, waiting times until the file is downloaded etc.
Click to expand...
Click to collapse
Yeah, I understand that , well at least on BD you're gonna save some bandwith
rovo89 said:
With the database approach and partial updates, I might be able to lower the update rate a bit, but I don't really see a necessity for that. There are about 10-20 updates per day, which affect usally none, sometimes one or two of your installed modules. And even when there are updates, there is hardly any reason that you have to update right now, instead of half a day later.
Just check the thread, I explained it there. There are no changes in the framework, only in the downloader, so it won't affect your modules.
Click to expand...
Click to collapse
Done. Thanks a lot
And about that "there is hardly any reason that you have to update right now, instead of half a day later", there are flashaholics and there are xposaholics :laugh::laugh:
Thanks mate!
Automatically optimizes SQLite databases on boot, on schedule, every X days.
Just a quick and dirty adaptation of an old SQLite3 optimizer script from init.d days to a Magisk Module using a service.sh script instead, with an internal date offset mechanism to run the script every X days (default: 3 - which is plenty, and specifically to stop silliness).
Update And credit note: finally after a fair bit of digging, i think i located the original author of the original init.d version of the main vacuum code snippet ive repurposed (stolen) for this module. It appears to be from user @mcbyte_it (doesnt seem to be current or active) and from a post here. Although i only saw derivative reworkings of this script, and used only the main sqllite script function, i arrived at a different method of tracking last run date, when his is arguably more mainstream/concise. Im fairly certain this is the originator especially as the post also has a zipalign script that was also suggested to me to do....and which i have been fiddling with for months....
The script will wait until boot is completed AND then until avg CPU usage is under 30%, to minimise the risk of possible corruption.
Disclaimer: As always any use of any 3rd party script/software/advice is at the users discretion. All reasonable efforts have been made to make this as safe as possible, but the responsibility ultimately falls to the user whether to use and run the script.
What it does:It:
Reindexes
Vacuums
Analyzes
all .db files under /data.
It runs a 1st run optimize after install (temp file optimsql_first_run on sdcard is used to enable this, and removed after first run), and then on schedule after that.
By default it logs just script progress to /storage/emulated/0/autosqlite.log, but you can choose to enable more detail in the log if you wish
User Configurable Options:The schedule and loglevel can be changed by an external file on sdcard:
Create a file named autosqlite_options on sdcard (/storage/emulated/0/)
Inside create the follow key=value pairs to suit your preference:
interval=x (where x is the number of days between script runs, for the love of god do not put 1 (this goes out especially to the kind of people who put every app on their Magisk magiskhide/deny list) there is no benefit and you just heighten the possibility of corruption)
loglevel=x (where x is either 1 (detailed logging) or 0 (basic logging - default))
Requirements:This module requires a working SQLite3 binary. If your ROM does not provide one (you can check via typing sqlite3 into a terminal), you can choose to use my SQLite3UniversalBinaries module located here:
https://github.com/stylemessiah/SQLite3UniversalBinaries
Dont forget you need to download a named SQLite3UniversalBinaries.vx.x.zip file from the Releases page under Assets. Do not try installing the source code with Magisk Manager, it will not go as you expect
All the modules action takes place in the service.sh file, its commented reasonably well for those wanting to check how it works
* See... now theres something else to use with my SQLite3 Universal Binaries module other than the GPay SQlite Fix Module <- dont use that anymore, it makes me stabby. It is literally the last resort for getting Google Pay/Wallet to work ,
Please note: the included LICENSE only covers the module components provided by the excellent work of @Zackptg5 's Magisk Module Extended, which is available for here for module creators
https://github.com/Zackptg5/MMT-Extended/
All other work is credited above and no one may fork or re-present this module as their own for the purposes of trying to monetize this module or its content without all parties permission. The module comes specifically without an overall license for this intent.
Download:
Repo: https://github.com/stylemessiah/AutomaticSQLiteDBOptimizer
Release: https://github.com/stylemessiah/AutomaticSQLiteDBOptimizer/releases/latest
Powered by Ponkle
@73sydney Nice! If you're looking for newer sqlite3 binaries, I have a build script you can use to build it or you can just grab the precompiled ones at my repo: https://github.com/Zackptg5/Cross-Compiled-Binaries-Android
Zackptg5 said:
@73sydney Nice! If you're looking for newer sqlite3 binaries, I have a build script you can use to build it or you can just grab the precompiled ones at my repo: https://github.com/Zackptg5/Cross-Compiled-Binaries-Android
Click to expand...
Click to collapse
cheers man, i was trying not to pinch all your toys
i mean i could have done an androidacy and scraped your repo for just the files i needed
plus im lazy
73sydney said:
optimizes SQLite databases
Click to expand...
Click to collapse
Nice
how about no schedule, i want to execute it whenever i want
possible ?
loopypalm said:
Nice
how about no schedule, i want to execute it whenever i want
possible ?
Click to expand...
Click to collapse
I refer the gentleman to the name of the module and title of the thread:
"Automatic SQLite DB Optimizer"
the distinct lack of the word "manual" should be a clue
what are they teaching in school these days?
i made this to be automatic and made special note about the intervals because theres people who would abuse this and run it every day if they could, both designed to avoid misuse/abuse:
interval=x (where x is the number of days between script runs, for the love of god do not put 1 (this goes out especially to the kind of people who put every app on their Magisk magiskhide/deny list) there is no benefit and you just heighten the possibility of corruption)
im not really interested in making a manual version option unless you can convince me where there is a use case benefit, because all i see is potential for misuse/abuse that i specifically made this to avoid....
running it by default it runs every 3 days which is twice a week and probably still overkill, but trying to cover all bases
your other option is to ask me for permission (sadly because of abuse in the past from certain entities trying to monetize my and others work, my contributions are not GPL etc) to fork and alter the code yourself....
73sydney said:
I refer the gentleman to the name of the module and title of the thread:
"Automatic SQLite DB Optimizer"
the distinct lack of the word "manual" should be a clue
what are they teaching in school these days?
i made this to be automatic and made special note about the intervals because theres people who would abuse this and run it every day if they could, both designed to avoid misuse/abuse:
interval=x (where x is the number of days between script runs, for the love of god do not put 1 (this goes out especially to the kind of people who put every app on their Magisk magiskhide/deny list) there is no benefit and you just heighten the possibility of corruption)
im not really interested in making a manual version option unless you can convince me where there is a use case benefit, because all i see is potential for misuse/abuse that i specifically made this to avoid....
running it by default it runs every 3 days which is twice a week and probably still overkill, but trying to cover all bases
your other option is to ask me for permission (sadly because of abuse in the past from certain entities trying to monetize my and others work, my contributions are not GPL etc) to fork and alter the code yourself....
Click to expand...
Click to collapse
i was planing to use it once a week
categorizing me with people who do dumb stuff is an 'insult'
i asked a simple question man, don't jump to level 3 mind game please ...
loopypalm said:
i was planing to use it once a week
categorizing me with people who do dumb stuff is an 'insult'
i asked a simple question man, don't jump to level 3 mind game please ...
Click to expand...
Click to collapse
Im sorry, did you miss the double smileys intended to completely avoid any misunderstanding that i was being jovial. the other tip off that i wasnt burning you was taking the time to write several paragraphs
Can i suggest you just set the options file interval to 6 or 7 and let the module do what its described in the title as doing, it would make no functional difference, other than saving you a trip to terminal?
As i said, give me a valid reason for adding a manual option and ill gladly consider it, choosing a reason that fits completely within the actual parameters of the existing module doesnt really fit that offer...
ill go further to why i specifically didnt put in an option to run it manually (really folks are lucky i didnt block the option to run it every day) because you should understand what the sqlite vacuuming and other processes do:
* They copy the data from the db file to temp file before optimising it and writing it back, at any point in this theres the potential for corruption (sudden power loss, other processes accessing the file etc). Ideally any such corruption would be minimised by journaling but anyone who had used Linux for long enough knows not to count on such things.
* If any db is being written to, then the optimization process will NOT optimize that db...for the above possible corruption reasons. Hence why i not only put in a wait till boot completed function BUT also added a function to block the script from running unless avg cpu load over 5 minutes is less than 30%. all this to try and make things as safe as possible. So im not overly fond of a manual option which bypasses these safeties i specifically put there to hopefully safeguard people as best as possible. I dont want to get messages about how my module hosed someones apps.
Perhaps that better addresses my reasons?
As said, you can also ask me to fork the code, and ill even tell you how to add a manual function, but i personally wont be releasing such a version without a better reason than i want to do it every 7 days, when the script as it stands can do that AND at a far more predictably safe time and way than you manually likely can or will use it....
BTw, how is actually responding, even taking the time to do so and being jovial about it "level 3 mind games?", whatever the hell that even is?
73sydney said:
.
Click to expand...
Click to collapse
i was planing to make a recovery backup of "DATA" after the optimization of DB files ...
if you want better result add a comand to kill the coresponding app, then it would be more stable
or tell the people to use FlushRam or something like that before the proccess start, to make sure the result will be good, or make a 2nd module for advanced users ...
even with the condition you said some system apps stays working in the background and messing with their data can lead to corruption
i don't know you or your previous work and i don't have time to care for that
i just asked a simple question
answer with "NO, my module will not do that"
this will save your time and other people time
loopypalm said:
i was planing to make a recovery backup of "DATA" after the optimization of DB files ...
if you want better result add a comand to kill the coresponding app, then it would be more stable
or tell the people to use FlushRam or something like that before the proccess start, to make sure the result will be good, or make a 2nd module for advanced users ...
even with the condition you said some system apps stays working in the background and messing with their data can lead to corruption
i don't know you or your previous work and i don't have time to care for that
i just asked a simple question
answer with "NO, my module will not do that"
this will save your time and other people time
Click to expand...
Click to collapse
1) whether before or after backup, the files will eventually get optimized - if you want a more manual version use SD Maid or as offfered you can ask me how to alter the code as it sits and make your own version.
2) Im not ever releasing a script that kills anyones apps in the background, i would consider that suboptimal, and basically malware
3) I never said that some "system apps stays working in the background and messing with their data can lead to corruption". What i said was i designed the script to best avoid corruption by running only after boot completed and when avg CPU usage over 5 minutes was less than 30%. The sqlite3 itself generally wont allow access to a file in use. Im just adding what i think is a reasonable level of extra safety by deciding when the optimization routine starts to run, and that it isnt abused/misused by running multiple times in an unreasonable timeframe
4) How does
I refer the gentleman to the name of the module and title of the thread:
"Automatic SQLite DB Optimizer"
the distinct lack of the word "manual" should be a clue
not equal "NO, my module will not do that", only with some attempt at humour and a long explanation (soon followed by a more reasoned explanation) because i didnt want to appear dismissive. Yet it looks like you’d have preferred dismissive??
You seem impossible to please anyway someone tried to do it....
To be honest Im tired of this circular conversation. Everyone else knew from the word automatic in the title how things was going to work. Ive made suggestions of alternatives (SD Maid) and even offered to give you the info to change the script for yourself, and cant do more than that.... please choose one and lets move on, please?