Database Develop

System Image and System Partiton

On M TWRP now shows a system partition and system image. Both take up approximately 2gb of storage. What purpose does this serve? Before M I just had a system partition.

If you're talking about the backup options in TWRP, it's nothing to do with M.. TWRP added the option to make a dump of the stock rom into a system image instead of a back up image (for fastboot flashing) The option is there regardless of what rom you are on.
The option is pointless for nexus users.

danarama said:
If you're talking about the backup options in TWRP, it's nothing to do with M.. TWRP added the option to make a dump of the stock rom into a system image instead of a back up image (for fastboot flashing) The option is there regardless of what rom you are on.
The option is pointless for nexus users.
Click to expand...
Click to collapse
Ah I see. Should have read TWRP's release notes. So in most cases you would chose one or the other, probably just a system partition backup? Choosing both would just inflate the backup without any addition benefits.

hecksagon said:
Ah I see. Should have read TWRP's release notes. So in most cases you would chose one or the other, probably just a system partition backup? Choosing both would just inflate the backup without any addition benefits.
Click to expand...
Click to collapse
A normal backup would be system and not system image.
System image would be used once before you flashed your first custom rom, so you could create a flashable stock image.. But we would never need that on this device.. The option is superfluous.

Are there 2 copies of /system on the phone?

My phone (3T) has an unlocked bootloader, is encrypted, not rooted, and running stock OOS 5.0.
I flashed TWRP and discovered that stock OOS restores the stock recovery in boot.
I saw the Oreo dm-verity thread by xenet, had a look at the zip file, noticed that it just modified fstab to prevent force encrypt, so I flashed it to see what happens.
And nothing happens. After the system had booted, fstab is unchanged from the original stock copy.
So I'm wondering whether this file is also restored when booting up on stock.
I get aggressive and go back to TWRP and delete /system/etc and /system/bin and modify build.prop.
Surely now the phone won't boot!
Wrong! It boots up and everything is back to normal in /system.
I go back to TWRP and have a look at /system and it shows me one without the etc and bin folders and has the modified build.prop.
What's going on? How can I see one version of /system in TWRP but a different version (ie, stock) when the phone has booted?
By the way I've been an Android user for many years and have rooted and flashed custom ROMs on a variety of phones and I've never seen anything like what's happening on my 3T. I'm sure that dm-verity is somehow involved in this.

Happened to me on my earlier OOS 5.0 attempts...
But i suspected Magisk is involved in my case.
I downloaded Magisk Module "System Terminal Debloater,"
remove some apps like Duo, Chrome, and Google Play Movies.
Some restarts, they magically re-appear again on Apps Drawer...
Haven't touch them yet again after....

nicknacknuke said:
Happened to me on my earlier OOS 5.0 attempts...
But i suspected Magisk is involved in my case.
I downloaded Magisk Module "System Terminal Debloater,"
remove some apps like Duo, Chrome, and Google Play Movies.
Some restarts, they magically re-appear again on Apps Drawer...
Haven't touch them yet again after....
Click to expand...
Click to collapse
Thanks.
I should have mentioned that I'm also not rooted. So stock OOS 5.0.
Sent from my OnePlus 3T using XDA Labs

When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?

Jackhass said:
When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?
Click to expand...
Click to collapse
No, I don't get that message because my phone is encrypted with a password. So the first thing I see in TWRP is the request for the password and then I'm presented with the menus.
However, in the Mounted menu, system isn't mounted and I have the option of mounting it in read-only mode.
Sent from my OnePlus 3T using XDA Labs

BillGoss said:
No, I don't get that message because my phone is encrypted with a password. So the first thing I see in TWRP is the request for the password and then I'm presented with the menus.
However, in the Mounted menu, system isn't mounted and I have the option of mounting it in read-only mode.
Click to expand...
Click to collapse
After first time flashing TWRP a folder gets created on your internal storage, with a hidden file called .twrps, go delete it and reboot recovery to trigger the message "allowing system modifications" on TWRP's first boot...
It's not about encryption, it's just that TWRP remember the decision you made due to the file I pointed out...
Sent from my OnePlus 3T using XDA Labs

Sam Nakamura said:
After first time flashing TWRP a folder gets created on your internal storage, with a hidden file called .twrps, go delete it and reboot recovery to trigger the message "allowing system modifications" on TWRP's first boot...
It's not about encryption, it's just that TWRP remember the decision you made due to the file I pointed out...
Click to expand...
Click to collapse
Somehow the attachment strikes on previous post
Edit: still not working, check your TWRP Folder on storage to find the file
Sent from my OnePlus 3T using XDA Labs

Sam Nakamura said:
Somehow the attachment strikes on previous post
Edit: still not working, check your TWRP Folder on storage to find the file
Click to expand...
Click to collapse
Thanks, you are correct. I'd forgotten that that TWRP remembers. Deleting .twrps does bring up the RO prompt after decrypting storage.
Jackhass said:
When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?
Click to expand...
Click to collapse
I had allowed changes to the system otherwise I couldn't have made changes to it, which includes the ability to restore the system partition.
But I'm still unclear why if I make changes to the system partition and boot with the stock kernel, then after the boot none of the changes are present in the system partition, but if I boot back into TWRP then the changes are all there.

I recall someone in another OOS 5 thread saying that the stock kernal replaces TWRP with stock recovery if you don't flash root (magisk/superSU). Is it possible that the kernel re-flashes system on boot? Another possibility is that TWRP thinks it's making changes to system but it's not actually? Not quite sure, I've never heard of anything like this before either, just throwing other ideas out there.
I've never read anything about the OP3T or any oneplus phones for that matter having A/B system partitions like the pixels. *shrug*

@nhshah7, something's like what you suggest must be going on to account for what I'm seeing. I'm hoping that someone can confirm my observations and provide a definite answer.

@BillGoss
My thread has been updated relating to all your queries...
Thank you...
https://forum.xda-developers.com/oneplus-3t/how-to/disable-dm-verity-force-encryption-op3t-t3688748

Xennet said:
@BillGoss
My thread has been updated relating to all your queries...
Thank you...
https://forum.xda-developers.com/oneplus-3t/how-to/disable-dm-verity-force-encryption-op3t-t3688748
Click to expand...
Click to collapse
Actually it doesn't explain how TWRP can make changes to system yet the phone boots up on an unmodified system if using the stock kernel. And then, when you boot back into TWRP and look at system, the changes are still there.
Where does the unmodified system come from?
Where does the modified system live?
Why doesn't modifying system result in a failed boot due to dm-verity, while restoring a backup of system does result in a failed boot?
So many questions with no answers.

BillGoss said:
....So many questions with no answers.
Click to expand...
Click to collapse
Not sure if this is applicable in your case but the following possibilities may be worth considering for you:
1. Are you sure that the system image is actually getting modified? If the system partition is not mounted before flashing the zip and the zip being flashed does not mount the system partition in read / write, then no changes to system partitions will actually be written.
2. If dm-verity is enabled, then restoring system could result in an error as this is different from restoring a system-image (nandroid copy of the whole partition and not just the files in the system partition). DM-verity can be triggered if the files are all the same but the dm-verity signature computed by hashing the system partition has changed.
3. For boot partitions, strange behaviour can occur if remnants of the previous boot.img are still in the partition (...e.g. if the previous boot.img was of larger size and a new boot.img of a smaller is flashed, then there will be some bytes after the new boot.img that are from the previous boot.img). To verify this, format the boot partition from fastboot and see if you notice anything different with the new boot.img.
4. In Oreo / 8.0, dm-verity flags are stored in dtb (device tree blobs) inside the kernel and not in the fstab file. Only data encryption can be changed from the fstab file and dm-verity needs to be changed from changing the dtb (...Magisk beta v1456 and SuperSu 2.82 SR4 do this, I think).

rk2612 said:
Not sure if this is applicable in your case but the following possibilities may be worth considering for you:
1. Are you sure that the system image is actually getting modified? If the system partition is not mounted before flashing the zip and the zip being flashed does not mount the system partition in read / write, then no changes to system partitions will actually be written.
2. If dm-verity is enabled, then restoring system could result in an error as this is different from restoring a system-image (nandroid copy of the whole partition and not just the files in the system partition). DM-verity can be triggered if the files are all the same but the dm-verity signature computed by hashing the system partition has changed.
3. For boot partitions, strange behaviour can occur if remnants of the previous boot.img are still in the partition (...e.g. if the previous boot.img was of larger size and a new boot.img of a smaller is flashed, then there will be some bytes after the new boot.img that are from the previous boot.img). To verify this, format the boot partition from fastboot and see if you notice anything different with the new boot.img.
4. In Oreo / 8.0, dm-verity flags are stored in dtb (device tree blobs) inside the kernel and not in the fstab file. Only data encryption can be changed from the fstab file and dm-verity needs to be changed from changing the dtb (...Magisk beta v1456 and SuperSu 2.82 SR4 do this, I think).
Click to expand...
Click to collapse
I'll come back to 1.
2. That makes sense and accounts for why a restore of the system partition with the stock boot image causes me to get dumped back in fastboot mode. If I flash the stock system zip file then the system boots properly.
3. I've not had any issues with strange boot behaviour. I'm always starting with stock or flashing kernels that modify the stock boot image, like Blu Spark.
4. I gathered this from my reading of various threads. If I want to make changes to the system partition and get them to stick and not fail dm-verity then I have to flash a custom kernel. I've proven this in my testing. (A rooting solution would also work, but I've not done this).
Back to 1:
Here's what I've done:
Starting with pure stock image (flash OOS 5.0).
Boot into fastboot and flash TWRP.
Boot into recovery.
Mount system as rw. (In ro mode the next step fails)
Delete the bin, etc, and lib folders in system using the TWRP file manager. (Screenshot a)
Reboot system.
... First interesting fact ...
System boots ok, deleted folders are present in file manager. (Screenshot b)
Boot into fastboot and flash TWRP. (Booting with stock restores stock recovery)
Mount system.
... Second interesting fact ...
TWRP file manager shows that deleted folders are missing. (Screenshot c)
Flash custom kernel or patched boot image
Reboot system
... Third interesting fact ...
System fails to boot. Hangs on splash screen.
So TWRP made the changes (otherwise how could they be visible between reboots, including a replacement of recovery) and I only did them once.
Yet they don't actually take effect until I replace the stock boot image.
So, where are the changes hiding? What did TWRP actually change?
Screenshots (note that TWRP has the wrong timezone set so the time shown is wrong):

BillGoss said:
....
Back to 1:
Here's what I've done:
Starting with pure stock image (flash OOS 5.0).
Boot into fastboot and flash TWRP.
Boot into recovery.
Mount system as rw. (In ro mode the next step fails)
Delete the bin, etc, and lib folders in system using the TWRP file manager. (Screenshot a)
Reboot system.
... First interesting fact ...
System boots ok, deleted folders are present in file manager. (Screenshot b)
Boot into fastboot and flash TWRP. (Booting with stock restores stock recovery)
Mount system.
... Second interesting fact ...
TWRP file manager shows that deleted folders are missing. (Screenshot c)
Flash custom kernel or patched boot image
Reboot system
... Third interesting fact ...
System fails to boot. Hangs on splash screen.
So TWRP made the changes (otherwise how could they be visible between reboots, including a replacement of recovery) and I only did them once.
Yet they don't actually take effect until I replace the stock boot image.
So, where are the changes hiding? What did TWRP actually change?
Screenshots (note that TWRP has the wrong timezone set so the time shown is wrong):
Click to expand...
Click to collapse
Some more thoughts for you to consider:
1. Have you tried this with the official TWRP recovery version 3.2.0-0?
2. Is there anything inside the folders that you see using the file manager after a regular boot? Folders of same name may exist in the boot ramdisk and these are merged with system folders after boot.
3. Try wiping cache between reboots and see if that changes any of your observations.

rk2612 said:
Some more thoughts for you to consider:
1. Have you tried this with the official TWRP recovery version 3.2.0-0?
2. Is there anything inside the folders that you see using the file manager after a regular boot? Folders of same name may exist in the boot ramdisk and these are merged with system folders after boot.
3. Try wiping cache between reboots and see if that changes any of your observations.
Click to expand...
Click to collapse
Good questions. They got me thinking more about how this could possibly work.
I had a look at the cache and there's definitely no copy of the system hiding there.
I also unpacked the ramdisk in the boot image and it had nothing in system. Furthermore, the boot position is only 64 MB, no where near enough to hold the system.
Then I installed Magisk so that I could browse around the phone's partitions and take copies.
I learnt two things from this:
1. If there's a second copy of the system there are only three partitions large enough to hold it (/proc/partitions shows the sizes in 1 kB blocks). The system is about 1 GB. There is space in the system partition (sde20) for 3 GB. There's also space in the data partition (sca15). And there's space in the major partition holding the modems (sdf).
I could eliminate the data partition by formatting it but restoring the internal storage (sdcard) is such a a pain.
So I'll just accept that there is space for a copy, but I'm unlikely to find out exactly where.
2. When I had Magisk installed installed and the system boot, I added a folder and file to /system/priv-app using a file manager (so not using TWRP). I then booted into recovery, flashed the stock boot image, and rebooted. I was expecting it to fail dm-verity (modified system) but it didn't. After booting up there's no evidence of the folder I added to priv-app.
And if I restore the Magisk boot image then the additions show up again.
I'm actually very impressed with how the stock system (kernel, recovery, system) protects itself from modification. Very cool!

Galaxy S8 boot loop after trying to install an older version of Magisk

First, I should point out that I'm just a user, not a developer or hacker. I'm posting this because I don't know where else to ask for help.
This will most likely look like the post of an idiot. But, oh, well, I did manage to mess up my phone, which means I am an idiot. So here it goes: I had 16.0 installed and I tried to install 14.0 from TWRP recovery mode. Basically, Magisk hide wasn't working very well so I wanted to see if it worked with an older version. Now my phone is stuck in a boot loop. It just shows "Samsung Galaxy S8 powered by Android" then restarts. I can boot into TWRP and connect the phone to the PC to access the storage this way, but I can't boot into the OS. Is there any way I can fix this, preferably without losing my data?
I tried flashing the official uninstaller, but I get the message "Update process ended with ERROR: 1"
Here's what the log says about the uninstaller:
Mounting /system, /vendor
-Device platform: arm64
cp: can't stat '/dev/tmp/install/arm64/.': No such file or directory
! Cannot find /data/adb/magisk
Updater process ended with ERROR: 1
I:Install took 1 second(s).
Error installing zip file '/sdcard/Download/Magisk-uninstaller-20180429.zip'
Updating partition details...
I: Data backup size is 32943MB, free: 2620MB.
I:Unable to mount '/usb-otg'
I:Actual block device: '', current file system: 'vfat'
...done
I:Set page: 'flash_done'
I: operation_end - status=1
I looked at the contents of the zip on my computer, and it doesn't have an arm64 folder. So the uninstaller doesn't work on arm64 phones?
Update: I managed to uninstall Magisk using a compatible uninstaller, but the phone still doesn't boot into the OS.
Update 2: if I have to flash the stock ROM, I'm planning to do the next thing: back up my data partition to an external hard drive with TWRP. After I flash the ROM and wipe everything, I want to reinstall TWRP and restore that backup. Would this cause any issues?

Yes, there's an issue on 64-bit devices with the latest available uninstaller. Use the next to last, here: https://github.com/topjohnwu/Magisk/releases/download/v16.3/Magisk-uninstaller-20180328.zip

Thanks. I've just left home and I'll be away for two days. I didn't take the S8 with me, so I won't be able to try any more fixes until Sunday. But I'll check then and update my post if there are any more issues.

Didgeridoohan said:
Yes, there's an issue on 64-bit devices with the latest available uninstaller. Use the next to last, here:
Click to expand...
Click to collapse
I managed to uninstall Magisk with that zip, but the phone still doesn't boot into Android.

vladone97 said:
I managed to uninstall Magisk with that zip, but the phone still doesn't boot into Android.
Click to expand...
Click to collapse
In that case, restoring the stock boot image manually might work. If not, a complete restoration of your device's stock setup might be necessary.
Your device's forum is likely the best place to get further help.

Didgeridoohan said:
In that case, restoring the stock boot image manually might work. If not, a complete restoration of your device's stock setup might be necessary.
Your device's forum is likely the best place to get further help.
Click to expand...
Click to collapse
I tried flashing the stock boot image and it still doesn't work. Stock restoration it is then.

Didgeridoohan said:
In that case, restoring the stock boot image manually might work. If not, a complete restoration of your device's stock setup might be necessary.
Your device's forum is likely the best place to get further help.
Click to expand...
Click to collapse
One more question, though: I backed up my data partition to an external hard drive with TWRP. After I flash the stock ROM and wipe my data, I want to reinstall TWRP and restore that backup. Would this work or would it cause problems?

vladone97 said:
One more question, though: I backed up my data partition to an external hard drive with TWRP. After I flash the stock ROM and wipe my data, I want to reinstall TWRP and restore that backup. Would this work or would it cause problems?
Click to expand...
Click to collapse
If you get your device up and running after the wipe and reinstall, you could always try restoring the backup. If it works, it works, and if it doesn't you can just wipe /data again.

Didgeridoohan said:
If you get your device up and running after the wipe and reinstall, you could always try restoring the backup. If it works, it works, and if it doesn't you can just wipe /data again.
Click to expand...
Click to collapse
Is there any other partition I should back up? That is, aside from the internal storage, which I've already taken care of.

vladone97 said:
Is there any other partition I should back up? That is, aside from the internal storage, which I've already taken care of.
Click to expand...
Click to collapse
Not if you're flashing back to stock anyways. Sounds like something is borked anyways so restoring anything might cause problems (including data). I would just start over. It's just data. Apps can be re installed. Contacts re sync.
I would start fresh, make sure it boots, make a full backup, then try restoring only data. Remember that /data may contain some of your old magisk stuff that may have added to the problem. So do that at your own risk. What data could be worth all that hassle?
Try to think of it like, if that phone got smashed, and you got a free replacement tomorrow. You'd survive right? Even without that data?
And definitely don't try to flash a magisk version that old over one that new one again.

vladone97 said:
Is there any other partition I should back up? That is, aside from the internal storage, which I've already taken care of.
Click to expand...
Click to collapse
Good advice by @madbat99.
Personally I only ever keep TWRP backups of the devices I test stuff on. But then I do a full backup of the device to have an easy way to get back to the setup I want.
On my everyday device I only keep a backup of the internal storage and messages (Signal). It's so easy to restore apps and other stuff anyway, so I don't really bother...

Didgeridoohan said:
Good advice by @madbat99.
Personally I only ever keep TWRP backups of the devices I test stuff on. But then I do a full backup of the device to have an easy way to get back to the setup I want.
On my everyday device I only keep a backup of the internal storage and messages (Signal). It's so easy to restore apps and other stuff anyway, so I don't really bother...
Click to expand...
Click to collapse
I did all that, but Magisk 19.1 still doesn't work. Some older versions work though. I tried Magisk 18.0, 18.0, 19.0 and 19.1. Only 18.0 and 18.1 worked. So I'm assuming there's a problem with the newer versions of Magisk?

Restoring system from TWRP image?

This might be a stupid question, but if I make a backup of my device in TWRP, I have an option for "system" and "system image" (and also vendor and vendor image) when I select which partitions to back up.
So if I for example make some modifications to /system and want to revert back to the state of my recent backup, which one should I select for restoring: "system" or "system image".
I have read that the "system image" keeps dm-verity intact, but does it matter if I already have an unlocked bootloader and a (magisk) rooted device? So will my phone boot just fine if I choose to restore only the system partition on TWRP?

vofferz said:
So will my phone boot just fine if I choose to restore only the system partition on TWRP?
Click to expand...
Click to collapse
System is where the action is. Restoring system is what you need to do.
But when I do a backup I back up everything and when I restore I resotre everything other than data, not sure if that makes a difference or not, but works for me.

Well, I was actually referring to the difference between "system" and "system image" when you make a backup in TWRP. But I guess it doesn't matter that much as Magisk removes dm-verity check anyway.

With TWRP + Backups Working Now - Which partitions make sense to backup regularly?

Until now, no my rooted Samsung Note 3 I have automatic TWRP backups running every night. This just mirrors my paranoia that I will manage to screw up something and recovery via TWRP restore is often faster/easier/more complete than trying to do anything else. I do this using the OpenRecoveryScript and have always used this backup command:
backup DSBCR DailyBackup
Which gets data, system, boot, cache, and recovery. I plan to do the same now that I know to put the OpenRecoveryScript into /data/cache/recovery instead of /cache/recovery on my Note 3.
So my question here is what partitions make sense to backup on the V40? And by "sense" I mean something likely to change or get broken (by errant apps or, more likely, my own doing) AND having the ability to restore using TWRP restore from backup.
There are a ton of partitions on this device - most of which I know little/nothing about. I do know that the system and vendor partitions have duplicates and so those are not needed, I guess. And it seems that the OpenRecoveryScript backup command has no way to handle anything except the following anyway:
S: system,
D: data,
C: cache,
R: recovery,
B: boot,
A: and-sec,
E: sd-ext (I don't need the external SD)
So there would seem to be no way to get modem and all the others I don't know much about anyhow.
When I tried to use the same command as above, I ended up with only boot and data - which is a great start! I think I saw an error that system was not mounted so maybe that explains its absence - perhaps I need to mount it first in my script (which I never have had to do on my Note 3)? Even though my V40 is rooted with Magisk, I am making changes to system so I do want to back that up (so long as I can use it during a restore).
Does it make sense to try and backup recovery since things seem really different recovery-wise on this device? Same question for cache and boot?
Thanks for your tolerance of my newbie questions.

Well from empirical experience, it seems that there is no way to backup anything save boot and data. I was able to mount /system_root but that had no impact on backing up system. And I was unable to mount cache or recovery explicitly so those are out as well.
PS - for anyone using the old 123 special partition for backups, they are no longer supported. I have never used them but decided to try and see what happened - only messages that they are no longer supported.
Perhaps there is some other way to accomplish this? Suggestions welcome. Thanks
Thanks.

That I'm aware of recovery should backup as part of the boot partition....I could be wrong tho

Ainz_Ooal_Gown said:
That I'm aware of recovery should backup as part of the boot partition....I could be wrong tho
Click to expand...
Click to collapse
I guess that does make sense. Thanks. I am still trying to get my mind around the whole new situation (compared to my old Note 3).