Hi. There is a thread here where we are wondering if SuperSu full unroot is enough to allow an OTA to succeed (assuming no other mods were made to the system other than root) (Nexus 6, Lollipop). There is speculation but hard to find a definitive answer on the subject. Anyone here able to shed light on that? Thanks!
It depends.
OTAs can update various parts of the system in various ways.
Previous to Lollipop, updates to the system partition were usually (but not always) done file-based. Which meant that if you removed/renamed all the modified files and put back their originals, the OTA would work. And thus, SuperSU's OTA survival option and/or the unroot option would allow OTAs to complete.
Partition-based OTAs have become standard with Lollipop. Some OEMs have used them before (different implementation), but it was rare. This new style OTA does not patch the differences between files, but on the underlying partition contents. If you make changes to a partition's filesystem and then revert them so that all the files and their contents are original again, this does not actually mean the partition contents are completely the same. In fact, just remounting /system read-write and not making any modification changes the partition contents - that specific change is actually the first thing these OTAs check.
If the OTA is partition based (which by far most OTAs upgrading from 5.0 to something newer will be) the only way to guarantee the correct contents of the system partition so that the patch script will succeed is to reflash the entire partition.
Chainfire said:
Partition-based OTAs have become standard with Lollipop.
Click to expand...
Click to collapse
I wonder if that is because of problems from rooted devices and modified System partitions. They now replace the entire partition to ensure a successful update.
Chainfire said:
In fact, just remounting /system read-write and not making any modification changes the partition contents - that specific change is actually the first thing these OTAs check.
Click to expand...
Click to collapse
Is that because some people leave '/system' read-write or is there some residual change even after the partition is returned to read-only. In other words: if someone remounts '/system' read-write and immediately returns it to read-only without any writes to it, is there some detectable change that remains.
Frank
Frank Westlake said:
I wonder if that is because of problems from rooted devices and modified System partitions. They now replace the entire partition to ensure a successful update.
Click to expand...
Click to collapse
No, it's for dmverity. A kernel-based security system that verifies the system partition contents are as the manufacturer intended. It requires the partition contents to be predictable and verifyable on block level. See https://source.android.com/devices/tech/security/secureboot/index.html for further details.
Frank Westlake said:
Is that because some people leave '/system' read-write or is there some residual change even after the partition is returned to read-only. In other words: if someone remounts '/system' read-write and immediately returns it to read-only without any writes to it, is there some detectable change that remains.
Click to expand...
Click to collapse
Yes, there is.
From what I can see though (in updater-script), in current OTA for past Nexus devices, they are still file-based, even for 5.0>5.0.1(2) upgrade path.
This is not truth for Nexus 6 and 9 OTAs, which are partition-based from day one.
My phone (3T) has an unlocked bootloader, is encrypted, not rooted, and running stock OOS 5.0.
I flashed TWRP and discovered that stock OOS restores the stock recovery in boot.
I saw the Oreo dm-verity thread by xenet, had a look at the zip file, noticed that it just modified fstab to prevent force encrypt, so I flashed it to see what happens.
And nothing happens. After the system had booted, fstab is unchanged from the original stock copy.
So I'm wondering whether this file is also restored when booting up on stock.
I get aggressive and go back to TWRP and delete /system/etc and /system/bin and modify build.prop.
Surely now the phone won't boot!
Wrong! It boots up and everything is back to normal in /system.
I go back to TWRP and have a look at /system and it shows me one without the etc and bin folders and has the modified build.prop.
What's going on? How can I see one version of /system in TWRP but a different version (ie, stock) when the phone has booted?
By the way I've been an Android user for many years and have rooted and flashed custom ROMs on a variety of phones and I've never seen anything like what's happening on my 3T. I'm sure that dm-verity is somehow involved in this.
Happened to me on my earlier OOS 5.0 attempts...
But i suspected Magisk is involved in my case.
I downloaded Magisk Module "System Terminal Debloater,"
remove some apps like Duo, Chrome, and Google Play Movies.
Some restarts, they magically re-appear again on Apps Drawer...
Haven't touch them yet again after....
nicknacknuke said:
Happened to me on my earlier OOS 5.0 attempts...
But i suspected Magisk is involved in my case.
I downloaded Magisk Module "System Terminal Debloater,"
remove some apps like Duo, Chrome, and Google Play Movies.
Some restarts, they magically re-appear again on Apps Drawer...
Haven't touch them yet again after....
Click to expand...
Click to collapse
Thanks.
I should have mentioned that I'm also not rooted. So stock OOS 5.0.
Sent from my OnePlus 3T using XDA Labs
When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?
Jackhass said:
When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?
Click to expand...
Click to collapse
No, I don't get that message because my phone is encrypted with a password. So the first thing I see in TWRP is the request for the password and then I'm presented with the menus.
However, in the Mounted menu, system isn't mounted and I have the option of mounting it in read-only mode.
Sent from my OnePlus 3T using XDA Labs
BillGoss said:
No, I don't get that message because my phone is encrypted with a password. So the first thing I see in TWRP is the request for the password and then I'm presented with the menus.
However, in the Mounted menu, system isn't mounted and I have the option of mounting it in read-only mode.
Click to expand...
Click to collapse
After first time flashing TWRP a folder gets created on your internal storage, with a hidden file called .twrps, go delete it and reboot recovery to trigger the message "allowing system modifications" on TWRP's first boot...
It's not about encryption, it's just that TWRP remember the decision you made due to the file I pointed out...
Sent from my OnePlus 3T using XDA Labs
Sam Nakamura said:
After first time flashing TWRP a folder gets created on your internal storage, with a hidden file called .twrps, go delete it and reboot recovery to trigger the message "allowing system modifications" on TWRP's first boot...
It's not about encryption, it's just that TWRP remember the decision you made due to the file I pointed out...
Click to expand...
Click to collapse
Somehow the attachment strikes on previous post
Edit: still not working, check your TWRP Folder on storage to find the file
Sent from my OnePlus 3T using XDA Labs
Sam Nakamura said:
Somehow the attachment strikes on previous post
Edit: still not working, check your TWRP Folder on storage to find the file
Click to expand...
Click to collapse
Thanks, you are correct. I'd forgotten that that TWRP remembers. Deleting .twrps does bring up the RO prompt after decrypting storage.
Jackhass said:
When you boot TWRP for the first time, it should ask you if you want to put the /system in read/write mode or if you want to leave it unchanged, did you choose the right option?
Click to expand...
Click to collapse
I had allowed changes to the system otherwise I couldn't have made changes to it, which includes the ability to restore the system partition.
But I'm still unclear why if I make changes to the system partition and boot with the stock kernel, then after the boot none of the changes are present in the system partition, but if I boot back into TWRP then the changes are all there.
I recall someone in another OOS 5 thread saying that the stock kernal replaces TWRP with stock recovery if you don't flash root (magisk/superSU). Is it possible that the kernel re-flashes system on boot? Another possibility is that TWRP thinks it's making changes to system but it's not actually? Not quite sure, I've never heard of anything like this before either, just throwing other ideas out there.
I've never read anything about the OP3T or any oneplus phones for that matter having A/B system partitions like the pixels. *shrug*
@nhshah7, something's like what you suggest must be going on to account for what I'm seeing. I'm hoping that someone can confirm my observations and provide a definite answer.
@BillGoss
My thread has been updated relating to all your queries...
Thank you...
https://forum.xda-developers.com/oneplus-3t/how-to/disable-dm-verity-force-encryption-op3t-t3688748
Xennet said:
@BillGoss
My thread has been updated relating to all your queries...
Thank you...
https://forum.xda-developers.com/oneplus-3t/how-to/disable-dm-verity-force-encryption-op3t-t3688748
Click to expand...
Click to collapse
Actually it doesn't explain how TWRP can make changes to system yet the phone boots up on an unmodified system if using the stock kernel. And then, when you boot back into TWRP and look at system, the changes are still there.
Where does the unmodified system come from?
Where does the modified system live?
Why doesn't modifying system result in a failed boot due to dm-verity, while restoring a backup of system does result in a failed boot?
So many questions with no answers.
BillGoss said:
....So many questions with no answers.
Click to expand...
Click to collapse
Not sure if this is applicable in your case but the following possibilities may be worth considering for you:
1. Are you sure that the system image is actually getting modified? If the system partition is not mounted before flashing the zip and the zip being flashed does not mount the system partition in read / write, then no changes to system partitions will actually be written.
2. If dm-verity is enabled, then restoring system could result in an error as this is different from restoring a system-image (nandroid copy of the whole partition and not just the files in the system partition). DM-verity can be triggered if the files are all the same but the dm-verity signature computed by hashing the system partition has changed.
3. For boot partitions, strange behaviour can occur if remnants of the previous boot.img are still in the partition (...e.g. if the previous boot.img was of larger size and a new boot.img of a smaller is flashed, then there will be some bytes after the new boot.img that are from the previous boot.img). To verify this, format the boot partition from fastboot and see if you notice anything different with the new boot.img.
4. In Oreo / 8.0, dm-verity flags are stored in dtb (device tree blobs) inside the kernel and not in the fstab file. Only data encryption can be changed from the fstab file and dm-verity needs to be changed from changing the dtb (...Magisk beta v1456 and SuperSu 2.82 SR4 do this, I think).
rk2612 said:
Not sure if this is applicable in your case but the following possibilities may be worth considering for you:
1. Are you sure that the system image is actually getting modified? If the system partition is not mounted before flashing the zip and the zip being flashed does not mount the system partition in read / write, then no changes to system partitions will actually be written.
2. If dm-verity is enabled, then restoring system could result in an error as this is different from restoring a system-image (nandroid copy of the whole partition and not just the files in the system partition). DM-verity can be triggered if the files are all the same but the dm-verity signature computed by hashing the system partition has changed.
3. For boot partitions, strange behaviour can occur if remnants of the previous boot.img are still in the partition (...e.g. if the previous boot.img was of larger size and a new boot.img of a smaller is flashed, then there will be some bytes after the new boot.img that are from the previous boot.img). To verify this, format the boot partition from fastboot and see if you notice anything different with the new boot.img.
4. In Oreo / 8.0, dm-verity flags are stored in dtb (device tree blobs) inside the kernel and not in the fstab file. Only data encryption can be changed from the fstab file and dm-verity needs to be changed from changing the dtb (...Magisk beta v1456 and SuperSu 2.82 SR4 do this, I think).
Click to expand...
Click to collapse
I'll come back to 1.
2. That makes sense and accounts for why a restore of the system partition with the stock boot image causes me to get dumped back in fastboot mode. If I flash the stock system zip file then the system boots properly.
3. I've not had any issues with strange boot behaviour. I'm always starting with stock or flashing kernels that modify the stock boot image, like Blu Spark.
4. I gathered this from my reading of various threads. If I want to make changes to the system partition and get them to stick and not fail dm-verity then I have to flash a custom kernel. I've proven this in my testing. (A rooting solution would also work, but I've not done this).
Back to 1:
Here's what I've done:
Starting with pure stock image (flash OOS 5.0).
Boot into fastboot and flash TWRP.
Boot into recovery.
Mount system as rw. (In ro mode the next step fails)
Delete the bin, etc, and lib folders in system using the TWRP file manager. (Screenshot a)
Reboot system.
... First interesting fact ...
System boots ok, deleted folders are present in file manager. (Screenshot b)
Boot into fastboot and flash TWRP. (Booting with stock restores stock recovery)
Mount system.
... Second interesting fact ...
TWRP file manager shows that deleted folders are missing. (Screenshot c)
Flash custom kernel or patched boot image
Reboot system
... Third interesting fact ...
System fails to boot. Hangs on splash screen.
So TWRP made the changes (otherwise how could they be visible between reboots, including a replacement of recovery) and I only did them once.
Yet they don't actually take effect until I replace the stock boot image.
So, where are the changes hiding? What did TWRP actually change?
Screenshots (note that TWRP has the wrong timezone set so the time shown is wrong):
BillGoss said:
....
Back to 1:
Here's what I've done:
Starting with pure stock image (flash OOS 5.0).
Boot into fastboot and flash TWRP.
Boot into recovery.
Mount system as rw. (In ro mode the next step fails)
Delete the bin, etc, and lib folders in system using the TWRP file manager. (Screenshot a)
Reboot system.
... First interesting fact ...
System boots ok, deleted folders are present in file manager. (Screenshot b)
Boot into fastboot and flash TWRP. (Booting with stock restores stock recovery)
Mount system.
... Second interesting fact ...
TWRP file manager shows that deleted folders are missing. (Screenshot c)
Flash custom kernel or patched boot image
Reboot system
... Third interesting fact ...
System fails to boot. Hangs on splash screen.
So TWRP made the changes (otherwise how could they be visible between reboots, including a replacement of recovery) and I only did them once.
Yet they don't actually take effect until I replace the stock boot image.
So, where are the changes hiding? What did TWRP actually change?
Screenshots (note that TWRP has the wrong timezone set so the time shown is wrong):
Click to expand...
Click to collapse
Some more thoughts for you to consider:
1. Have you tried this with the official TWRP recovery version 3.2.0-0?
2. Is there anything inside the folders that you see using the file manager after a regular boot? Folders of same name may exist in the boot ramdisk and these are merged with system folders after boot.
3. Try wiping cache between reboots and see if that changes any of your observations.
rk2612 said:
Some more thoughts for you to consider:
1. Have you tried this with the official TWRP recovery version 3.2.0-0?
2. Is there anything inside the folders that you see using the file manager after a regular boot? Folders of same name may exist in the boot ramdisk and these are merged with system folders after boot.
3. Try wiping cache between reboots and see if that changes any of your observations.
Click to expand...
Click to collapse
Good questions. They got me thinking more about how this could possibly work.
I had a look at the cache and there's definitely no copy of the system hiding there.
I also unpacked the ramdisk in the boot image and it had nothing in system. Furthermore, the boot position is only 64 MB, no where near enough to hold the system.
Then I installed Magisk so that I could browse around the phone's partitions and take copies.
I learnt two things from this:
1. If there's a second copy of the system there are only three partitions large enough to hold it (/proc/partitions shows the sizes in 1 kB blocks). The system is about 1 GB. There is space in the system partition (sde20) for 3 GB. There's also space in the data partition (sca15). And there's space in the major partition holding the modems (sdf).
I could eliminate the data partition by formatting it but restoring the internal storage (sdcard) is such a a pain.
So I'll just accept that there is space for a copy, but I'm unlikely to find out exactly where.
2. When I had Magisk installed installed and the system boot, I added a folder and file to /system/priv-app using a file manager (so not using TWRP). I then booted into recovery, flashed the stock boot image, and rebooted. I was expecting it to fail dm-verity (modified system) but it didn't. After booting up there's no evidence of the folder I added to priv-app.
And if I restore the Magisk boot image then the additions show up again.
I'm actually very impressed with how the stock system (kernel, recovery, system) protects itself from modification. Very cool!
Hi guys
I ask for something that I didn't understand so well, but now every time I change roms do I have to flash the DM verity zip? Because the last time I tried the lineage 17 the system was not able to boot I think because of this dm-verity zip that I didn't flash and the internal archive encryption.
However, my internal archive is not encrypted because the first time I installed xiaomi.eu I made Format data and typed Yes and then I also made all the wipes and installed the ROM in this way I removed the encryption from the archive interior right?
But do I still need to flash the DM verity zip every time I switch Rom?
For example, now I want to try havoc 3.0.
I have a xiaomi mi 8 with the latest wikley from xiaomi.eu 9.10.24
Up!
Inviato dal mio MI 8 utilizzando Tapatalk
Since your Filesystem is not encrypted, you have to do that.
The War Profiteer said:
Since your Filesystem is not encrypted, you have to do that.
Click to expand...
Click to collapse
But do I have to flash the DM verity zip every time I update my Rom? (when I update my rom I don't do a clean install I only do Wipe Cache and Devilk)
Or should I flash it only when I change ROM then after a clean installation?
(When I change ROM I make wipe data and system)
andrea0807 said:
But do I have to flash the DM verity zip every time I update my Rom? (when I update my rom I don't do a clean install I only do Wipe Cache and Devilk)
Or should I flash it only when I change ROM then after a clean installation?
(When I change ROM I make wipe data and system)
Click to expand...
Click to collapse
Probably yes.
I'm just going to say this again, DM-Verity stands for "D"-Device "M"-Mapper verity (verification). This protection looks at hashes for the mounted partitions and kernel to make sure they have not been modified. If the hashes don't match, the boot chain fails to load. Once again, this really doesn't have to do with Encryption. Encryption was sometimes included in the DM-Verity ZIP files also called FEC remover (Forced Encryption Remover). Even though the are a lot of times included in the 'DM-Verity" remover zip files, they are separate things! If you modify the stock MIUI partitions at all, it needs to have the DM-Verity removed -but doesn't have to have the FEC (encryption) removed. These forums are full of everyone saying flash the DM-Verity to remove encryption, which is only correct in a sense because most of the ZIP files floating around does in-fact remove the FEC (forced encryption) at the same time. If i am running modified stock, i want to keep encryption and just remove the verity bit that fails.
Info on DM-Verity:
https://source.android.com/security/verifiedboot/dm-verity
./
Agimax said:
I'm just going to say this again, DM-Verity stands for "D"-Device "M"-Mapper verity (verification). This protection looks at hashes for the mounted partitions and kernel to make sure they have not been modified. If the hashes don't match, the boot chain fails to load. Once again, this really doesn't have to do with Encryption. Encryption was sometimes included in the DM-Verity ZIP files also called FEC remover (Forced Encryption Remover). Even though the are a lot of times included in the 'DM-Verity" remover zip files, they are separate things! If you modify the stock MIUI partitions at all, it needs to have the DM-Verity removed -but doesn't have to have the FEC (encryption) removed. These forums are full of everyone saying flash the DM-Verity to remove encryption, which is only correct in a sense because most of the ZIP files floating around does in-fact remove the FEC (forced encryption) at the same time. If i am running modified stock, i want to keep encryption and just remove the verity bit that fails.
Info on DM-Verity:
https://source.android.com/security/verifiedboot/dm-verity
./
Click to expand...
Click to collapse
Thanks for clearing that up, there is a lot of conflicting information out there and it's difficult finding information that is currently valid and applicable to our phones.
Edit :
I should add that I never flash dm-verity and that I do not have issues. However, I also do not change the boot, except for Magisk.
Can anybody explain why isn't there a wipe system option in the twrp in advanced wipe section...
In earlier devices that I've used ,the wipe system option has always been there.
Or is this happening only in the nebrassy recovery for sweet?
just a query!
Same question.. Can an expert answer this? Thanks
In my opinion normally it's not needed and for newbie it's dangerous
- I'm no expert at all, but missing option for wipe /system is not there for a reason (on my previous surya device (when it was launched) the first builds of recoveries had option to wipe /system, but people find out very quickly that wipe /system on super partition is not a good idea, very few ended with bricked devices, after that wipe /system was removed from recoveries)
- on A9 /system had its own partition and very simply put - it was easy to manipulate, situation changed with A10 (A11), when /system become like virtual partition (along with /vendor, /product and whatever manufacturer of the phone decided to put there) or directory if you like in super partition and its RO by default (unless flashing to it, you can't manipulate it directly on file system level - every wipe option in recovery means you simple delete some files here and there and you can't delete files on Read Only virtual partition)
- problem is much more complicated and I'm explaining it as I understand it - I can be wrong on detail, but this is roughly the situation
- if you're active tester and/or flashaholic who likes to test ROMs as they are released it is a good idea to flash (before you flash new custom ROM) either recovery version of latest MIUI available for your device (or maybe Xiomi.eu ROM based on that) - inclueded installation scripts will clean your super partition to the certain level - so you can for example flash stubborn ROM, that refused boot so far, etc. - its not wipe and I agree - clean slate is always better, than breadcrumbs of the previous flashes left here and there, but here we are...
- sure - direct control over /system part of super partition would be nice - and I hope that one day TWRP or OFOX devs will tame that "super" beast and return wipe /system when it will be safe for us to use it as before...