I had an experience the other night that I found rather frightening and enlightening, and felt the need to contribute my learning. I attempted to install a new ROM on my Nexus 5 and encountered a freeze during the /system write process. I waited 30 minutes and decided it wasn’t going to finish, so I powered off. As an avid and experienced ROM flasher I knew this could hurt, and knew it was an inevitable requirement.
Upon rebooting back to recovery (I had already wiped /system /data, etc. in preparation for a new ROM) I found that I was unable to mount /data and subsequently /sdcard. I received a few different errors depending on which recovery I attempted (fastboot flashed). I was given neat things like “Enter password to decrypt /data” – TWRP, or “E: Unable to mount /data, invalid argument”, “E: Error mounting /data”, “E: Error mounting /sdcard”.
During all of this I realized I could get an ADB shell going from within Philz recovery (I am not sure if TWRP or CWM do this, nor did I try). After much research, and very much to my dismay, I was unable to find a complete guide or walkthrough for how to go about recovering from this. This is the procedure I glued together from my own digging and manual learning of the commands required. This procedure may or may not work in all cases but is a viable step to recovering your /data partition before going the nuke & pave route with the factory image!
(NOTE: I’ll try to keep this as simple as possible. I am not a dev. I do this for fun/learning in the little free time I get and am not typically available to assist, so please use this only if you feel comfortable with the material. Flashing a ROM/Kernel/etc. is one thing. Digging in the innards of the Android partitions/file system is a whole other beast! To be VERY clear: I am not responsible for damages or loss incurred by the use or misuse of this information.) Also note, if you have an ADB enabled recovery, you can probably try this process against your devices partitions as well.
The following instructions assume you are unable to mount /data or /sdcard on the Nexus 5 and are receiving errors. This also assumes that the wipe /cache function is not resolving the issue for you (as this seems to work for some depending on the issue).
Prerequisites:
- Philz recovery for LGE Nexus 5 (http://forum.xda-developers.com/showthread.php?t=2565174)
- Android ADB interface drivers and software (Available from existing root toolkits and SDK)
- Fastboot executable (Also available from existing root toolkits and SDK)
***Note: I performed all of this from Windows 8. The toolkit I used was Wug’s Nexus Root Toolkit (http://forum.xda-developers.com/showthread.php?t=2517778). If you are a Windows user, read and follow the toolkit instructions to install the drivers. Use the Advanced Utilities[Launch]->Manual Input[Launch CMD Prompt] to bring up a Command prompt and perform the steps below.
- For simplicity, ensure that the ADB and Fastboot binaries are in the same location if you are gathering these binaries from the SDK or other source.
- Create a folder that you can later use to pull recovered files from your phone. (if so desired)
Terminologies used:
- Terminal = Command Prompt (Windows), Bash/etc (Linux), Terminal (MacOSX)
- Fastboot = Application that interfaces with the Bootloader (flash/wipe partitions, etc)
- ADB = Android Debug Bridge (Provides an interface to interact with Android filesystem)
- Binaries = Files containing executable code (for this context, ADB and Fastboot applications)
- Bootloader = Think of it as the EFI or BIOS equivalent for your phone. (Accessed with VOLDN+PWR)
- Recovery = CWM/TWRP/Philz (We will be using Philz, as this was what I used to write this guide)
Steps:
1. Install all required drivers and prerequisites listed above.
a. Use Wug’s toolkit or whatever method gets your drivers installed for your OS.b. If you are not familiar with this process, then please research!
2. Power off your phone and connect via USB to your PC.
3. Power your phone into the bootloader by holding the PWR+VOLDN keys
4. In a Terminal, browse to the location of your fastboot binaries. (See prerequisite note for Wug’s)
a. Enter ‘fastboot devices’ to verify that your phone is detected. If it returns a “<serialnumberofdevice> fastboot” then you may proceed. If it does not, please check your driver installation and start over.
5. Flash and boot to Philz recovery:
a. Extract the ‘recovery.img’ file from Philz Recovery ZIP and place in the same location that fastboot and adb binaries are located.b. Enter ‘fastboot flash recovery recovery.img’ (without quotes) in the Terminal. This will write the new recovery.c. Once complete (takes a brief second usually), use the VOLUP/DN keys until the Bootloader shows “Recovery Mode” and press the power key. Your phone should now be booting into Philz recovery.
6. Use ADB to launch a root shell from within Philz recovery
a. Ensure that the drivers are now detecting your device as an ADB ready device by entering: ‘adb devices’. If it returns “<serialnumberofdevice> recovery” then you are ready to proceed. If it does not, please verify your ADB driver installation and check again.b. Enter: ‘adb shell’ to gain full root shell access to the Android file system. (You should see a “~#” prompt to show that you’re now in a root shell)
7. Attempt repair of /data partition. This partition includes /sdcard and dalvik. (/sdcard is linked to /data/media) At the ~# prompt:
a. Enter: ‘e2fsck /dev/block/platform/msm_sdcc.1/by-name/userdata’b. You should now be prompted to repair or fix errors. Press [Enter] key to use the default answer of for yes. If you are not, then this fix may not apply to you.c. Continue pressing [Enter] until the process completes.d. Proceed to next step if you have gotten this far.
8. Attempt to manually mount the data partition.
a. We are going to mount /data as read-only for now. To do this, enter: ‘mount /dev/block/platform/msm_sdcc.1/by-name/userdata /data’b. If mount is successful, it will simply drop down to the next blank shell prompt. If it fails, you will receive an error. If you receive an error, then the repair has failed.c. If you did not receive an error, you can now browse to your /data and /sdcard partitions. This is accomplished by entering: ‘cd /data’ for /data, and ‘cd /data/media/0’ OR ‘cd /sdcard’ for your /sdcard (/sdcard is a link that will drop you directly to /data/media/0)
9. BACKUP YOUR ENTIRE SDCARD TO YOUR COMPUTER!!! (Optional, but highly recommended)
a. Enter: ‘adb pull /data/media <path_on_computer>’b. <path_on_computer> will be a folder you created in the prerequisites.c. Wait for this process to complete.
Congratulations! You may now proceed to launch recovery and flash at will. I highly recommend after making the backup in step 9 that you perform a full factory reset to correct any potential lingering issues with the partitions. You can then push the files back to your /sdcard by mounting /data as RW using ADB shell through Philz recovery (please reference the ‘mount’ and ‘adb’ command utilities for information about how to do this).
If you find yourself in the unfortunate position to have to use this, please share your results in the forum for others. If you find something does not work, please post it (I will check occasionally and update as I can). I do not frequent my PM’s, and again, I am not available to provide support. Please understand this and use at your own risk.
If this helped you, then my job here is done . Thanks for reading!
(I am not affiliated with any of the devs that provide these tools. Be sure to thank them if you used their tools, as they deserve all credit for enabling and teaching us to do these things – even when we do break our own stuff!)
Nice guide I'm sure this is going to be very useful for a lot of people. :thumbup::thumbup:
Sent from my Nexus 5 using Xparent BlueTapatalk 2
Really helpful guide, I corrupted the data and cache partition while updating (via official OTA by sideload) to 4.4.1 (full stock Nexus 5), and I had to format all flashing then factory image and losing all the data...
Now if it does happens another time, I'll know what to do.
Thanks
Good write up but, I don't recommend tool kits. You can do all this with fastboot commands. You could just flash userdata image with fastboot
As a Linux admin, I'd wanna format /data after pulling off any files I wanted to keep rather than just rely on fsck . Is that an option? Is mkfs even on the recovery?
Sent from my BN NookHD+ using xda app-developers app
Thanks for sharing.
My nexus5 cant mount cache partition and i cant flash to stock factory，this maybe helpful.
Thank you again.
ywt474000158 said:
Thanks for sharing.
My nexus5 cant mount cache partition and i cant flash to stock factory，this maybe helpful.
Thank you again.
Click to expand...
Click to collapse
Can you fastboot flash the cache.img?
razholio said:
As a Linux admin, I'd wanna format /data after pulling off any files I wanted to keep rather than just rely on fsck . Is that an option? Is mkfs even on the recovery?
Sent from my BN NookHD+ using xda app-developers app
Click to expand...
Click to collapse
I'm no Linux admin but there's a wipe /data button which does that in TWRP and also probably in CWM.
Sent from my Galaxy Nexus using Tapatalk
Somehow it seems as though all of my partitions have been wiped/corrupted and I was directed to this thread as possible help. When trying to run step 7 command(had to do it with CWM v6.0.4.5 as it doesn't seem to want to keep philz when sideloading or fastboot flashing it), it gives me the following response:
e2fsck: 1.41.14 (22-Dec-2010)
e2fsck: Superblock invalid, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/block/platform/msm_sdcc.1/by-name/userdata
The super block could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device>
My phone was having freezes and shutdowns so I decided to flash a new ROM to freshen things up and it failed and aborted. After that I've got no android to boot into and I can't mount or access any of the partitions. I've got fastboot and adb, but no clue where to begin as it seems like I have to recreate my partition table but can't find any info on doing that for the nexus 5 itself
tidoubleger said:
Somehow it seems as though all of my partitions have been wiped/corrupted and I was directed to this thread as possible help. When trying to run step 7 command(had to do it with CWM v6.0.4.5 as it doesn't seem to want to keep philz when sideloading or fastboot flashing it), it gives me the following response:
e2fsck: 1.41.14 (22-Dec-2010)
e2fsck: Superblock invalid, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/block/platform/msm_sdcc.1/by-name/userdata
The super block could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device>
My phone was having freezes and shutdowns so I decided to flash a new ROM to freshen things up and it failed and aborted. After that I've got no android to boot into and I can't mount or access any of the partitions. I've got fastboot and adb, but no clue where to begin as it seems like I have to recreate my partition table but can't find any info on doing that for the nexus 5 itself
Click to expand...
Click to collapse
Have you tried this yet http://forum.xda-developers.com/showthread.php?p=47156064
jd1639 said:
Have you tried this yet http://forum.xda-developers.com/showthread.php?p=47156064
Click to expand...
Click to collapse
Yes I tried it by using the flash-all.bat, flashing individually, and even Wug's toolkit as a last resort there but got nothing. When flashing the bootloader and radio it sends over fine but then gives me FAILED (remote: flash write failure) and finishes. When trying to flash the system or userdata it gives me FAILED (remote: failed to erase partition) and ends.
tidoubleger said:
Yes I tried it by using the flash-all.bat, flashing individually, and even Wug's toolkit as a last resort there but got nothing. When flashing the bootloader and radio it sends over fine but then gives me FAILED (remote: flash write failure) and finishes. When trying to flash the system or userdata it gives me FAILED (remote: failed to erase partition) and ends.
Click to expand...
Click to collapse
I'm thinking your emmc went bad. Rma is probably your option right now
jd1639 said:
I'm thinking your emmc went bad. Rma is probably your option right now
Click to expand...
Click to collapse
Well as far as Google knowing it, if I am able to successfully reset the tamper flag and lock the bootloader am I good to go or will they still find out?
tidoubleger said:
Well as far as Google knowing it, if I am able to successfully reset the tamper flag and lock the bootloader am I good to go or will they still find out?
Click to expand...
Click to collapse
If all your positions are corrupted I doubt you'll be able to reset the tamper flag. But also, no one will be able to see the tamper flag. But try to reset it and try to relock the bootloader. I don't think anyone will be able to tell if you modified anything or not.
not know
Did not know this could be done
jd1639 said:
If all your positions are corrupted I doubt you'll be able to reset the tamper flag. But also, no one will be able to see the tamper flag. But try to reset it and try to relock the bootloader. I don't think anyone will be able to tell if you modified anything or not.
Click to expand...
Click to collapse
Hmm, I can see that the tamper flag is still true even after running that zip because it still fails because it can't mount /cache. I'm getting the feeling that the partitions aren't corrupted but have just been wiped or something. When I booted into TWRP through fastboot I was able to adb push to the sdcard and it showed up in the list when trying to install a zip, however the Internal Storage still read 0MB. I won't be able to RMA it for a few more days so if any other ideas come about, I'm all for it as I'd rather not have to RMA this lol
tidoubleger said:
Hmm, I can see that the tamper flag is still true even after running that zip because it still fails because it can't mount /cache. I'm getting the feeling that the partitions aren't corrupted but have just been wiped or something. When I booted into TWRP through fastboot I was able to adb push to the sdcard and it showed up in the list when trying to install a zip, however the Internal Storage still read 0MB. I won't be able to RMA it for a few more days so if any other ideas come about, I'm all for it as I'd rather not have to RMA this lol
Click to expand...
Click to collapse
In twrp 2.6.3.4 have you tried going into wipe and then format data?
@tidoubleger I am going to call in some RCs that were having this problem on an N7. They eventually got it sorted and up and running. They might be able to help.
@demkantor @MameTozhio @kilometers4
Fellas, would you be able to help this person out the same way you did here.
jd1639 said:
In twrp 2.6.3.4 have you tried going into wipe and then format data?
Click to expand...
Click to collapse
Yeah it fails due to being unable to mount /cache.
Woody said:
@tidoubleger I am going to call in some RCs that were having this problem on an N7. They eventually got it sorted and up and running. They might be able to help.
@demkantor @MameTozhio @kilometers4
Fellas, would you be able to help this person out the same way you did here.
Click to expand...
Click to collapse
Thanks Woody, I appreciate the call to arms I'm currently at Disneyworld, so I won't be able to test anything until this evening but will do my best to answer questions until then
tidoubleger said:
Somehow it seems as though all of my partitions have been wiped/corrupted and I was directed to this thread as possible help. When trying to run step 7 command(had to do it with CWM v6.0.4.5 as it doesn't seem to want to keep philz when sideloading or fastboot flashing it), it gives me the following response:
e2fsck: 1.41.14 (22-Dec-2010)
e2fsck: Superblock invalid, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/block/platform/msm_sdcc.1/by-name/userdata
The super block could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device>
My phone was having freezes and shutdowns so I decided to flash a new ROM to freshen things up and it failed and aborted. After that I've got no android to boot into and I can't mount or access any of the partitions. I've got fastboot and adb, but no clue where to begin as it seems like I have to recreate my partition table but can't find any info on doing that for the nexus 5 itself
Click to expand...
Click to collapse
I had the same problem. This is how I got it sorted.
follow these steps and you should get it sorted
fastboot erase system -w
fastboot erase boot
fastboot erase recovery
fastboot flash bootloader bootloader.img
fastboot flash recovery recovery.img
once it flashes, boot into recovery.
format sdcard (i believe it is)
Now adb sideload ROM and gapps
I have class until this afternoon, but that should get it fixed
If you have any hangups post it here and I'll help you when I'm free.
Good Luck
Hi developers. I am sorry for posting this. I spent the last week trying to solve it by myself with no hope. This is my second time installing something on a phone, but it is my only phone, so I beg anyone for a help...
-What I did:
Some days ago I downgraded to this ROM C5503_10.1.1.A.1.310_GLOBAL-LTE.ftf to use DoomLord rooting script. I did it with flashtool for linux and I applied his .bat step by step in the terminal since windows would not detect my phone.
It worked. I had root for some days, but I was still annoyed by sony default android. So I decided to install Cyanogenmod.
I unlocked the device with sony official system and wen't straight to this instructions, before the first reboot
wiki.cyanogenmod.org/w/Install_CM_for_yuga
I booted succesfully in CWM, followed everything as it says there. But that's where weird things happened:
-The problems:
-The backup
I tried, it wouldn't mount /sdcard. Since I don't understand much about this, I thought it was normal. The next choice was sdcard1, I backed up there. Or so I thought...
-The factory reset
I factory reset, again, not mounting sdcard. Here is the message that shows when I try this now:
can't mount /data!
Error mounting /sdcard.android_secure
Skipping format...
Data wipe complete.
Since it said it is complete, I went on installing the zip file from my sdcard1. Both CM 10.2.1 (dogo, the right one for my phone) and the appropriate GAPPS.
Now it loops on the CM loop animation and I have to remove the battery...
-The restore problem
It still boots on the recovery mode. So I tried recovering my backup from sdcard1. But the image name is 1970.01.01.00.03.16. And it says "md5 mismatch"
I tried flashing again the stock rom with flashtool. The proccess goes on but nothing happens. I still have CWM and the boot loop.
I read elsewhere someone with a similar problem who solved using sony "emma" software. I installed it, it won't even recocnize my phone.
It recocnizes that there is a phone, but don't know which one.
But that has alway been the case with windows. I haven't been able to do anything in windows other then accessing the sdcard (when the phone worked).
Is there something I can do? I imagine that somehow, for some reason, the /data and /sdcard partitions got corrupted. I imagine I would need to repartition this and install again, but I have no idea how this happens on phones...
I can mount /system /cache and /storage/sdcard1. just /data I can´t. Says "error mounting /data"
This is my only phone and a vey recent $400 thing. I was very stupid to do that withouth a replacement and really need this phone. I greatly appreciate any help...
I found this post forum.cyanogenmod.com/topic/6433-solved-messed-up-partitions-on-internal-storage/ searching the internet. Is it possible that this would solve my problem? or would it finish bricking the phone?
Here's what you'll need:
Working recovery, basic knowledge of adb & the shell
Parted (download here)
stock PB31IMG.zip
Note also that I had run unrevoked forever (so my phone was S-OFF) ... I'm not sure if that's required or not.
So, grab parted from the link above. Now you need to extract the individual binaries from the .zip (the 6 files in the sdparted folder within the zip), ideally to your android-sdk\tools directory. Now push all 6 files (adb push [file] /sbin/). Next, we need to make them useable, so go into the shell (adb shell). Change to your /sbin/ directory, and run: chmod 0755 <file> on each of the 6 files.
Now, we need to fix the partitions. This is assuming that the partitions are there, just the wrong format (which is what happened to me .. I accidentally made them FAT32 instead of ext). So, run the following: parted /dev/block/mmcblk0 mkfs ext2. It will ask if you want to continue, hit yes. When it asks for the partition number, enter 1. Next, when it asks for the format, enter ext2. Let it do its thing. Now, once it's done, run parted again. This time, enter partition 2 (everything else is the same).
Click to expand...
Click to collapse
as usual, if anything goes wrong, no responsibility etc
The official update tutorial for rooted users doesn't keep app data, only internal storage! This tutorial keeps EVERYTHING
This method allows updating from 4.5.15 (unlocked, encrypted, rooted) to 5.0.2 without any data (sys settings + app data + user data) loss
A FAQ section is present at the end of this post and will be regularly updated.
List of files to download while doing the following steps:
OOS 5.0.2 ROM
Codeworkx TWRP recovery
Latest Magisk
Terms and software used in this guide:
Fastboot / Bootloader = bootloader of the phone, it's a very low level mode of the phone that allows booting into recovery. Can be accessed by using the advanced reboot menu (enable in dev options) or by "adb reboot fastboot".
Recovery = a small operating system on the phone that allows you to do various operations even when the main OS (Android) is broken. This includes flashing ROMs, modifying stuff on the storage, etc. It's the Android swiss army knife. If you can get a phone to boot TWRP, then you can do almost anything.
Magisk = rooting software that uses a systemless method to keep SafetyNet working. Systemless = instead of modifying the system, every change is put in a separate image that is mounted "over" the system. When the system tries to access a file modified by Magisk, instead of reading it from the partition, it reads it from Magisk. It's recommended to use Magisk instead of SuperSU as of 2018.
ADB = tool that allows controlling the phone from your PC through USB. You can use it when you're in Android if USB debugging is enabled in the settings, or when you're in TWRP. Here, we mostly use it for transferring files directly (without MTP) and running commands (using "adb shell")
Note: For this guide you will be required to download and install Magisk. If you don't want your phone to be rooted, then at the end of this guide reboot into TWRP, wipe both caches and re-flash the OS. This will uninstall Magisk and any other root patch. Beware: it will reflash stock recovery, so if you ever want to re-root, you'll need to reboot to fastboot and flash TWRP manually.
Convention for commands that you will have to run:
a command line starting with "C:" means that it should be run on your PC
a command line starting with "~ #" means that it should be run on your phone (through adb shell) while in TWRP
a command line starting with "OnePlus5:/ $" means that it should be run on your phone (through adb shell) while in OxygenOS
Although the commands start with "C:", this is just for readability purposes. You should run everything from inside an empty directory with enough disk space and writing access.
Your phone will have to be plugged in to your PC from the beginning to the end. Also, make sure it has at least 80% battery before beginning, just in case.
I know, the tutorial is huge. This is simply due to the fact that if I just wrote "make a nandroid backup of this and that, flash, and restore the backup while doing this", then some people may encounter problems because not everyone knows how to do a nandroid backup, restore it, etc. Also, there are a lot of things that need to be done precisely that way and not another way, which explains why the tutorial is huge. Also, you may notice that there is a lot of commands to run throughout the tutorial, this is because that way, I'm sure that at the end, you will have done everything like I did it on my phone, so that if you have a problem it's much easier to figure out where it comes from.
Summary of what you need to do (this is only a SUMMARY to give you a preview of what the whole thing looks like, you shouldn't follow it except if you're really an expert since a lot of things need to be done precisely, instead you should follow the easier complete steps below):
Make a Nandroid backup of /data
Backup files on internal storage
Wipe everything (internal storage + /data + system + caches), and then format data (important!)
Push and flash the OOS zip
Wipe caches and reboot (to Oreo!)
When it reboots, make sure everything (features, like Wi-Fi and fingerprint sensor) works. Don't "save anything" though, everything you do will be erased when we'll restore your backup. This is just a "test drive" for Oreo.
Reboot to TWRP, wipe Data and restore the /data backup
Run the three commands to fix Wi-Fi and fingerprints
Reboot (to System) and check everything works (don't do anything, don't change any setting, just make sure it works)
Reboot to TWRP, rename the "Android" folder to "Android_oreo" on sdcard, delete everything else on sdcard and restore your internal files
Rename the freshly restored Android (nougat) folder to "Android_nougat" and rename "Android_oreo" to "Android".
Flash Magisk, wipe dalvik+cache and reboot to System
When in Android, everything should work except some apps won't have their data. This is normal. Open a terminal (either on your phone using Termux or from your PC using adb shell), elevate using su and rename "Android" to "Android_oreo" and "Android_nougat" to "Android" (this is so that it correctly restores permissions)
If everything works fine, delete the "Android_oreo" folder
First, if you have Xposed Framework (systemless or not) installed, uninstall it. Next, if needed, uninstall any Magisk module that is "Nougat-only" to prevent any problems afterwards.
Boot the phone to bootloader/fastboot (either using advanced reboot, or by using volume down button when you start your phone) and boot to the TWRP recovery by doing
Code:
C:\> fastboot boot twrp-3.2.1-0-oreo-8.1-codeworkx-cheeseburger.img
from your PC.
Next, in TWRP, make a backup of /data (using the Backup button). Then, still while in TWRP, run the following commands:
Code:
C:\> adb shell
~ # cd /sdcard
/sdcard # tar cvf twrp.tar TWRP
/sdcard # md5sum twrp.tar
<< md5 checksum of twrp.tar >>
/sdcard # exit
C:\> adb pull -p /sdcard/twrp.tar
When the above command has finished, make sure that the checksum of the received twrp.tar file matches the one previously displayed.
If it doesn't match, delete the file and run adb pull again. Don't continue following this guide until you have received a 1:1 (checksum-wise) backup of /data.
Code:
C:\> adb shell
<< WARNING: dangerous command! double check the following line is correct before pressing enter! >>
~ # rm -rf /sdcard/TWRP
~ # rm /sdcard/twrp.tar
That was for /data. Now, the backup for the internal storage:
Code:
~ # cd /sdcard
/sdcard # du -csh
<< you should see here the total size of sdcard, that'll give you an idea of how long it'll take >>
/sdcard # tar cvf sd.tar element1 element2 element... elementN
<< in the command above, replace "element1..N" by a space-separated list of what you want to have in the backup.
Keep in mind that your list HAS to contain the element "Android" (case is important). It contains the app data.
Let's say for example you want to keep only the photos you have taken (and nothing, nothing else that was on internal storage).
The photos are in the folder DCIM, so the command will look like this:
tar cvf sd.tar Android DCIM
(because you want DCIM, and Android has to be in the list, no matter where)
>>
/sdcard # md5sum sd.tar
<< md5 checksum of sd.tar >>
/sdcard # exit
C:\> adb pull -p /sdcard/sd.tar
When the above command has finished, make sure that the checksum of the received sd.tar file matches the one previously displayed.
If it doesn't match, delete the file and run adb pull again.
Keep in mind that anything you don't put in that list will not be backed up and will be lost!
Now, you have a backup for all the important stuff so we can start doing the real sh*t.
Now, the important step:
Go back to the TWRP home screen, press "Wipe", "Advanced Wipe" and there check "Dalvik / ART Cache", "Cache", "System", "Data" and "Internal Storage". Confirm using the slider at the bottom of the screen. Press the home button, then "Reboot" and "FastBoot". Now, type the same fastboot command as in the previous step to boot the recovery image. You'll enter the recovery as before.
Now, on your PC, in the terminal, type
Code:
adb push -p OnePlus5Oxygen_23_OTA_029_all_1801292040_d71af3d.zip /sideload
(note: here, we are not using "adb sideload", we are really using "adb push"). In TWRP, click Install, in the file manager go to /sideload and select the OOS zip file. Confirm by sliding. If you get an error, go back to home, click Mount and ensure System is not checked. Then try installing again. If it still does not work, reboot to fastboot, type command again, get to the recovery and install again.
When the zip-file is installed, go home, click "Wipe", "Advanced Wipe" and check both caches and confirm. Then, go home, click "Wipe" and then "Format Data". Then, go home, click "Reboot" and then "System". Your phone will now reboot to Oreo. It will take a long time, but do not turn off the phone. Let it run. On my phone, it took on average 2 minutes for that boot.
You'll be greeted by the "first boot" page. It'll ask you if you want to restore a backup or start anew, choose start anew. Connect to your Wi-Fi network and Google account. Follow all the instructions until you get to the home screen. There, make sure everything works (especially Wi-Fi and fingerprint sensor). Don't save your fingerprints yet, they will be erased afterwards. If everything works, you can continue following these instructions. If not, post a comment down there.
Now that you're at the home screen, go in the settings, About Android and click the build number 8 times to enable Developer Options. Go in there and enable advanced reboot. Then, reboot your phone into fastboot/bootloader using the power button. Type the exact same command as before to start TWRP. Once that you are in TWRP, run the following commands:
Code:
C:\> adb push -p twrp.tar /sdcard/
C:\> adb shell
~ # cd /sdcard
/sdcard # tar xvf twrp.tar
/sdcard # cp /data/misc/wifi/WifiConfigStore.xml /sdcard/
In TWRP, click "Wipe", "Advanced Wipe" and check only the "Data" partition. Confirm. Press home, then "Restore" and choose the backup in the list. Confirm to restore. Back to the terminal, we need to run the following commands otherwise Wi-Fi and fingerprints won't work:
Code:
/sdcard # cp WifiConfigStore.xml /data/misc/wifi/
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
<< WARNING: dangerous command! double check the following line is correct before pressing enter! >>
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
Note: the command above are ran from your PC in an adb shell while the phone is still in TWRP.
Reboot the phone to system and ensure Wi-Fi and fingerprints are still working. Right now you should already see your old home screen and all your apps, but the internal storage isn't there yet. Reboot in fastboot, run the command to get in TWRP.
Once TWRP has booted, run the following commands:
Code:
C:\> adb push -p sd.tar /sdcard/
C:\> adb shell
~ # cd /sdcard
<< WARNING: dangerous command! double check the following line is correct before pressing enter! >>
/sdcard # rm -rf Alarms Albums DCIM Download Movies Music Notifications Pictures Podcasts Ringtones
/sdcard # ls
<< now, look at the list of files that were printed, and rm anything left that is not called "Android" or "sd.tar"
<< WARNING: dangerous command! double check everything is correct before pressing enter! >>
if when you do rm <the thing> it tells you it's a directory, then do: rm -rf <thething>
if there's a folder called SomeFolder, do "rm -rf SomeFolder"
next, run this:
/sdcard # ls
Android sd.tar <-- expected output
/sdcard # mv Android Android_oreo
/sdcard # tar xvf sd.tar
/sdcard # mv Android Android_nougat
/sdcard # mv Android_oreo Android
If you don't have Magisk somewhere on your sd card, download it and upload it using MTP or adb. Then flash it using the Install button. Clear dalvik/cache and reboot to system.
When the phone has booted (again, it might take time), make sure USB debugging is enabled and run the following commands:
Code:
C:\> adb shell
OnePlus5:/ $ su
<< here, you might see a Magisk screen asking for superuser access. Allow. >>
OnePlus5:/ $ cd /sdcard
OnePlus5:/sdcard $ mv Android Android_oreo && mv Android_nougat Android
Now, try some apps and make sure all the data is there (especially games and Netflix/Hulu/etc). If everything is there, and the phone works properly, go back in the terminal and type:
Code:
OnePlus5:/sdcard $ rm -rf Android_oreo
Optionally, start the TWRP app and flash it, it can always be useful. You can also reboot to fastboot to do that.
Now reboot your phone (normal reboot) one last time.
There, working OOS 5.0.2 / Android 8.0.0 phone with no data loss.
Frequently Asked Questions
How long does the whole thing take?
Highly depends on the amount of data you have on your phone. Since it the USB port only supports USB 2.0, it may take 4 or 5 hours in total.
Will doing this void my warranty?
No.
Will I be able to install future OTA updates using the regular download-reboot-flash-twrp procedure?
Yep. Just use the regular method as you would have on Nougat.
Will I lose my data?
If you follow all the instructions, no. Even if you don't follow them, as soon as you have made a backup of /data and internal storage, then no matter how bad you screw up you could always get a working phone back.
I followed the instructions and now my phone doesn't work
Boot in TWRP, wipe everything, reflash.
questions will be added there in the future
Having WiFi and fingerprint issues
You sure it is a good idea to just delete those files? I would have guessed that I need to replace these (nougat version from backup) with the oreo version to have it working just like before the restore.
Code:
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
I'm having the issues with wifi and fingerprints. Neither one is working. I'll try to figure out how to fix this.
@zdimension Thanks for this guide, I don't have time to test it yet, but I have a question
pdluke said:
Code:
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
Click to expand...
Click to collapse
At this point in the procedure, adb shell is still using root (before flashing magisk) ? How is that possible ? Does the adb /sideload preserve root ?
olivier380 said:
@zdimension Thanks for this guide, I don't have time to test it yet, but I have a question
At this point in the procedure, adb shell is still using root (before flashing magisk) ? How is that possible ? Does the adb /sideload preserve root ?
Click to expand...
Click to collapse
These commands should be run while the phone is in TWRP. Also note that adb /sideload is not used here, only adb push.
pdluke said:
You sure it is a good idea to just delete those files? I would have guessed that I need to replace these (nougat version from backup) with the oreo version to have it working just like before the restore.
Code:
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
I'm having the issues with wifi and fingerprints. Neither one is working. I'll try to figure out how to fix this.
Click to expand...
Click to collapse
If you delete them, they will be generated automatically at the next system boot. But you could also make a backup of those three files before wiping /data, store that somewhere, restore Nougat /data and then restore your backup of those three files. The result would be the same.
Note: actually, not exactly. Erasing the first file won't change anything since it's not used anymore in Oreo, but the two other files contain the fingerprint configuration (list of saved fingerprints). So,
Either you remove the files and you have to save your fingerprints again at next boot
Either you restore them from an Oreo backup and you'll get the fingerprints you had saved during the "first boot" procedure when you rebooted the phone right after flashing the OS
But the result is mostly the same: everything works. Deleting the files ensures you get something clean. If you restore from an Oreo backup I can't guarantee the result (as it may interfere with other files from the Nougat backup).
10 bucks to make a script to do this all for me haha.
@zdimension Thanks for the clarification Another thing you might add to the files to download would be Magisk (optionally). In this kind of guide, I've always find it useful to download everything first.
olivier380 said:
@zdimension Thanks for the clarification Another thing you might add to the files to download would be Magisk (optionally). In this kind of guide, I've always find it useful to download everything first.
Click to expand...
Click to collapse
Oops, forgot to add it
I added the link, and also instructions for how to un-root afterwards for those who would want it.
To improve the guide, here are some ideas :
- You should highlight that rm -rf is a very dangerous command, and that it needs to be checked twice (especially the targeted folder)
- It could be useful to use the du -csh command to check the size of a folder (to estimate the backup time for example).
- As a safety measure, one could md5sum the tar file before and after using adb pull
What do you think ?
olivier380 said:
To improve the guide, here are some ideas :
- You should highlight that rm -rf is a very dangerous command, and that it needs to be checked twice (especially the targeted folder)
- It could be useful to use the du -csh command to check the size of a folder (to estimate the backup time for example).
- As a safety measure, one could md5sum the tar file before and after using adb pull
What do you think ?
Click to expand...
Click to collapse
Thanks for the ideas! I updated the post (and I added a changelog at the bottom for future reference).
Followed guide for successful upgrade from 4.5.15 encrypted, unlocked bootloader w/ Magisk root.
One note, after the first complete wipe and flash of the full ROM, it was getting stuck on first boot and never completed. Discovered that I needed to not just wipe the Data partition but Format it in TWRP, to clear out the old encryption I think. Magisk wouldn't install either until I did this.
@debork thanks for the positive feedback (all the merit goes to @zdimension of course)
@zdimension there are many people in the other thread https://forum.xda-developers.com/oneplus-5/how-to/official-oxygenos-4-5-2-7-1-1-ota-t3627003 that tried (unsucessfully) to upgrade from 4.5.15 to 5.0.1, maybe a link to this topic could be useful for them (if it's not too late).
Regarding the
Go back to the TWRP home screen, press "Wipe", "Advanced Wipe" and there check "Dalvik / ART Cache", "Cache", "System", "Data" and "Internal Storage".
Click to expand...
Click to collapse
I think it should be highlighted in red, since it is the actual "clean flash" (AFAIU, correct me if I'm wrong).
if we only have the BL unlocked non root and stock recovery can we only ota without any loss of data ?
debork said:
Followed guide for successful upgrade from 4.5.15 encrypted, unlocked bootloader w/ Magisk root.
One note, after the first complete wipe and flash of the full ROM, it was getting stuck on first boot and never completed. Discovered that I needed to not just wipe the Data partition but Format it in TWRP, to clear out the old encryption I think. Magisk wouldn't install either until I did this.
Click to expand...
Click to collapse
Thanks for feedback, I will add that to the guide (although it worked with just Wipe for me )
zdimension said:
Thanks for feedback, I will add that to the guide (although it worked with just Wipe for me )
Click to expand...
Click to collapse
Have you rooted the 4.5.15 with Magisk or SuperSU (which is not compatible with Oreo anymore) ?
olivier380 said:
Have you rooted the 4.5.15 with Magisk or SuperSU (which is not compatible with Oreo anymore) ?
Click to expand...
Click to collapse
I stopped using SuperSU when it was sold to that shady company. Also, Magisk is better imo.
quick05 said:
if we only have the BL unlocked non root and stock recovery can we only ota without any loss of data ?
Click to expand...
Click to collapse
Official OnePlus support said that nothing is guaranteed if your bootloader is unlocked. But since you're on stock recovery + non rooted, you could always try. But backup everything first. Some people here on XDA have reported that it doesn't work, though.
Just followed your guide with no problems. Thank you very much!! I can confirm also that you need to format data after the wipe otherwise it gets stuck in a bootloop!
Thanks so much for this. I was able to successfully follow the guide and get upgraded to 5.0.1 without losing any data. In fact, I even messed up one step by failing to include the Android directory in the sdcard.tar backup (perhaps that should be more explicit), but it doesn't seem to have affected everything; all of my apps seem to have retained their data.
A few notes:
1. The file size of twrp.tar was ~14GB but when executing the pull command, it recognized it as only ~1.3 GB. As a result, the pull was not complete until it reached over 1000%. All the more reason to do the md5 check.
2. As others stated, I needed to format the data partition, not just wipe it.
3. I might recommend also including a "summary" version somewhere on what this guide does. Scrolling through the guide the first time, it seemed pretty daunting, but really all that you're doing is: backing up data partition and internal storage; wiping device; flashing Oreo ROM; tweaking a few files; and restoring backed up data and internal storage.
Thank you again so much! Glad to finally be on Oreo.
elight3 said:
Thanks so much for this. I was able to successfully follow the guide and get upgraded to 5.0.1 without losing any data. In fact, I even messed up one step by failing to include the Android directory in the sdcard.tar backup (perhaps that should be more explicit), but it doesn't seem to have affected everything; all of my apps seem to have retained their data.
A few notes:
1. The file size of twrp.tar was ~14GB but when executing the pull command, it recognized it as only ~1.3 GB. As a result, the pull was not complete until it reached over 1000%. All the more reason to do the md5 check.
2. As others stated, I needed to format the data partition, not just wipe it.
3. I might recommend also including a "summary" version somewhere on what this guide does. Scrolling through the guide the first time, it seemed pretty daunting, but really all that you're doing is: backing up data partition and internal storage; wiping device; flashing Oreo ROM; tweaking a few files; and restoring backed up data and internal storage.
Thank you again so much! Glad to finally be on Oreo.
Click to expand...
Click to collapse
Thanks for the feedback! I'll add a summary to the guide.
Hi everyone. Got a puzzle for you forensic data recovery guys out there. I am a fool having a bad day today. For the first time in 10 years of screwing with Android, I backed up my ROM with twrp and also backed up my sms and calls, but then managed to somehow forget to copy those files off the internal storage before trying to flash my phone to LineageOS and botching the internal storage. Here is what happened.
0) Before state: Rooted (I think) and for sure running OxygenOS 5.0.3, flashed via TWRP. TWRP version 3.2.1-0. 128 GB OnePlus 3T
1) Backed up System, data, etc to backup on internal storage via TWRP. Booted into OS and backed up SMS and calls to .xml file.
2) Wiped data, system, cache and dalvik/art cache.
3) Downloaded LineageOS w/ microG via zip in TWRP and attempted to install. Received error that I needed to update the modem to at least 9.0.3.
4) Mounted storage from TWRP and copied in Stable9.0.3+Firmware+Modem_OnePlus+3T.zip. Ran the install and was successful.
5) Installed lineage-17.1-20200826-microG-oneplus3.zip via TWRP. Left prompt to install TWRP checked and booted into OS.
6) Upon boot, saw the message: "Decryption unsuccessful" The password that you entered is correct but unfortunately your data is corrupt.
7) Rebooted back into TWRP to mount storage. Could not get anything to mount because it would not accept my decrypt passcode.
8) Figured TWRP bug, flashed twrp-3.4.0-0-oneplus3.img via fastboot. Could decrypt now with passcode but all storage read 0MB
9) Flashed modem back to OnePlus3T_5.0.3-25-05-18-FIRMWARE-flashable.zip via adb sideload method which is what I assume I was running before since that was the OS version. Rebooted
10) Booted back into TWRP, discovered I still couldn't see files. Downgraded TWRP back to 3.2.1-0 via fastboot.
11) Unable to decrypt from TWRP or even install zips. No change at any point in the OS. Flashed TWRP back to 3.4.0.0.
12) Ran adb shell twrp decrypt XXXXXX (where XXXXXX is my pin) which gave me this message:
Attempting to decrypt data partition or user data via command line.
(then it returned to shell so I assume it worked)
13) Ran an ADB pull command (adb pull /dev/block/dm-0 backup.img). There is obviously something there because it downloaded 111 GB worth of .img file (which I'm 99% certain was the amount of space I had used on the phone). Tried to open the .img file with multiple EXT4 programs and they all said the file/partition was corrupt.
14) Tried to wipe data via TWRP. Failed to wipe because it couldn't be mounted. Followed this:
https://************/how-to-fix-twrp-unable-to-mount-storage-internal-storage-0mb/
15) Ended up with MTP mount of 101 GB free space. Everything seems to be lost on the phone at present. No programs I use to recover the data off the MTP mount (just doing a raw file scrape) can see it as a OnePlus in Windows.
After finding a better guide on XDA, it sounds like this is a known issue that happens when you go from 5.0.x to 9.x on this phone where your encryption breaks and you lose all files. Unfortunately I saw that about 15 minutes after step 14...
My next thoughts would be to:
a) downgrade everything (ROM, firmware, etc) and then try to push the 111 GB .img file back to the phone via ADB, then flash a stock OnePlus 5.0.3 ROM over it to see if I could get the internal storage to mount again. Don't even need to boot into the OS--just to get the internal storage mounted again through TWRP.
--OR--
b) convert the .img file to a .raw and then possibly a .vhd and use Recuva or something to scan the image of the phone and see if I can find those .xml files and maybe a twrp backup if I'm lucky. Although if the entire .img contents is still encrypted and not actually just a corrupt android filesystem, I think I am hosed on this option.
I would prefer option a) but I didn't know if anyone had any insight on this. I thought maybe there would be a way for me to get into the filesystem via ADB maybe and clear out some botched encryption cache or keystore/etc in order to get it to mount again. I'm also not sure if you're able to downgrade the firmware or modem either from 9 to 5 (even though I never got errors on downgrade). Thoughts? Am I just royally hosed?
Tried option b. Nothing but garbage. I made a backup of the .img file this time... so still would be willing to try option a) unless anyone thinks that this is impossible.
Edit: yep... may be too late... https://forum.xda-developers.com/on...-9-0-3-5-0-8-firmware-barrier-t3941164/page18
To my knowledge the data partitions content is unrecoverable lost after the reboot from 5) to 6). The img contains the messed up (and still encrypted) dm0 container but without the enctyption footer which is "outside" the dm0 container in sda15 (aka userdata) partition. To decrypt the img you are missing the device dependent key (an encrypzed partition can be decrypted only on the device the enctyption has been done on) as well as the encryption footer (which is messed up after the reboot).
Format data (not only wipe!) and start over.
Thanks for the reply. That makes sense and is what I was afraid of.
I wonder if it would be possible to regenerate the crypto footer by doing something like this to get the HMAC keys and then flashing a rom to recreate the cryptofooter and then editing the key values to reflect the old cryptofooter values based on the KeyMaster extraction via this exploit.
http://bits-please.blogspot.com/2016/06/extracting-qualcomms-keymaster-keys.html?m=1
It might depend on how much of the cryptofooter is left and whether or not 5.0.3 is vulnerable to these CVEs...
I’m not sure if I know enough about the KeyMaster structure to know how to rebuild it even if I were able to retrieve the HMAC keys.
I also know TWRP now supports EDL so perhaps there is something with a Qualcomm tool I could yse to assist.
Obviously this would mean security issues for everyone on these older builds if I am able to regenerate the crypto footer.
Are you aware of adb commands or likewise that I could run to survey the KeyMaster damage?
As stated: to my knowledge the data is lost after the first reboot with the 9.0.x bootloader.
If you search the official los thread (don't remember if 17 or 16) you'll find some detaled postings on restoring the keys after updating the bl from 5.0.x to 9.0.x. The user, who has examined that has failed, but perhaps it helps to shine some light on your questions.