Related
How can I access the /data/ partition?
I was using the touchpad, installed some apps like teamviewer/unified remote/xbmc from the market, and all was going well and then for no reason the tablet froze, CM9 just stopped responding. So I rebooted it (holding the home button and power button was the only way to get it to do anything) but it's stuck at the Cyanogenmod boot animation. It's been on that animation for 5-10 minutes, so I rebooted again, went into clockworkmod and reflashed the latest nightly and gapps. I then rebooted, android booted up, said it was upgrading/optimizing 77 something apps, then it said it was starting the apps, and then it froze at that point, wouldn't proceed. I then reboot and am stuck at the boot animation again. My best guess is that one of the apps I installed is hosing it. I plugged the usb cable in and normally during the boot animation of my phone I can adb logcat but I can't seem to adb detect this device during boot animation. Windows see's it as a MTP device but no filesystem comes up in my computer. So adb logcat and adb shell so I can get into /data/app to wipe out the last few apps I installed won't work, but on my phone I can also adb shell in from CWM. So I rebooted into CWM but when I connect the cable to the laptop it still doesn't see it as an Android ADB device. No adb shell. Not like my phone at all.
So back to my original question, short of wiping /data from CWM, is there anyway for me to get at /data/app to delete some files? I'm curious what's causing this.
You can do an and pull from clockwork mod or have you tried mounting it on the computer from the clockwork mod options?
I've mounted before in clockwork but only sdcard, so that i can put a zip on it to flash. Can you also usb mount the /data partition?
Sent from my SGH-I777 using XDA
If you can reboot into WebOS, I know a way you can mount it and grab what you need.
Boot into WebOS, and connect your tablet to your PC. Then start up Novaterm (C:\Program Files\Palm, Inc\terminal\novaterm.bat). Connect to your tablet.
From there, you'll be at a root prompt. This is good. Type the following commands:
- mount /dev/mapper/store-media /media/card
- mount /dev/mapper/store-cm--data /media/hdd
- cd /media/hdd
- mkdir /media/card/cmdata (this will make a backup folder for you)
From there, you can use basic linux commands to copy files from your data partition to your media card. The ones you should need (forgive me listing the extra commands, even if you know them):
- ls (LS) will give you a directory listing
- cp -r <Folder> /media/card/cmdata (this will backup whatever folder you wanna backup to the cmdata on your card)
- rm -r <folder> (this will remove whatever folder you wanna remove)
- rm <filename> (this will remove whatever file you wanna remove)
I just tested this method, and it allows you to back up whatever you want, as long as you can get into your /data partition. You can do whatever you need to do. Just be careful you don't remove something you don't have to.
Hope this helps you, mate. If so, hit Thanks, if not, drop me a PM and we'll discuss other options.
glitchsys said:
I've mounted before in clockwork but only sdcard, so that i can put a zip on it to flash. Can you also usb mount the /data partition?
Click to expand...
Click to collapse
Yes, you can, but you can only read it from a Linux PC. A Windows system will not be able to access the partition, because it is an ext3 filesystem.
You can also use the "USB mode" used to install CM, but it will run into the same problem. Without Linux, you will not be able to access files.
I would clear the cache, that usually seems to solve these kind of issues. (Of course it could also mess it up further...)
Update 11-26-2012
The issue that this procedure was designed to overcome has been resolved in the latest version of CWM (6.0.1.9 or newer). Please view http://forum.xda-developers.com/showthread.php?t=1781899 for details and download links. If you do not want to upgrade to the latest CWM the directions below will still work but you should really just update.
Background
Due to the multi-user support capabilities of Android 4.2(+) the clockworkmod directory created by ClockworkMod and used to store downloads and backups is being created in /data/media. Unfortunately this directory is not accessible to individual users and therefore backups created in recovery can not be viewed or modified by ROM Manager and vice versa. ROM Manager will look for the directory /data/media/<user number> (e.g. /data/media/0) to which /sdcard is mounted when you are logged in as the user whose number is <user number> (e.g. 0).
Requirements
ClockworkMod Recovery installed and functioning
ROM Manager
ADB installed on your computer and functioning
Willingness to accept that you are on your own and I am not responsible if you mess something up!
Solution
1. Boot into CWM recovery
2. Attach your Nexus 7 to your computers USB port
3. Open an ADB shell using a console from your computer
Code:
adb shell
4. Confirm that you have a clockworkmod directory at /sdcard/clockworkmod (I am using user number 0 for the rest of this procedure, if you wish to use a different user substitute 0 with your user number)
Code:
cd /data/media/0
ls
4a. If you do not see a clockworkmod directory create one with the command below, otherwise proceed to step 5
Code:
mkdir /data/media/0/clockworkmod
5. See if you have a clockworkmod directory at /data/media
Code:
cd /data/media
ls
5a. If you see a clockworkmod directory type the command below, otherwise proceed to step 6
Code:
mv clockworkmod clockworkmod_bak
6. Create the symbolic link
Code:
ln -s /data/media/0/clockworkmod/ clockworkmod
7. If you already had backups in clockworkmod (now clockworkmod_bak) you can move them to the new folder by executing the following command
Code:
mv clockworkmod_bak/backup/ clockworkmod/
mv clockworkmod_bak/blobs/ clockworkmod/
mv clockworkmod_bak/download/ clockworkmod/
WARNING: I tested a backup from recovery and verified the backup was in ROM Manager, renamed the backup in ROM Manager, and confirmed the backup was renamed in recovery. Due to time constraints I did not restore the backup though I can't imagine why it would fail. As mentioned above do this at your own risk. If you do not have a unix/linux experience I would suggest that you wait for someone with experience to follow these steps (I wrote them after I did the procedure so they haven't been vetted) before you attempt it yourself.
Very helpful, i was wondering what they did to mess up clockwork mod
been trying to undo 4.2 for hours as i dont quite like it
truehybridx said:
Very helpful, i was wondering what they did to mess up clockwork mod
been trying to undo 4.2 for hours as i dont quite like it
Click to expand...
Click to collapse
I am on the fence about 4.2 myself and when I couldn't see my 4.1.2 backup I had a panic moment. I have to admit that either 4.2 is growing on me or I am already forgetting how good 4.1.2 was .
sandnap said:
I am on the fence about 4.2 myself and when I couldn't see my 4.1.2 backup I had a panic moment..
Click to expand...
Click to collapse
+1! Panic indeed! LOL
Great tutorial. Thanks and have a happy thanksgiving
Is there a way to do this without using adb? I don't know how to get it working.
It looks like you are just moving all the files from one location to another, right? If thats the case can I just move them with a file explorer? Also where are the back-ups from pre 4.2 stored, I thought it was sdcard/clockworkmod/ Can I move the back-ups from there to a data/media/clockworkmod folder?
StarOrc said:
Is there a way to do this without using adb? I don't know how to get it working.
It looks like you are just moving all the files from one location to another, right? If thats the case can I just move them with a file explorer? Also where are the back-ups from pre 4.2 stored, I thought it was sdcard/clockworkmod/ Can I move the back-ups from there to a data/media/clockworkmod folder?
Click to expand...
Click to collapse
Please see my update to the original post. Once you update to the latest CWM you can copy any existing backups to the new backup and blobs folders under /data/media/clocworkmod. This is discussed in more detail in the thread I linked to.
Code:
mv clockworkmod_bak/backup/ clockworkmod/
mv clockworkmod_bak/blobs/ clockworkmod/
mv clockworkmod_bak/download/ clockworkmod/
This is me just being nit picky, but this looks better:
Code:
mv clockworkmod_bak/* clockworkmod/
I guess for people not confident in their linux abilities that '*' misplaced could be deadly though...anyway thanks for the simple and great workaround/explanation.
Wiping/ Factory resets (in CWM or TWRP) don't erase this data folder? Are we confident in this?
255|[email protected]:/data/media/clockworkmod # mv backup/ /sdcard/clockworkmod/
failed on 'backup/' - Cross-device link
How are you able to move the directories?
This seriously annoys me. Despite the recent surge of newer android devices not containing any sd card slots for secure file storage away from ROM flashing, everything is now embedded. I have had several accidental internal storage deletions containing backups and roms and no the clockworkmod is even harder to locate and backup conveniently through usb/ftp. Thanks for the tut.
TheAtheistReverend said:
Wiping/ Factory resets (in CWM or TWRP) don't erase this data folder? Are we confident in this?
Click to expand...
Click to collapse
Yeah i am wondering that too.it doesnt feel safe for me that the backups are in the system but not in sdcard.
nexus file locations
On the nexus devices (certainly the 7, which has no external sdcard), the only real mounted portions are cache, system and data. Locations like /sdcard/ /storage/emulated/0 etc. are all fuse mounted from /data/media
Supposedly most of the data portion will survive flashing new ROMs and recoveries, everything except unlocking the boot loader, which wipes data f to meet the android security model.
Doesn't feel very safe, though.
amp said:
This seriously annoys me. Despite the recent surge of newer android devices not containing any sd card slots for secure file storage away from ROM flashing, everything is now embedded. I have had several accidental internal storage deletions containing backups and roms and no the clockworkmod is even harder to locate and backup conveniently through usb/ftp. Thanks for the tut.
Click to expand...
Click to collapse
I would use TWRP. I have switched and never looked back. It has quite a few more advanced options that I didn't see with clockworkmod, and overall has be MUCH more reliable.
The clockworkmod recovery issue stated in this thread does not affect TWRP at all.
Thanx for this, it seems to work even for the latest CWM.
Sent with desire from My One
angusc said:
Thanx for this, it seems to work even for the latest CWM.
Sent with desire from My One
Click to expand...
Click to collapse
Iget an error: Read-only File system :crying:
fokus said:
Iget an error: Read-only File system :crying:
Click to expand...
Click to collapse
If doing commands with adb did you remember to use
adb shell
And check that your command line starts with #......
Sent with desire from My One
angusc said:
If doing commands with adb did you remember to use
adb shell
And check that your command line starts with #......
Sent with desire from My One
Click to expand...
Click to collapse
Yes, that is exactly what i did.
I am on ARHD 11.0
S-off
ElementalX 2.2
I'm on ARHD 11.0 as well.
Strange, I followed the OP from step 1 to 6. I couldn't get step 7 to work so I just used my root explorer to copy the directory contents across in to the newly created clockworkmod folder, then deleted the clockworkmod.bak directory and all was good.
Tip: copy and paste each cmd from the OP in to your adb commands window......
Sent with desire from My One
angusc said:
I'm on ARHD 11.0 as well.
Strange, I followed the OP from step 1 to 6. I couldn't get step 7 to work so I just used my root explorer to copy the directory contents across in to the newly created clockworkmod folder, then deleted the clockworkmod.bak directory and all was good.
Tip: copy and paste each cmd from the OP in to your adb commands window......
Sent with desire from My One
Click to expand...
Click to collapse
THX. That is what i did. I also tried it via Terminal on the device - same error: System read only... hmmm
EDIT: Works fine now after e vew reboots. THX
This thread is for brainstorming and developing safe ways of booting our OUYAs to use custom ROMs.
I've developed an initial method that allows OUYA to boot CM 10 from a usb thumb drive that does not require you to mess with any of your internal storage at all! This allows you to keep your OUYA in pristine stock condition, while still enjoying CM 10 from an external thumb drive.
Unfortunately it does require you to setup the thumb drive in a particular way to allow it to be used and this may be difficult for some users without Linux experience. I hope some will find it useful and plan to develop easier methods in the future (probably involving a custom recovery image with the ability to setup the thumb drive for you)
I plan to post more details on how I achieved this so that others can use the knowledge and apply it to other roms or develop improved solutions, but I might work on some other things first. Mainly it involved unpacking/repacking the Android boot.img (google it and there are tutorials about how to do this, I also recommend checking out the "abootimg" program from the Ubuntu packages) and modifying a couple of the init/fstab files.
Note that the zip I am going to link to is not a flashable zip, extract it and follow the instructions in the README which I am including below. Also note the the Google apps are not included - so if you want them you will need to add them yourself. Just be aware things you add to the system partition will need correct permission set.
One of my other ideas I'm going to eventually try is swapping the boot/recovery partitions so that the device normally boots up into recovery and then will have an option to reboot into the recovery partition which would actually boot a full rom for regular use. This may still be dangerous on OUYA though, so is not recommended unless you know what you're doing. I have a Notion Ink Adam which I can boot into APX mode (a low-level nVidia recovery mode that can be used to restore the device in the case of bricking, unfortunately we do not have this level of acces on OUYA), so it will be much safer for me to experiment on it. I haven't really developed on it before though (in fact this OUYA bit is my first major Android development apart from doing/tweaking some CM builds from source myself for some of my devices), so it may take me some time to set stuff up to experiment with it.
Eventually I think we need some form of bootloader for the OUYA. I have read about some very interesting kexec hardboot patches that were developed for the original Nexus 7 that it would be very awesome if we could port to OUYA - that would allow us to boot the patched kernel and have it boot us into kernels/roms stored elsewhere.
Please post any feedback or ideas. Sorry the current method isn't that easy yet and I hope the instructions make sense - hopefully others can also help clarify.
If you'd like to donate to me, I'm certainly not going to turn you away, but keep the other devs in mind 'cause I haven't done that much yet! ( PayPal: [email protected] )
~Troop
ouya-sda123-boot by Trooper_Max
Safely boot your OUYA to CM 10 without messing with your internal storage! (fast thumb drive recommended!)
========
Credits:
========
khanning88 for the initial CM10 Experimental ROM - this is just a repackaging/reconfiguration of it.
mybook4, sonofskywalker3, rayman, professorpoptart for their CWM recovery
And of course the CyanogenMod/ClockworkMod team for the basis of everything!
=========
Contents:
=========
ouya-cm10-system.img:
ext4 system image dumped from CM 10 experimental OUYA rom
ouya-sda123-boot.img:
Android boot.img to boot CM 10 from thumb drive partitions (details below)
==============
Prerequisites:
==============
For this current method, you will need to be able to partition the thumb drive into three ext4 partitions. This probably means you need Linux - if you don't have a Linux system, I highly recommend checking out pendrivelinux.com for methods of booting Linux off of a thumb drive (I recommend Ubuntu 12.04 LTS or whatever version you are comfortable with).
I hope to develop more methods in the future that will be easier than this, but this is the initial method.
=============
Instructions:
=============
1. Partition the thumb drive with three ext4 partitions - note that they must be the first three partitions on the thumb drive, and I would likely just dedicate a thumb drive to this. I recommend using Disk Utility or gparted (usually both available from an Ubuntu thumb drive).
* The first partition will be the data partition and should be the largest
* Note that Android will create the virtual sd card at /data/media
* The second partition will be system (OUYA internal system is 512 M)
* The third partition will be cache (OUYA internal cache is 768 M)
* Feel free to adjust the size of the partitions, but I'd recommend sticking close to the stock sizes
* I've tested this on an 8 GB as well as a smaller 4 GB thumb drive, bigger should not be a problem
2. Note the device name of your thumb drive - It will likely be sdX where X is a letter corresonding to the order it was mounted in - I would expect it to be sdd if it happens to be the fourth drive connected to your machine. The system partition will then be sdX2.
3. Write the ouya-cm10-system.img to the second partition of the thumb drive. It is crucial that the files get copied into the partition with the correct permissions.
* The easiest way to ensure this is to use dd to do a byte for byte copy of the system image directly over the partition, but this is also very dangerous if you type it wrong, so be sure you have the write device name for your thumbdrive.
* I recommend looking in the Disk Utility or gparted, or running "mount", "df -h", and "cat /proc/partitions", to make sure you have an understanding of what drives are what device names before continuing
* Once you are certain of the device name, ie sdX2, where X is the letter for the thumb drive and 2 denotes the second partition which we are using as system, run "dd if=ouya-cm10-system.img of=/dev/sdX2 bs=4M" as administrator (on Ubuntu either by sticking the word "sudo" in front of the command or running "sudo su" first to switch to root)
* This command will not show any output until the end and may take a little while.
4. Clean/fix the filesystem and resize the filesystem to fill the partition.
* Run "esfsck -fp /dev/sdX2" as administrator
* Run "resize2fs /dev/sdX2" as administrator
5. Thumb drive is now ready (the data/cache partitions can be empty ext4, Android will fill them in). Connect it to the OUYA. Through a hub is fine - just be sure it is the only thumb drive connected at boot.
6. Boot the OUYA into fastboot mode. The only real way to do this right now is to first boot up the OUYA normally, then use "adb reboot bootloader" to reboot into the bootloader. You should be able to run "fastboot devices" then and see a device listed.
7. Boot ouya-sda123-boot.img using the command "fastboot boot ouya-sda123-boot.img". You should see it download to the OUYA and it should start booting.
8. Wait patiently. Remember that CM boots slow the first time and depending on the speed of your thumb drive may boot even slower. You can however type "adb devices" to see if it has started the adb daemon. If you don't see a boot animation after a while, you can try running "adb shell" and if you get a permission error, it probably means you didn't flash the system partition correctly, but hopefully all goes well. Note though that once it gets farther into the boot sequence it will turn off the adb daemon so you will lose adb access until it boots up and you can re-enable it.
9. Enjoy CM 10 on your OUYA without having messed with your internal storage! Just be sure not to let your OUYA fall asleep, as it may not be able to wake back up! I'm guessing this is because I was using a hub and so when it falls asleep, the thumb drive essentially gets disconnected and it cannot immediately find it again when it tries to resume. I'd recommend using a wakelock application or power toggle to keep the screen on all the time, etc.
========
For Help
========
Look for us on the XDA Developers Forum under the appropriate threads! Keep in mind that using this method to boot CM 10 may introduce new bugs that would not have occurred using CM 10 the regular way, so be sure to report problems in the appropriate place and mention what methods you used!
Click to expand...
Click to collapse
http://troopermax.com/releases/ouya-sda123-boot.zip
(be gentle - it's 145 MB, I use shared hosting, so feel free to mirror)
md5sum: 49c8e16e27b6deb9d1e8e86363b56f2f
Mirror: http://www.mediafire.com/download/hban76kzeys6ybd/ouya-sda123-boot.zip
~Troop
I'm including some rough developer details here for now about how I did it.
Many thanks once again to the OUYA CWM recovery, I found it insanely useful: http://forum.xda-developers.com/showthread.php?t=2295645
The main part was unpacking/modifying/repacking the boot.img (which contains the kernel/ramdisk/etc)
abootimg helped with this greatly:
http://manpages.ubuntu.com/manpages/precise/man1/abootimg.1.html
Then I followed instructions I found via google for unpacking/repacking the ramdisk:
http://android-dls.com/wiki/index.php?title=HOWTO:_Unpack,_Edit,_and_Re-Pack_Boot_Images
The partitions the OUYA uses are mostly defined in fstab.cardhu:
# Android fstab file.
#<src> <mnt_point> <type> <mnt_flags> <fs_mgr_flags>
# The filesystem that contains the filesystem checker binary (typically /system) cannot
# specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK
# We remount because we first mount as rw in order to generate NVSI symlink. See init.rc for details.
/dev/block/platform/sdhci-tegra.3/by-name/APP /system ext4 ro,remount wait
/dev/block/platform/sdhci-tegra.3/by-name/CAC /cache ext4 noatime,nosuid,nodev,journal_async_commit,data=writeback,nodelalloc,errors=panic wait
/dev/block/platform/sdhci-tegra.3/by-name/UDA /data ext4 noatime,nosuid,nodev,journal_async_commit,data=writeback,nodelalloc,errors=panic wait,encryptable=/dev/block/platform/sdhci-tegra.3/by-name/MDA
Click to expand...
Click to collapse
I found that I could use fastboot to boot into the CWM recovery and then use adb pull to dump images of the internal OUYA partitions, using the paths above, ex. "adb pull /dev/block/platform/sdhci-tegra.3/by-name/APP ouya-system.img" to dump the system partition. I did a regular clockwordmod backup first, but adb pull was useful here for pulling raw images.
So after backing up my OUYA stock system, I actually did flash the CM10 rom so that I could dump the partitions to my computer using adb. The main reason I did this was to ensure the system partition gets generated with the correct permissions. I didn't boot into CM10 though, though I was tempted. Instead I restored back to my backup to put my OUYA back in stock condition.
With the system image dumped into a file, I used resize2fs to shrink the file to the minimum size (resize2fs -fM ouya-system.img) This probably wasn't absolutely necessary since it would compress down in the zip, but this allows it to be written to smaller partition sizes.
http://manpages.ubuntu.com/manpages/precise/en/man8/resize2fs.8.html
Here is what I modified the fstab.cardhu to:
# Android fstab file.
#<src> <mnt_point> <type> <mnt_flags> <fs_mgr_flags>
# The filesystem that contains the filesystem checker binary (typically /system) cannot
# specify MF_CHECK, and must come before any filesystems that do specify MF_CHECK
# We remount because we first mount as rw in order to generate NVSI symlink. See init.rc for details.
/dev/block/sda2 /system ext4 rw wait
/dev/block/sda3 /cache ext4 noatime,nosuid,nodev,journal_async_commit,data=writeback,nodelalloc,errors=panic wait
/dev/block/sda1 /data ext4 noatime,nosuid,nodev,journal_async_commit,data=writeback,nodelalloc,errors=panic wait
Click to expand...
Click to collapse
Pretty straightforward, except that I removed the encryptable part on /data (not sure how that bits work and if I might have broken it) and made system read/write.
Also note, that I removed the initial read/write mounting of system in init.cardhu.rc since I was having trouble with it mounting from there (I'm guessing because my usb hub/thumb drive weren't yet available when that tried)
Here's the relevant part of init.cardhu.rc:
Before:
on fs
setprop ro.crypto.tmpfs_options size=128m,mode=0771,uid=1000,gid=1000
setprop ro.crypto.umount_sd false
# PLEASE DO NOT REMOVE NVSI SYMLINK! IF CHANGES ARE NEEDED PLEASE ENSURE THAT NVSI SYMLINK IS ALWAYS CREATED.
# Mount system to allow NVSI symlink
mount ext4 /dev/block/platform/sdhci-tegra.3/by-name/APP /system rw
# Create NVSI filter symlink
symlink /data/data/com.nvidia.NvCPLSvc/files/com.nvidia.nvsiutilv1.xml /system/etc/permissions/com.nvidia.nvsiutilv1.xml
mount_all /fstab.cardhu
#chmod for OUYA parameters
chmod 0644 /dev/block/mmcblk0p5
Click to expand...
Click to collapse
After:
on fs
setprop ro.crypto.tmpfs_options size=128m,mode=0771,uid=1000,gid=1000
setprop ro.crypto.umount_sd false
# PLEASE DO NOT REMOVE NVSI SYMLINK! IF CHANGES ARE NEEDED PLEASE ENSURE THAT NVSI SYMLINK IS ALWAYS CREATED.
# Mount system to allow NVSI symlink
#mount ext4 /dev/block/sda2 /system rw
mount_all /fstab.cardhu
# Create NVSI filter symlink
symlink /data/data/com.nvidia.NvCPLSvc/files/com.nvidia.nvsiutilv1.xml /system/etc/permissions/com.nvidia.nvsiutilv1.xml
#chmod for OUYA parameters
chmod 0644 /dev/block/mmcblk0p5
Click to expand...
Click to collapse
Basically I commented out the mount command - before it was mounting system read/write for a moment to insert the symlink and let the filesystem update itself before remounting read-only in the fstab. The way I do it now it is simply mounted read/write from the fstab and left that way (some may consider this unsafe, but it is good for development... not too hard to remount it read-only if you so desire) I also shifted the symlink command down below the mount all command so it would hopefully still work... not sure what that was for, but they made it sound really important! XD
I also commented out this part about the usb drive since it doesn't really make sense anymore:
# Mount usb drives as /usbdrive. Generally usb drives are formatted with FAT
# filesystem, so we support FAT as of now.
#on device-added-/sys/block/sda
# mount vfat /dev/block/sda /mnt/usbdrive
Click to expand...
Click to collapse
I think those were all the changes I made before repacking it, though it did take me 5 attempts to get a working ramdisk so I may have lost track of something at some point. Those should be the key changes at least.
I also initially used raw dumps of the data and cache partitions on my thumb drive as well to make sure it would work. Then I tested to make sure it would work if they were empty ext4 partitions, which they do - Android fills them in as necessary. So only the system image was necessary to include. (and of course the boot.img for fastbooting)
This was a little bit harder than I had anticipated because I'd previously only looked into this type of thing for Gingerbread devices which aren't set up the same way. I'd kinda been hoping to get a solution where the user could just drop a system.img and data.img onto a fat32 usb thumb drive and the boot.img would loop mount them off the thumb drive. I haven't tried this for Gingerbread, but it seems pretty straightforward. It's not straightforward anymore here since fat32 sdcards aren't used anymore. Being familiar with Linux though, I am happy they are using ext4 everywhere and using the same space alloation for /data and /sdcard so you don't have to worry about which one to make big, but it does make some things more complicated for more average users. It's probably still possible to get the OUYA to do what I wanted, but more complicated and not sure it's worth pursuing over other methods which may be more fruitful.
Hope that's enough details to enlighten some people who may be wondering about the methods used here. My goal is to not only develop new ways of booting the OUYA, but also to share how I accomplished things so that others can learn from it.
~Troop
Is this similar to the Nook HD + CM10 boot?
---------- Post added at 05:07 AM ---------- Previous post was at 04:58 AM ----------
Wait that was blank a second ago. I am downloading now. I will mirro with my mediafir stuff if I can get it to work.
kairnage said:
Is this similar to the Nook HD + CM10 boot?
---------- Post added at 05:07 AM ---------- Previous post was at 04:58 AM ----------
Wait that was blank a second ago. I am downloading now. I will mirro with my mediafir stuff if I can get it to work.
Click to expand...
Click to collapse
Sorry, I was setting up the posts...
I'm not sure this is similar to any solution currently used on other devices, but I do know that other developers have used similar methods to develop their roms.
I don't have a Nook and haven't looked into what they use too much, but we may very well be able to learn from their work.
~Troop
So I dug up an old nvflash package from my Notion Ink Adam and tested it to make sure it still works (with the Adam in APX mode it allows me to directly flash the device with a new partition layout and flash the partitions directly). After verifying I had it setup correctly, I swapped the boot and recovery images and ran it again and sure enough... now when I boot my Adam normally, the recovery rom comes up, and when I boot my Adam into recovery it loads the normal rom - so the theory seems to work.
So I think the next thing I will try is building a new recovery for OUYA from source and adding an option to reboot into "recovery" to the menu, allowing it to be stored on the boot partition and the real boot.img to be stored on the recovery partition, allowing you to boot the OUYA and it comes up in recovery, then you select the new reboot option and it would reboot into "recovery" which would load a full rom. You could then safely flash boot images to the recovery partition without fear of bricking your device.
I think it should work nicely, but not sure how long it will take me since I need to set my build machine back up.
For those who are confused, read more about how these ideas started below!
~Troop
Trooper_Max said:
I think this is a great idea that needs more attention:
mybook4 said:
Devs, I propose the following to get rid of the potential brick risk:
Since we can't get into recovery manually (via HW buttons), let's reverse the role of SOS (recovery) and LNX (kernel). Since LNX is the booting kernel partition, let's flash recovery there and flash the kernel to the recovery partition. I believe we could do this by modifying the fstab and having our updater-scripts flash to the appropriate partitions.
From a cold start, a user will enter recovery (a minor inconvenience for safety). Depending on how we modify the recovery.fstab, getting to the ROM could be as simple as pressing power twice (recovery does a reboot system now and its fstab has the system actually reboot to the recovery partition, which is the ROM's kernel).
Definitely not straightforward, but should prevent bricks. Thoughts?
Sent from my Nexus 7 using xda premium
Click to expand...
Click to collapse
But then I'm definitely biased, because I had the same idea:
Trooper_Max said:
I agree with everyone it would be nice if we had a hardware means to boot into some kind of recovery or flashing mode to flash stock images.
But I was trying to think if there is anything we can do if we don't get that... (guess that makes me a pessimist >_>)
Let me preface this with don't try anything I'm describing here unless you know what you're doing, these are half-baked thoughts of someone who doesn't even have their OUYA yet:
What if we could swap the boot and recovery partitions? (I haven't received my OUYA yet, so I'm not even aware of what partitions it has, so assuming it has them, and yeah, it wouldn't be quite that simple)
Basically what I mean is when the device tries to boot normally, it would boot up a stable recovery rom. (ie the boot partition that normally has the kernel and then loads the rest of the rom instead just has a recovery rom)
Then you have the recovery partition be what boots the full rom. That way you only chance bricking your device when you flash the boot partition, which if we can get a stable recovery rom that works for this, wouldn't be often. We would probably want to modify the recovery to have a 3 second/configurable timer where if you don't do anything it boots into the "recovery" partition which would boot up the full rom.
Basically the boot partition becomes a new recovery rom which gets used like GRUB to boot into the "recovery" partition which boots up your actual rom, or maybe it could also boot from USB or netboot or whatever...
Pretty much what we need is a solid bootloader, sound about right? Let me know if any of this makes sense/doable or if I'm entirely off base here! I don't know if I will have the time to try any of this when I do get my OUYA, but wanted to share some ideas, please proceed at your own risk!
EDIT: Alternatively, the boot partition could be left stock and the recovery partition could be used as a bootloader to boot into USB or other options for loading roms without messing with the stock boot experience and risking bricking the device. ie in this configuration if you boot your device normally it would be stock, if you reboot into "recovery" it would load up a custom rom. Or instead of a custom rom, a custom recovery with bootloader capabilities.
~Troop
Click to expand...
Click to collapse
And I do plan to try this idea out eventually, but I've got some other ideas I intend to try out first.
The first one I hope to have done by the end of this weekend (hopefully sooner though) is to reconfigure this experimental CM build to be usable without touching any of the internal storage space. I plan to have a modified boot.img that will load system/data/sdcard folders off a usb thumb drive. This method would be completely safe, though a little inconvenient because you would need to use adb to reboot the OUYA into bootloader mode so that you can then fastboot it to load the modified boot.img, but after that it could be disconnected and would be running using only external storage. You're essentially using a computer to jumpstart your OUYA into CM10, while leaving everything on the OUYA itself in pristine stock condition. Then I plan to try out arm versions of adb/fastboot so I can use my tablet or possibly phone (USB OTG) to jumpstart the OUYA instead of having to rely on my computer all the time.
I think this will be an extremely safe way of using external ROMs until we can get some kind of special bootloader figured out.
So if anyone is thinking about trying CM10, but leery of messing with their OUYA, just wait another week or so and I should have a safe, non-intrusive solution worked out!
I welcome any input/thoughts on these ideas! And if anyone knows how to do the things I'm describing and wants to beat me to the punch, feel free and run with these ideas - I won't mind as long as you share your work and give me a little credit if any credit is due!
~Troop
Click to expand...
Click to collapse
I just ordered a 64GB PNY off Amazon. Once I get it Tuesday I am going to try this. I am uploading to my mediafire now to mirror.
Trooper_Max said:
So I dug up an old nvflash package from my Notion Ink Adam and tested it to make sure it still works (with the Adam in APX mode it allows me to directly flash the device with a new partition layout and flash the partitions directly). After verifying I had it setup correctly, I swapped the boot and recovery images and ran it again and sure enough... now when I boot my Adam normally, the recovery rom comes up, and when I boot my Adam into recovery it loads the normal rom - so the theory seems to work.
Click to expand...
Click to collapse
After some more playing with my Adam, I noticed that when I reboot into "recovery" which I have booting a CyanogenMod rom and then I reboot and from the reboot menu select "Reboot", it reboots into CyanogenMod again (even though I didn't expect it to). From the reboot menu I tell it to reboot into "recovery" and it boots into CM again, which that part makes sense, since I have the CM boot image stored in the recovery partition. So I got to thinking that maybe when you tell it to reboot into recovery it sets an SOS signal (the recovery partition is sometimes refered to as SOS) and when you boot a normal rom from recovery, it never clears that signal because it isn't normally booting that way. Whereas a true recovery rom knows it needs to clear that signal so you don't get stuck booting into the recovery rom all the time.
I was close - I scanned the recovery source on CM's github and found this bit:
https://github.com/CyanogenMod/android_bootable_recovery/blob/jellybean/recovery.c said:
// Reset to normal system boot so recovery won't cycle indefinitely.
struct bootloader_message boot;
memset(&boot, 0, sizeof(boot));
set_bootloader_message(&boot);
Click to expand...
Click to collapse
So there seems to be a "bootloader message" that tells it whether to boot from the boot or recovery partition. I was going to say we need to be careful how we reboot the device and probably modify the custom rom to be able to reset the boot message, but I just realized we can do better and use this to our advantage. We could set our custom rom to always clear the boot message so that you have to go through the recovery rom each time to boot the system normally, but we could also have the option of selectively resetting the bootloader message - purposefully get ourselves stuck rebooting into "recovery" all the time because that is our normal rom, then from our normal rom have the option of resetting the bootloader message when we want to go into our recovery image stored on the boot partition.
Something to think about - I don't like getting stuck out of my recovery image either, so we gotta be sure we can reset the bootloader message when we need to... I'm going to play with this some more tomorrow, but not sure how far I will get.
EDIT: More info on the bootloader message - seems it is stored on the misc partition. The more I think about this too, the more I'm not sure swapping the boot/recovery is much safer - if we had them swapped, but then the rom gets stuck in a boot loop before it can reset the bootloader message, we'd be stuck always booting into "recovery" and into the boot loop, so we'd be in just as much trouble as if we'd gotten our boot partition to cause boot loops. So whereas normally flashing the boot partition is potentially dangerous, in this scenario flashing the recovery partition would be potentially dangerous - you gotta be sure it's going to be able to reset the bootloader message. We could try to mitigate this by modifying the boot process to reset the bootloader message very early in the boot process (ie before the potential bootloop) so that if we get stuck, the next time we reboot, we'll reboot normally into the recovery rom.
But then we could also do just the opposite of that without swapping the boot/recovery partitions - very early in the boot process, modify the bootloader message to tell the bootloader to boot into recovery - that way if we get stuck the next time we boot up will be into recovery. Then later when the system successfully boots, we could reset the bootloader message so that after a successful boot the next boot will be another normal one instead of into recovery. If we get this right a failed boot would automatically take us to recovery on the next boot, while a successful boot would boot normally on the next boot. So this would be a way to build in some "brick-protection" into the boot.img to make it safer to flash. It'd probably be safest to not reset the bootloader message until the user actually selects the shutdown option - that way any abnormal reboot would cause it to come up into recovery mode.
So I'm not sure swapping boot/recovery partitions really buys us anything anymore, since it also swaps which partition is potentially dangerous to flash. The solution in either case is going to be to cleverly manipulate the bootloader message as described above. So we might just want to not swap them to minimize confusion and focus instead on building "brick-protection" like this into our boot images.
https://github.com/CyanogenMod/android_bootable_recovery/blob/jellybean/bootloader.h said:
/* Bootloader Message
*
* This structure describes the content of a block in flash
* that is used for recovery and the bootloader to talk to
* each other.
*
* The command field is updated by linux when it wants to
* reboot into recovery or to update radio or bootloader firmware.
* It is also updated by the bootloader when firmware update
* is complete (to boot into recovery for any final cleanup)
*
* The status field is written by the bootloader after the
* completion of an "update-radio" or "update-hboot" command.
*
* The recovery field is only written by linux and used
* for the system to send a message to recovery or the
* other way around.
*/
struct bootloader_message {
char command[32];
char status[32];
char recovery[1024];
};
/* Read and write the bootloader command from the "misc" partition.
* These return zero on success.
*/
int get_bootloader_message(struct bootloader_message *out);
int set_bootloader_message(const struct bootloader_message *in);
Click to expand...
Click to collapse
kairnage said:
I just ordered a 64GB PNY off Amazon. Once I get it Tuesday I am going to try this. I am uploading to my mediafire now to mirror.
Click to expand...
Click to collapse
Sounds nice - I'm tempted to get a nice big new thumb drive myself... I don't have any that big >_>
~Troop
I've been following your think tank and I agree with your latest idea 100%
Sending a message for bootloader to always boot into recovery if the system wasn't shut down normally is clean and simple.
I am curious though what about development, with nvflash requiring encryption of commands first mistake will render your ouya unusable.
dexter84 said:
I've been following your think tank and I agree with your latest idea 100%
Sending a message for bootloader to always boot into recovery if the system wasn't shut down normally is clean and simple.
I am curious though what about development, with nvflash requiring encryption of commands first mistake will render your ouya unusable.
Click to expand...
Click to collapse
Glad you agree with my new idea!
As far as development, yeah, that is why I used my Notion Ink Adam for some of the testing that I did.
As far as developing for OUYA, I don't think we are limited by not having nvflash, we just have to be more careful. Personally, if I were developing anything, I would use my above methods to fastboot boot images that load everything else from a thumb drive at least until whatever I'm developing becomes stable enough that I want to use it more permanently. Then I would be very careful about how I flash it.
Heck, if you wanted to be really safe, you could flash recovery images to both the boot and recovery partitions and just use external boot methods
In the long term though, I hope we can develop a bootloader based on the kexec hardboot patches for the original Nexus 7, which would allow booting completely off boot images stored in other locations, loading partitions from other locations. Other locations could even just be the virtual sd card if you don't want to use external media, but at that point we'd have a safe extra bootloader stored on the boot partition.
http://forum.xda-developers.com/showthread.php?t=2104706
Hope that makes sense.
~Troop
I was not aware that any content of RAM can survive a reboot, hard kexec booting custom OS from USB or image is definitely a safe solution, looks like all the pieces are there.
Wish I had something more than ouya alone, since you have your Adam are you going to try to develop something ?
dexter84 said:
I was not aware that any content of RAM can survive a reboot, hard kexec booting custom OS from USB or image is definitely a safe solution, looks like all the pieces are there.
Wish I had something more than ouya alone, since you have your Adam are you going to try to develop something ?
Click to expand...
Click to collapse
I do intend to, but I don't have a great deal of time to devote to Android development, so not sure how long it will take me. I'm trying to find what time I can though. This past weekend/last week I had a bit more than usual.
~Troop
Trooper_Max said:
http://troopermax.com/releases/ouya-sda123-boot.zip
(be gentle - it's 145 MB, I use shared hosting, so feel free to mirror)
md5sum: 49c8e16e27b6deb9d1e8e86363b56f2f
~Troop
Click to expand...
Click to collapse
Here is a mirror for you.
http://www.mediafire.com/download/hban76kzeys6ybd/ouya-sda123-boot.zip
I tried this on my new 64GB and it wouldn't boot, but it's a brand new card so it might be the problem. I'll try a known good card later.
kairnage said:
Here is a mirror for you.
http://www.mediafire.com/download/hban76kzeys6ybd/ouya-sda123-boot.zip
I tried this on my new 64GB and it wouldn't boot, but it's a brand new card so it might be the problem. I'll try a known good card later.
Click to expand...
Click to collapse
Sorry you had trouble - a couple other things to be wary of:
Have only the one thumb drive connected, at least until it boots - the boot.img looks for the data partition as sda1, the system partition as sda2, and the cache partition as sda3. If some other storage device is connected, it might be labeled sda by the OUYA depending on the order it finds them in and then the thumb drive might become sdb or so on.
It worked for me through a hub, but it may also be worth trying it directly if you have trouble.
Guessing you did this, but give it some time, booting from thumb drive especially on the first boot is slower. It's not too long, but it was long enough that for a moment I thought I'd failed again. Of course the thumb drive I was using wasn't built for speed, so YMMV.
If you're still having trouble - even on my failed attempts at boot images for this (it took me 5 tries at getting the ramdisk to get it bootable), I was still able to connect to it using adb. I ran adb on Linux, running adb as root to bypass all that udev configuration stuff - it's just easier, and if the OUYA/device failed to mount the system partition, it might not match the configuration you expect anyway. If you're able to run "adb shell" but it gives you a permission error about not being able to access the shell binary on the system partition, that probably means you didn't setup the system partition correctly - it's crucial that it get set up with the right permissions, which is why I provided the image so it could just be written directly with dd.
I'm going to try to put together a recovery image that can partition the thumb drive for you and flash an arbitrary update.zip to the thumb drive, but not sure when I will be able to have that ready. The initial version will not be able to patch the boot.img though, so that will still need to be modified like the one I have provided, but maybe in a future version I can get it to also modify the boot.img for loading the information from the thumb drive.
Let me know if you continue to have issues or if you do get it working! And if anyone else has got it working, please post so I can at least know someone got it working. It worked for me, and I don't see why it shouldn't work for anyone else, but if there is something conflicting I want to figure it out.
EDIT: I just remembered when I was looking into the swapping boot/recovery stuff, I noticed in the recovery source that there already are some tools that might be able to help partition the thumb drive and fix permissions included in the recovery source:
https://github.com/CyanogenMod/android_bootable_recovery/tree/jellybean/utilities
Tonight I'll see if I can put together some new instructions using those tools that might be easier and less error-prone. I'm hoping those tools can be applied to the thumb drive anyway, I'll try and figure that out tonight.
~Troop
Trooper_Max said:
Sorry you had trouble - a couple other things to be wary of:
Have only the one thumb drive connected, at least until it boots - the boot.img looks for the data partition as sda1, the system partition as sda2, and the cache partition as sda3. If some other storage device is connected, it might be labeled sda by the OUYA depending on the order it finds them in and then the thumb drive might become sdb or so on.
It worked for me through a hub, but it may also be worth trying it directly if you have trouble.
Guessing you did this, but give it some time, booting from thumb drive especially on the first boot is slower. It's not too long, but it was long enough that for a moment I thought I'd failed again. Of course the thumb drive I was using wasn't built for speed, so YMMV.
If you're still having trouble - even on my failed attempts at boot images for this (it took me 5 tries at getting the ramdisk to get it bootable), I was still able to connect to it using adb. I ran adb on Linux, running adb as root to bypass all that udev configuration stuff - it's just easier, and if the OUYA/device failed to mount the system partition, it might not match the configuration you expect anyway. If you're able to run "adb shell" but it gives you a permission error about not being able to access the shell binary on the system partition, that probably means you didn't setup the system partition correctly - it's crucial that it get set up with the right permissions, which is why I provided the image so it could just be written directly with dd.
I'm going to try to put together a recovery image that can partition the thumb drive for you and flash an arbitrary update.zip to the thumb drive, but not sure when I will be able to have that ready. The initial version will not be able to patch the boot.img though, so that will still need to be modified like the one I have provided, but maybe in a future version I can get it to also modify the boot.img for loading the information from the thumb drive.
Let me know if you continue to have issues or if you do get it working! And if anyone else has got it working, please post so I can at least know someone got it working. It worked for me, and I don't see why it shouldn't work for anyone else, but if there is something conflicting I want to figure it out.
EDIT: I just remembered when I was looking into the swapping boot/recovery stuff, I noticed in the recovery source that there already are some tools that might be able to help partition the thumb drive and fix permissions included in the recovery source:
https://github.com/CyanogenMod/android_bootable_recovery/tree/jellybean/utilities
Tonight I'll see if I can put together some new instructions using those tools that might be easier and less error-prone. I'm hoping those tools can be applied to the thumb drive anyway, I'll try and figure that out tonight.
~Troop
Click to expand...
Click to collapse
I could get into ADB as well but was only seeing the stock system. Actually now I think it was the adapter I was using. At first I assumed it was the microsd because it would hang on boot. But after trying a known good card it did the same. I remembered having issues with an adapter that has the indicator light to show it was connected on a tablet before. I tried a generic Kingston adapter and it sees the cards now. Of course I had already wiped them back to stock partitions. LOL Oh well, start again tomorrow.
For anyone who doen't have a Linux box handy or an extra thumb drive to boot a live USB, here are some quick instructions on how to partition the thumb drive from the OUYA itself. You can connect the thumb drive either before or after booting, it doesn't really matter. I'm using all fastboot/adb commands here so they are easy to recognize. If you're knowledgeable about how to use adb/fastboot, feel free to execute the commands however you're comfortable.
Type all the adb/fastboot commands as specified - if it doesn't begin with adb or fastboot, it isn't a command. All /dev paths you type into the commands should be related to /dev/block/sda (your thumb drive), not any internal partitions or you could mess up your OUYA!!!
Boot up the OUYA normally, then reboot into the bootloader:
adb reboot bootloader
Boot CWM using fastboot:
fastboot boot OuyaCWMrecovery6.0.3.2.img
Print partition information for the thumb drive:
adb shell parted /dev/block/sda print
Model: USB Flash Memory (scsi)
Disk /dev/block/sda: 8128MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32.3kB 6000MB 6000MB primary ext4
2 6000MB 7000MB 1000MB primary ext4
3 7000MB 8000MB 1000MB primary ext4
Click to expand...
Click to collapse
(note in the example output above I already have it partitioned)
Here I was using an 8 GB thumb drive, you can see the size where it says 8128MB.
Delete Partitions from thumb drive: (be sure you don't have anything important on your thumb drive)
adb shell parted /dev/block/sda rm 1
adb shell parted /dev/block/sda rm 2
adb shell parted /dev/block/sda rm 3
(note that you only need to delete partitions that showed up in the table when you printed it, here I deleted 1,2,3, since I had 3 partitions above and the parted commands will say you may need to update your fstab, but that is not relevant here so don't worry about it)
Create 3 primary partitions (adjust the sizes as necessary):
adb shell parted /dev/block/sda mkpart primary 1 6000
adb shell parted /dev/block/sda mkpart primary 6000 7000
adb shell parted /dev/block/sda mkpart primary 7000 8128
Check your paritions again:
adb shell parted /dev/block/sda print
Format your partitions as ext4:
adb shell mke2fs -t ext4 /dev/block/sda1
adb shell mke2fs -t ext4 /dev/block/sda2
adb shell mke2fs -t ext4 /dev/block/sda3
Push the cm10 system image onto the device: (this will take a little while)
adb push ouya-cm10-system.img /tmp/ouya-cm10-system.img
Check that it copied correctly:
adb shell ls -l /tmp
-rw-rw-rw- 1 root root 260308992 Jul 27 2013 ouya-cm10-system.img
-rw-rw-rw- 1 root root 3587 Jan 1 02:12 recovery.log
Click to expand...
Click to collapse
(Note the size... I'm pretty sure copying it /tmp put it into the ramdisk, but our OUYA has enough ram for this)
Write the cm10 system image onto /dev/block/sda2: (this will take a little while)
adb shell dd if=/tmp/ouya-cm10-system.img of=/dev/block/sda2 bs=4M
Check/fix the filesystem:
adb shell e2fsck -fp /dev/block/sda2
Resize the filesystem to fill the partition:
adb shell resize2fs /dev/block/sda2
(Unfortunately, it looks like resize2fs is not included in CWM, so this step would need to be done on the computer)
At this point, the thumb drive should be good, so you just gotta reboot back to the bootloader and boot the boot image:
Reboot to the bootloader again:
adb reboot bootloader
Boot the modified boot image I provided:
fastboot boot ouya-sda123-boot.img
Not sure if that's necessarily easier or harder to follow than the other instructions... I'd probably rather do the partitioning on a desktop with GUIs, but for some who don't have a Linux box handy it might be easier to use the OUYA as a little Linux box
I'm sure you noticed this method still used dd to write the system image, just run from the OUYA instead. I looked into just extracting the system files from the CM10 zip onto the OUYA, but fixing the permissions was not as easy as I thought it would be - the /system partition permissions are actually fixed by the updater script as referred to here:
http://fokke.org/site/content/howto-create-android-updatezip-package
It's actually a scripting language and the script also creates symlinks and does other stuff.
You'd have to extract/modify the updater script to not use the internal partitions and then execute it (I think using the edify program)... still less straightforward than using dd if you have the image handy like I gave you... But I may look into this as another option. The knowledge will certainly be relevant when I look to build a custom CWM recovery to allow you to flash zips to the thumb drive instead of the internal partitions.
~Troop
Thanks to csonger for feedback, which may be handy if you have system partition issues:
csonger said:
I tried both methods (under linux and with ouya as well), but neither of them worked for me. My 8 gig pen couldn't boot.
The first thing is that the system image is 256MB and the tutorial says the System partition needs 512MB or 1GB.
After the boot failures I checked the pendrive. The second (system) partition was 256MB. It is ok when we use dd, but it was full. 100% usage.
First I run the e2fsck command to check for filesystem problems and it found some inode issues on it.
Following the fs fix the resize2fs command did the job and I had got a 1 gig System partition.
After that my pendrive could boot the cyanogenmod image without problems.
May be someone else has the same problem as me. Please share with them.
Thanks for your work:
csonger
Click to expand...
Click to collapse
I didn't have those issues myself for some reason, but I'll investigate and see what instructions I need to update!
Thanks again for the feedback!
~Troop
Finally got my Linux partition back up and running so I can give this a go again.
It's been trying to boot CM for about 15 minutes now, ADB does see it, but I think it might no like the partitions I did.I did 2GB for system and 1GB for cache. I will repartition and try again.
kairnage said:
It's been trying to boot CM for about 15 minutes now, ADB does see it, but I think it might no like the partitions I did.I did 2GB for system and 1GB for cache. I will repartition and try again.
Click to expand...
Click to collapse
That should be fine, I would think... If you continue to have trouble, try running e2fsck and resize2fs on the system partition, as csonger suggested. (In case you didn't already)
I was confused when I came up with the instructions that it worked without me doing resize2fs since I figured that would be needed to expand the filesystem to fill the partition, but somehow it worked anyway and even looked like it was expanded, but I'd probably just forgot to refresh my view after dd.
Note that you can run the commands with just the partition as an argument: (I added -fp to e2fsck to automatically find/fix problems)
e2fsck -fp /dev/sdX2
resize2fs /dev/sdX2
(Where X is the letter for your thumb drive)
Without a size specified, resize2fs resizes the filesystem to fill the partition.
Hope that helps!
EDIT: I went back and fixed the instructions with this addition now that I've tested it a little more. Unfortunately, I noticed that CWM does not include resize2fs and I didn't spot any other programs handy within CWM to do the trick, so you can't quite do this part on the OUYA. I will look into alternate instructions or building my own CWM that includes resize2fs or just plain automates this more.
~Troop
Hi developers. I am sorry for posting this. I spent the last week trying to solve it by myself with no hope. This is my second time installing something on a phone, but it is my only phone, so I beg anyone for a help...
-What I did:
Some days ago I downgraded to this ROM C5503_10.1.1.A.1.310_GLOBAL-LTE.ftf to use DoomLord rooting script. I did it with flashtool for linux and I applied his .bat step by step in the terminal since windows would not detect my phone.
It worked. I had root for some days, but I was still annoyed by sony default android. So I decided to install Cyanogenmod.
I unlocked the device with sony official system and wen't straight to this instructions, before the first reboot
wiki.cyanogenmod.org/w/Install_CM_for_yuga
I booted succesfully in CWM, followed everything as it says there. But that's where weird things happened:
-The problems:
-The backup
I tried, it wouldn't mount /sdcard. Since I don't understand much about this, I thought it was normal. The next choice was sdcard1, I backed up there. Or so I thought...
-The factory reset
I factory reset, again, not mounting sdcard. Here is the message that shows when I try this now:
can't mount /data!
Error mounting /sdcard.android_secure
Skipping format...
Data wipe complete.
Since it said it is complete, I went on installing the zip file from my sdcard1. Both CM 10.2.1 (dogo, the right one for my phone) and the appropriate GAPPS.
Now it loops on the CM loop animation and I have to remove the battery...
-The restore problem
It still boots on the recovery mode. So I tried recovering my backup from sdcard1. But the image name is 1970.01.01.00.03.16. And it says "md5 mismatch"
I tried flashing again the stock rom with flashtool. The proccess goes on but nothing happens. I still have CWM and the boot loop.
I read elsewhere someone with a similar problem who solved using sony "emma" software. I installed it, it won't even recocnize my phone.
It recocnizes that there is a phone, but don't know which one.
But that has alway been the case with windows. I haven't been able to do anything in windows other then accessing the sdcard (when the phone worked).
Is there something I can do? I imagine that somehow, for some reason, the /data and /sdcard partitions got corrupted. I imagine I would need to repartition this and install again, but I have no idea how this happens on phones...
I can mount /system /cache and /storage/sdcard1. just /data I can´t. Says "error mounting /data"
This is my only phone and a vey recent $400 thing. I was very stupid to do that withouth a replacement and really need this phone. I greatly appreciate any help...
I found this post forum.cyanogenmod.com/topic/6433-solved-messed-up-partitions-on-internal-storage/ searching the internet. Is it possible that this would solve my problem? or would it finish bricking the phone?
Here's what you'll need:
Working recovery, basic knowledge of adb & the shell
Parted (download here)
stock PB31IMG.zip
Note also that I had run unrevoked forever (so my phone was S-OFF) ... I'm not sure if that's required or not.
So, grab parted from the link above. Now you need to extract the individual binaries from the .zip (the 6 files in the sdparted folder within the zip), ideally to your android-sdk\tools directory. Now push all 6 files (adb push [file] /sbin/). Next, we need to make them useable, so go into the shell (adb shell). Change to your /sbin/ directory, and run: chmod 0755 <file> on each of the 6 files.
Now, we need to fix the partitions. This is assuming that the partitions are there, just the wrong format (which is what happened to me .. I accidentally made them FAT32 instead of ext). So, run the following: parted /dev/block/mmcblk0 mkfs ext2. It will ask if you want to continue, hit yes. When it asks for the partition number, enter 1. Next, when it asks for the format, enter ext2. Let it do its thing. Now, once it's done, run parted again. This time, enter partition 2 (everything else is the same).
Click to expand...
Click to collapse
as usual, if anything goes wrong, no responsibility etc
The official update tutorial for rooted users doesn't keep app data, only internal storage! This tutorial keeps EVERYTHING
This method allows updating from 4.5.15 (unlocked, encrypted, rooted) to 5.0.2 without any data (sys settings + app data + user data) loss
A FAQ section is present at the end of this post and will be regularly updated.
List of files to download while doing the following steps:
OOS 5.0.2 ROM
Codeworkx TWRP recovery
Latest Magisk
Terms and software used in this guide:
Fastboot / Bootloader = bootloader of the phone, it's a very low level mode of the phone that allows booting into recovery. Can be accessed by using the advanced reboot menu (enable in dev options) or by "adb reboot fastboot".
Recovery = a small operating system on the phone that allows you to do various operations even when the main OS (Android) is broken. This includes flashing ROMs, modifying stuff on the storage, etc. It's the Android swiss army knife. If you can get a phone to boot TWRP, then you can do almost anything.
Magisk = rooting software that uses a systemless method to keep SafetyNet working. Systemless = instead of modifying the system, every change is put in a separate image that is mounted "over" the system. When the system tries to access a file modified by Magisk, instead of reading it from the partition, it reads it from Magisk. It's recommended to use Magisk instead of SuperSU as of 2018.
ADB = tool that allows controlling the phone from your PC through USB. You can use it when you're in Android if USB debugging is enabled in the settings, or when you're in TWRP. Here, we mostly use it for transferring files directly (without MTP) and running commands (using "adb shell")
Note: For this guide you will be required to download and install Magisk. If you don't want your phone to be rooted, then at the end of this guide reboot into TWRP, wipe both caches and re-flash the OS. This will uninstall Magisk and any other root patch. Beware: it will reflash stock recovery, so if you ever want to re-root, you'll need to reboot to fastboot and flash TWRP manually.
Convention for commands that you will have to run:
a command line starting with "C:" means that it should be run on your PC
a command line starting with "~ #" means that it should be run on your phone (through adb shell) while in TWRP
a command line starting with "OnePlus5:/ $" means that it should be run on your phone (through adb shell) while in OxygenOS
Although the commands start with "C:", this is just for readability purposes. You should run everything from inside an empty directory with enough disk space and writing access.
Your phone will have to be plugged in to your PC from the beginning to the end. Also, make sure it has at least 80% battery before beginning, just in case.
I know, the tutorial is huge. This is simply due to the fact that if I just wrote "make a nandroid backup of this and that, flash, and restore the backup while doing this", then some people may encounter problems because not everyone knows how to do a nandroid backup, restore it, etc. Also, there are a lot of things that need to be done precisely that way and not another way, which explains why the tutorial is huge. Also, you may notice that there is a lot of commands to run throughout the tutorial, this is because that way, I'm sure that at the end, you will have done everything like I did it on my phone, so that if you have a problem it's much easier to figure out where it comes from.
Summary of what you need to do (this is only a SUMMARY to give you a preview of what the whole thing looks like, you shouldn't follow it except if you're really an expert since a lot of things need to be done precisely, instead you should follow the easier complete steps below):
Make a Nandroid backup of /data
Backup files on internal storage
Wipe everything (internal storage + /data + system + caches), and then format data (important!)
Push and flash the OOS zip
Wipe caches and reboot (to Oreo!)
When it reboots, make sure everything (features, like Wi-Fi and fingerprint sensor) works. Don't "save anything" though, everything you do will be erased when we'll restore your backup. This is just a "test drive" for Oreo.
Reboot to TWRP, wipe Data and restore the /data backup
Run the three commands to fix Wi-Fi and fingerprints
Reboot (to System) and check everything works (don't do anything, don't change any setting, just make sure it works)
Reboot to TWRP, rename the "Android" folder to "Android_oreo" on sdcard, delete everything else on sdcard and restore your internal files
Rename the freshly restored Android (nougat) folder to "Android_nougat" and rename "Android_oreo" to "Android".
Flash Magisk, wipe dalvik+cache and reboot to System
When in Android, everything should work except some apps won't have their data. This is normal. Open a terminal (either on your phone using Termux or from your PC using adb shell), elevate using su and rename "Android" to "Android_oreo" and "Android_nougat" to "Android" (this is so that it correctly restores permissions)
If everything works fine, delete the "Android_oreo" folder
First, if you have Xposed Framework (systemless or not) installed, uninstall it. Next, if needed, uninstall any Magisk module that is "Nougat-only" to prevent any problems afterwards.
Boot the phone to bootloader/fastboot (either using advanced reboot, or by using volume down button when you start your phone) and boot to the TWRP recovery by doing
Code:
C:\> fastboot boot twrp-3.2.1-0-oreo-8.1-codeworkx-cheeseburger.img
from your PC.
Next, in TWRP, make a backup of /data (using the Backup button). Then, still while in TWRP, run the following commands:
Code:
C:\> adb shell
~ # cd /sdcard
/sdcard # tar cvf twrp.tar TWRP
/sdcard # md5sum twrp.tar
<< md5 checksum of twrp.tar >>
/sdcard # exit
C:\> adb pull -p /sdcard/twrp.tar
When the above command has finished, make sure that the checksum of the received twrp.tar file matches the one previously displayed.
If it doesn't match, delete the file and run adb pull again. Don't continue following this guide until you have received a 1:1 (checksum-wise) backup of /data.
Code:
C:\> adb shell
<< WARNING: dangerous command! double check the following line is correct before pressing enter! >>
~ # rm -rf /sdcard/TWRP
~ # rm /sdcard/twrp.tar
That was for /data. Now, the backup for the internal storage:
Code:
~ # cd /sdcard
/sdcard # du -csh
<< you should see here the total size of sdcard, that'll give you an idea of how long it'll take >>
/sdcard # tar cvf sd.tar element1 element2 element... elementN
<< in the command above, replace "element1..N" by a space-separated list of what you want to have in the backup.
Keep in mind that your list HAS to contain the element "Android" (case is important). It contains the app data.
Let's say for example you want to keep only the photos you have taken (and nothing, nothing else that was on internal storage).
The photos are in the folder DCIM, so the command will look like this:
tar cvf sd.tar Android DCIM
(because you want DCIM, and Android has to be in the list, no matter where)
>>
/sdcard # md5sum sd.tar
<< md5 checksum of sd.tar >>
/sdcard # exit
C:\> adb pull -p /sdcard/sd.tar
When the above command has finished, make sure that the checksum of the received sd.tar file matches the one previously displayed.
If it doesn't match, delete the file and run adb pull again.
Keep in mind that anything you don't put in that list will not be backed up and will be lost!
Now, you have a backup for all the important stuff so we can start doing the real sh*t.
Now, the important step:
Go back to the TWRP home screen, press "Wipe", "Advanced Wipe" and there check "Dalvik / ART Cache", "Cache", "System", "Data" and "Internal Storage". Confirm using the slider at the bottom of the screen. Press the home button, then "Reboot" and "FastBoot". Now, type the same fastboot command as in the previous step to boot the recovery image. You'll enter the recovery as before.
Now, on your PC, in the terminal, type
Code:
adb push -p OnePlus5Oxygen_23_OTA_029_all_1801292040_d71af3d.zip /sideload
(note: here, we are not using "adb sideload", we are really using "adb push"). In TWRP, click Install, in the file manager go to /sideload and select the OOS zip file. Confirm by sliding. If you get an error, go back to home, click Mount and ensure System is not checked. Then try installing again. If it still does not work, reboot to fastboot, type command again, get to the recovery and install again.
When the zip-file is installed, go home, click "Wipe", "Advanced Wipe" and check both caches and confirm. Then, go home, click "Wipe" and then "Format Data". Then, go home, click "Reboot" and then "System". Your phone will now reboot to Oreo. It will take a long time, but do not turn off the phone. Let it run. On my phone, it took on average 2 minutes for that boot.
You'll be greeted by the "first boot" page. It'll ask you if you want to restore a backup or start anew, choose start anew. Connect to your Wi-Fi network and Google account. Follow all the instructions until you get to the home screen. There, make sure everything works (especially Wi-Fi and fingerprint sensor). Don't save your fingerprints yet, they will be erased afterwards. If everything works, you can continue following these instructions. If not, post a comment down there.
Now that you're at the home screen, go in the settings, About Android and click the build number 8 times to enable Developer Options. Go in there and enable advanced reboot. Then, reboot your phone into fastboot/bootloader using the power button. Type the exact same command as before to start TWRP. Once that you are in TWRP, run the following commands:
Code:
C:\> adb push -p twrp.tar /sdcard/
C:\> adb shell
~ # cd /sdcard
/sdcard # tar xvf twrp.tar
/sdcard # cp /data/misc/wifi/WifiConfigStore.xml /sdcard/
In TWRP, click "Wipe", "Advanced Wipe" and check only the "Data" partition. Confirm. Press home, then "Restore" and choose the backup in the list. Confirm to restore. Back to the terminal, we need to run the following commands otherwise Wi-Fi and fingerprints won't work:
Code:
/sdcard # cp WifiConfigStore.xml /data/misc/wifi/
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
<< WARNING: dangerous command! double check the following line is correct before pressing enter! >>
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
Note: the command above are ran from your PC in an adb shell while the phone is still in TWRP.
Reboot the phone to system and ensure Wi-Fi and fingerprints are still working. Right now you should already see your old home screen and all your apps, but the internal storage isn't there yet. Reboot in fastboot, run the command to get in TWRP.
Once TWRP has booted, run the following commands:
Code:
C:\> adb push -p sd.tar /sdcard/
C:\> adb shell
~ # cd /sdcard
<< WARNING: dangerous command! double check the following line is correct before pressing enter! >>
/sdcard # rm -rf Alarms Albums DCIM Download Movies Music Notifications Pictures Podcasts Ringtones
/sdcard # ls
<< now, look at the list of files that were printed, and rm anything left that is not called "Android" or "sd.tar"
<< WARNING: dangerous command! double check everything is correct before pressing enter! >>
if when you do rm <the thing> it tells you it's a directory, then do: rm -rf <thething>
if there's a folder called SomeFolder, do "rm -rf SomeFolder"
next, run this:
/sdcard # ls
Android sd.tar <-- expected output
/sdcard # mv Android Android_oreo
/sdcard # tar xvf sd.tar
/sdcard # mv Android Android_nougat
/sdcard # mv Android_oreo Android
If you don't have Magisk somewhere on your sd card, download it and upload it using MTP or adb. Then flash it using the Install button. Clear dalvik/cache and reboot to system.
When the phone has booted (again, it might take time), make sure USB debugging is enabled and run the following commands:
Code:
C:\> adb shell
OnePlus5:/ $ su
<< here, you might see a Magisk screen asking for superuser access. Allow. >>
OnePlus5:/ $ cd /sdcard
OnePlus5:/sdcard $ mv Android Android_oreo && mv Android_nougat Android
Now, try some apps and make sure all the data is there (especially games and Netflix/Hulu/etc). If everything is there, and the phone works properly, go back in the terminal and type:
Code:
OnePlus5:/sdcard $ rm -rf Android_oreo
Optionally, start the TWRP app and flash it, it can always be useful. You can also reboot to fastboot to do that.
Now reboot your phone (normal reboot) one last time.
There, working OOS 5.0.2 / Android 8.0.0 phone with no data loss.
Frequently Asked Questions
How long does the whole thing take?
Highly depends on the amount of data you have on your phone. Since it the USB port only supports USB 2.0, it may take 4 or 5 hours in total.
Will doing this void my warranty?
No.
Will I be able to install future OTA updates using the regular download-reboot-flash-twrp procedure?
Yep. Just use the regular method as you would have on Nougat.
Will I lose my data?
If you follow all the instructions, no. Even if you don't follow them, as soon as you have made a backup of /data and internal storage, then no matter how bad you screw up you could always get a working phone back.
I followed the instructions and now my phone doesn't work
Boot in TWRP, wipe everything, reflash.
questions will be added there in the future
Having WiFi and fingerprint issues
You sure it is a good idea to just delete those files? I would have guessed that I need to replace these (nougat version from backup) with the oreo version to have it working just like before the restore.
Code:
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
I'm having the issues with wifi and fingerprints. Neither one is working. I'll try to figure out how to fix this.
@zdimension Thanks for this guide, I don't have time to test it yet, but I have a question
pdluke said:
Code:
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
Click to expand...
Click to collapse
At this point in the procedure, adb shell is still using root (before flashing magisk) ? How is that possible ? Does the adb /sideload preserve root ?
olivier380 said:
@zdimension Thanks for this guide, I don't have time to test it yet, but I have a question
At this point in the procedure, adb shell is still using root (before flashing magisk) ? How is that possible ? Does the adb /sideload preserve root ?
Click to expand...
Click to collapse
These commands should be run while the phone is in TWRP. Also note that adb /sideload is not used here, only adb push.
pdluke said:
You sure it is a good idea to just delete those files? I would have guessed that I need to replace these (nougat version from backup) with the oreo version to have it working just like before the restore.
Code:
/sdcard # rm /data/misc/wifi/wpa_supplicant.conf
/sdcard # rm -rf /data/system/users/0/fpdata
/sdcard # rm /data/system/users/0/settings_fingerprint.xml
I'm having the issues with wifi and fingerprints. Neither one is working. I'll try to figure out how to fix this.
Click to expand...
Click to collapse
If you delete them, they will be generated automatically at the next system boot. But you could also make a backup of those three files before wiping /data, store that somewhere, restore Nougat /data and then restore your backup of those three files. The result would be the same.
Note: actually, not exactly. Erasing the first file won't change anything since it's not used anymore in Oreo, but the two other files contain the fingerprint configuration (list of saved fingerprints). So,
Either you remove the files and you have to save your fingerprints again at next boot
Either you restore them from an Oreo backup and you'll get the fingerprints you had saved during the "first boot" procedure when you rebooted the phone right after flashing the OS
But the result is mostly the same: everything works. Deleting the files ensures you get something clean. If you restore from an Oreo backup I can't guarantee the result (as it may interfere with other files from the Nougat backup).
10 bucks to make a script to do this all for me haha.
@zdimension Thanks for the clarification Another thing you might add to the files to download would be Magisk (optionally). In this kind of guide, I've always find it useful to download everything first.
olivier380 said:
@zdimension Thanks for the clarification Another thing you might add to the files to download would be Magisk (optionally). In this kind of guide, I've always find it useful to download everything first.
Click to expand...
Click to collapse
Oops, forgot to add it
I added the link, and also instructions for how to un-root afterwards for those who would want it.
To improve the guide, here are some ideas :
- You should highlight that rm -rf is a very dangerous command, and that it needs to be checked twice (especially the targeted folder)
- It could be useful to use the du -csh command to check the size of a folder (to estimate the backup time for example).
- As a safety measure, one could md5sum the tar file before and after using adb pull
What do you think ?
olivier380 said:
To improve the guide, here are some ideas :
- You should highlight that rm -rf is a very dangerous command, and that it needs to be checked twice (especially the targeted folder)
- It could be useful to use the du -csh command to check the size of a folder (to estimate the backup time for example).
- As a safety measure, one could md5sum the tar file before and after using adb pull
What do you think ?
Click to expand...
Click to collapse
Thanks for the ideas! I updated the post (and I added a changelog at the bottom for future reference).
Followed guide for successful upgrade from 4.5.15 encrypted, unlocked bootloader w/ Magisk root.
One note, after the first complete wipe and flash of the full ROM, it was getting stuck on first boot and never completed. Discovered that I needed to not just wipe the Data partition but Format it in TWRP, to clear out the old encryption I think. Magisk wouldn't install either until I did this.
@debork thanks for the positive feedback (all the merit goes to @zdimension of course)
@zdimension there are many people in the other thread https://forum.xda-developers.com/oneplus-5/how-to/official-oxygenos-4-5-2-7-1-1-ota-t3627003 that tried (unsucessfully) to upgrade from 4.5.15 to 5.0.1, maybe a link to this topic could be useful for them (if it's not too late).
Regarding the
Go back to the TWRP home screen, press "Wipe", "Advanced Wipe" and there check "Dalvik / ART Cache", "Cache", "System", "Data" and "Internal Storage".
Click to expand...
Click to collapse
I think it should be highlighted in red, since it is the actual "clean flash" (AFAIU, correct me if I'm wrong).
if we only have the BL unlocked non root and stock recovery can we only ota without any loss of data ?
debork said:
Followed guide for successful upgrade from 4.5.15 encrypted, unlocked bootloader w/ Magisk root.
One note, after the first complete wipe and flash of the full ROM, it was getting stuck on first boot and never completed. Discovered that I needed to not just wipe the Data partition but Format it in TWRP, to clear out the old encryption I think. Magisk wouldn't install either until I did this.
Click to expand...
Click to collapse
Thanks for feedback, I will add that to the guide (although it worked with just Wipe for me )
zdimension said:
Thanks for feedback, I will add that to the guide (although it worked with just Wipe for me )
Click to expand...
Click to collapse
Have you rooted the 4.5.15 with Magisk or SuperSU (which is not compatible with Oreo anymore) ?
olivier380 said:
Have you rooted the 4.5.15 with Magisk or SuperSU (which is not compatible with Oreo anymore) ?
Click to expand...
Click to collapse
I stopped using SuperSU when it was sold to that shady company. Also, Magisk is better imo.
quick05 said:
if we only have the BL unlocked non root and stock recovery can we only ota without any loss of data ?
Click to expand...
Click to collapse
Official OnePlus support said that nothing is guaranteed if your bootloader is unlocked. But since you're on stock recovery + non rooted, you could always try. But backup everything first. Some people here on XDA have reported that it doesn't work, though.
Just followed your guide with no problems. Thank you very much!! I can confirm also that you need to format data after the wipe otherwise it gets stuck in a bootloop!
Thanks so much for this. I was able to successfully follow the guide and get upgraded to 5.0.1 without losing any data. In fact, I even messed up one step by failing to include the Android directory in the sdcard.tar backup (perhaps that should be more explicit), but it doesn't seem to have affected everything; all of my apps seem to have retained their data.
A few notes:
1. The file size of twrp.tar was ~14GB but when executing the pull command, it recognized it as only ~1.3 GB. As a result, the pull was not complete until it reached over 1000%. All the more reason to do the md5 check.
2. As others stated, I needed to format the data partition, not just wipe it.
3. I might recommend also including a "summary" version somewhere on what this guide does. Scrolling through the guide the first time, it seemed pretty daunting, but really all that you're doing is: backing up data partition and internal storage; wiping device; flashing Oreo ROM; tweaking a few files; and restoring backed up data and internal storage.
Thank you again so much! Glad to finally be on Oreo.
elight3 said:
Thanks so much for this. I was able to successfully follow the guide and get upgraded to 5.0.1 without losing any data. In fact, I even messed up one step by failing to include the Android directory in the sdcard.tar backup (perhaps that should be more explicit), but it doesn't seem to have affected everything; all of my apps seem to have retained their data.
A few notes:
1. The file size of twrp.tar was ~14GB but when executing the pull command, it recognized it as only ~1.3 GB. As a result, the pull was not complete until it reached over 1000%. All the more reason to do the md5 check.
2. As others stated, I needed to format the data partition, not just wipe it.
3. I might recommend also including a "summary" version somewhere on what this guide does. Scrolling through the guide the first time, it seemed pretty daunting, but really all that you're doing is: backing up data partition and internal storage; wiping device; flashing Oreo ROM; tweaking a few files; and restoring backed up data and internal storage.
Thank you again so much! Glad to finally be on Oreo.
Click to expand...
Click to collapse
Thanks for the feedback! I'll add a summary to the guide.