Remove NOOBS and migrate Raspbian to new microSD card

[e-raser]

Hi @ all,

my target is the following:
1) Remove all other partitions/OS including NOOBS, so at the end only boot and data partition from Raspbian remains (and data partition is expanded for using all available card storage)
2) This is urgently needed because of a) switch from SD to microSD (Pi Mod. B --> Pi 2 Mod. B) and b) the data partition of Raspbian has only ~ 5 MB left which makes it nearly impossible to run the Pi (think about package updates etc.).

I´m really suffering on this topic, I already spent more than 12 hours without success! As a past Windows geek I´d simply use Acronis True Image Home which does the job perfectly - but not for Linux based OS.
Therefore I hope to get a little help of all the experts here.

current partition layout on SD card (df -h):

Code: Select all

Dateisystem           Größe Benutzt Verf. Verw% Eingehängt auf
rootfs                 4,9G    4,6G  4,3M  100% /
/dev/root              4,9G    4,6G  4,3M  100% /
devtmpfs               230M       0  230M    0% /dev
tmpfs                   47M    304K   47M    1% /run
tmpfs                  5,0M       0  5,0M    0% /run/lock
tmpfs                   94M       0   94M    0% /run/shm
/dev/mmcblk0p9          60M     15M   45M   25% /boot
tmpfs                   94M    4,0K   94M    1% /tmp
/dev/mmcblk0p6         3,6G    7,4M  3,4G    1% /media/Storage
/dev/mmcblk0p5         127M    101M   26M   80% /media/SYSTEM
/dev/mmcblk0p3          27M    423K   25M    2% /media/SETTINGS
/dev/mmcblk0p7          79M    8,2M   71M   11% /media/BOOT
/dev/mmcblk0p8         3,9G    780M  3,0G   21% /media/root
/dev/mmcblk0p11        488M    408K  452M    1% /media/data

In my opinion/my destined partition layout on new microSD card:

Code: Select all

=> /dev/mmcblk0p5  delete
=> /dev/mmcblk0p6  delete
=> /dev/mmcblk0p7  delete
=> /dev/mmcblk0p8  delete
=> /dev/mmcblk0p11 delete
=> /dev/mmcblk0p9  KEEP (Raspbian!)
=> /dev/mmcblk0p10 KEEP (Raspbian!)

I have the following hardware-/software-tools:
- a second Pi hardware (Pi 2 mod. b)
- a new microSD card
- a microSD/SD-to-USB card reader which can be used under Linux
- a GParted in a Linux-VM (usable with USB-card reader)

I used next to this topic these following information (effectively useless so far):
http://raspberrypi.stackexchange.com/qu ... ep-raspian

I already tried a few things (different strategy approaches) and spent almost 12 hours on it with my little basic linux knowledge, not successfully so far:
1) DD-backup from within running Raspbian, restore backup on new SD --> filesystem errors when attempting to boot, not repairable
2) Installed Raspbian from scratch to new blank microSD, which gave the partition layout. Restored TAR backup to new sd card --> also gave errors, don´t remind it correctly (was about 2 months ago, no docu)

I now see the following different strategies:
1) Create a duplicate backup of the SD to the new microSD, remove NOOBS and other unwanted OS (on microSD, not backup source), make some small partition adoptions, expand filesystem, ready
2) Create a backup of only needed Raspbian partitions (boot + data), restore it to new microSD, <do some magic to bring Raspbian to boot>, expand filesystem, ready

Question 1: Which one should I prefer? I tend to #2 because backing up all the anyway unwanted data at #1 only takes unnecessary time.
Question 2: What would be the exact, detailed steps for strategy #2?

I see the following actions, please correct me (order, needed commands, ...):
1) power down Pi mod. B
2) remove SD
3) put SD in sd-to-usb card reader, start Linux on PC with mounted card reader
4) Linux: copy relevant partitions (DD? TAR?) to local partition
5) Linux: switch SD to microSD in card reader and restore backup from local partition to new, empty microSD
6) Linux: make some filesystem adaptions to successfully boot Raspbian: e.g. modify /etc/fstab of root partition
7) Linux: Run GParted and check filesystem. expand data filesystem (or better use raspi-config later?)
8) put microSD back to new Pi 2 mod. B
9) power up Pi mod. B and pray for successful boot

What do you think?!?

[rpdom]

I'd be tempted to do the whole thing on the Pi, with the new card in the card reader.

1. Repartition new card to have two partitions. The first one exactly the same type and size as /boot, and the second one using the rest of the card as type Linux.
2. dd if=/dev/mmcblk0p(whatever /boot is) of=/dev/sda1 (the first partition on the new card)
3. mkfs -t ext4 /dev/sda2 (the new Linux partition)
4. mount -t ext4 /dev/sda2 /mnt
5. rsync -ax --progress / /mnt
6. nano /mnt/etc/fstab and change /boot and / to mmcblk0p1 and mmcblk0p2 respectively.
7. umount /mnt
8. mount -t vfat /dev/sda1 /mnt
9. nano /mnt/cmdline.txt and change the root= to /dev/mmcblk0p2
10. umount /mnt
11. Remove card and use it.

That should work, unless I've missed anything.
[edit] Mistakes now corrected

[e-raser]

Thank you @rpdom, the idea of doing the whole action directly on the Pi didn´t come to my mind - because I thought it would somehow be better to do it "offline" without the running OS I´m copying. But I´ll give it a try! As long as I don´t use one of the commands in the wrong direction, the SD card (source) shouldn´t be affected I think.

A few questions in advance:
a) Should I stop some processes on the pi (e.g. the owncloud services)? Or isn´t it necessary (I try do perform no changes to the oc database or file structure while copying it).
b) What is sda1 and sda2 in your listed commands?
to 1) Do you have the corresponding commands for me? Altough I created this topic in Advanced section, I´m still a beginner...
to 5): Is rsync the "perfect" tool for this job? Does it copy really everything including hidden files, file/directory rights etc.?

[kusti8]

Rsync would really be the best tool for this. Sda1 is the first partition on the device named sda. You can check if this is your device by removing it and check if /dev/sda exists.

You have to have an os running om another SD card ie the old one and then use a USB card reader to copy it to the new SD card.

You should really have as little running as possible so that nothing messes up the files.

All the commands are relative to how your pi turns out, what os is on your pi etc. There really are no specific commands. Follow rpdom instructions and then replace the variables ie /dev/sd* and your partition numbers.

[e-raser]

Alright the microSD is connected via USB card reader to the Pi, it is /dev/sda.

I now need some help on
"1. Repartition new card to have two partitions. The first one exactly the same type and size as /boot, and the second one using the rest of the card as type Linux."

fdisk is clear, but what commands for creating both partitions?

sudo fdisk -l gives:

Code: Select all

Disk /dev/mmcblk0: 32.5 GB, 32528924672 bytes
4 heads, 16 sectors/track, 992704 cylinders, total 63533056 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d0f53

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1            8192     2833984     1412896+   e  W95 FAT16 (LBA)
/dev/mmcblk0p2         2834432    31323135    14244352   85  Linux extended
/dev/mmcblk0p3        31323136    31388671       32768   83  Linux
/dev/mmcblk0p5         2842624     3104767      131072    c  W95 FAT32 (LBA)
/dev/mmcblk0p6         3112960    10784767     3835904   83  Linux
/dev/mmcblk0p7        10788864    10952703       81920    c  W95 FAT32 (LBA)
/dev/mmcblk0p8        10960896    19525631     4282368   83  Linux
/dev/mmcblk0p9        19529728    19652607       61440    c  W95 FAT32 (LBA)
/dev/mmcblk0p10       19660800    30253055     5296128   83  Linux
/dev/mmcblk0p11       30261248    31309823      524288   83  Linux

I have no idea what the commands are. Which partition type (primary, logical, ...) and how to give the same size (for /boot).

Update: sudo parted /dev/mmcblk0 gives more information. But Which partition on the new microSD should be set as primary? The boot partition I guess?

Code: Select all

Model: SD SD (sd/mmc)
Disk /dev/mmcblk0: 32,5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system  Flags
 1      4194kB  1451MB  1447MB  primary   fat32        lba
 2      1451MB  16,0GB  14,6GB  extended
 5      1455MB  1590MB  134MB   logical   fat32        lba
 6      1594MB  5522MB  3928MB  logical   ext4
 7      5524MB  5608MB  83,9MB  logical   fat32        lba
 8      5612MB  9997MB  4385MB  logical   ext4
 9      9999MB  10,1GB  62,9MB  logical   fat32        lba
10      10,1GB  15,5GB  5423MB  logical   ext4
11      15,5GB  16,0GB  537MB   logical   ext4
 3      16,0GB  16,1GB  33,6MB  primary   ext4

Update: I managed to get the boot partition, but with wrong size. Source is 62,9 MB, if I try to use this (62.9M) as new size it says something like "not supported size format"!

[rpdom]

Ok, "sudo fdisk /dev/sda" to get to the new card. If there are any partitions already there you should delete them by pressing "d".

Then create the partitions like this

Code: Select all

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-xxxxxxx, default 2048): 8192
Last sector, +sectors or +size{K,M,G} (8192-xxxxxxx, default xxxxxxx): 131071

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): c
Changed system type of partition 1 to c (W95 FAT32 (LBA))

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): p
Partition number (1-4, default 2): 2
First sector (2048-xxxxxxx, default 2048): 131072
Last sector, +sectors or +size{K,M,G} (90112-xxxxxxx, default xxxxxxx): 
Using default value xxxxxxx

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 83

Command (m for help): w
The partition table has been altered!


WARNING: If you have created or modified any DOS 6.x
partitions, please see the fdisk manual page for additional
information.
Syncing disks.

The xxxxxxx value will depend on the size of your card. Just accept the default at that point.

[e-raser]

Alright! Step 1 finished. Going over now to step 2 (DD).

[e-raser]

Step 2 finished.

[e-raser]

Step 3 done.

[e-raser]

Step 4: There seems to be an error at

Code: Select all

4. mount -t ext4 /dev/sda1 /mnt

What is the target here? sda1 is the w95 (type c) partition, sda2 is the ext4 (linux data).
I guess according to step 5 (copy all stuff using rsync) I should use sda2.

[e-raser]

Update: Step 4 finished. Step 5 currently running; will take some time (5 GB at ~ 2 MB per second = ~ 43 minutes).

But don´t I need to exclude the /mnt from the rsync of step 5? Could result in an "inception"? Or how will it behave when rsync-ing the mounted file (in this case /mnt/temp I created for sda2) itself?

[rpdom]

Yes, you've spotted the deliberate mistake Step 4 should be sda2.

The "x" option on rsync means to exclude any other filesystems.

[e-raser]

Ha, to be honest I´m a lil bit lucky - the amateur spotted a pro´s error Anyway, rsync is in work (see my last post above).
I found a second one: The numeration (two times 5.). So the list has 11 instead of 10 todo items.

Time for focussing on the remaining steps 6 to 11:

Code: Select all

6. nano /mnt/etc/fstab and change /boot and / to mmcblk0p1 and mmcblk0p2 respectively.
7. umount /mnt
8. mount -t vfat /dev/sda1 /mnt
9. nano /mnt/cmdline.txt and change the root= to /dev/mmcblk0p1
10. umount /mnt
11. Remove card and use it.

What I´m doing here is basically modifying the "boot list entry points", right?

[kusti8]

Ha, to be honest I´m a lil bit lucky - the amateur spotted a pro´s error Anyway, rsync is in work (see my last post above).
I found a second one: The numeration (two times 5.). So the list has 11 instead of 10 todo items.

Time for focussing on the remaining steps 6 to 11:

Code: Select all

6. nano /mnt/etc/fstab and change /boot and / to mmcblk0p1 and mmcblk0p2 respectively.
7. umount /mnt
8. mount -t vfat /dev/sda1 /mnt
9. nano /mnt/cmdline.txt and change the root= to /dev/mmcblk0p1
10. umount /mnt
11. Remove card and use it.

What I´m doing here is basically modifying the "boot list entry points", right?

We all make mistakes.

Basically step 6 you are changing the partition numbers, since NOOBS has different partitions to the boot and linux partition.

[rpdom]

Ha, to be honest I´m a lil bit lucky - the amateur spotted a pro´s error Anyway, rsync is in work (see my last post above).
I found a second one: The numeration (two times 5.). So the list has 11 instead of 10 todo items.

That's the "Easter Egg" - two 5's for the price of one

Time for focussing on the remaining steps 6 to 11:

Code: Select all

6. nano /mnt/etc/fstab and change /boot and / to mmcblk0p1 and mmcblk0p2 respectively.
7. umount /mnt
8. mount -t vfat /dev/sda1 /mnt
9. nano /mnt/cmdline.txt and change the root= to /dev/mmcblk0p1
10. umount /mnt
11. Remove card and use it.

What I´m doing here is basically modifying the "boot list entry points", right?

As kusti said, step 6 is changing the mount points to use the new partition numbers.
Step 9 is telling the kernel which partition to use as /

[e-raser]

Ok next (hopefully last) problem:

Code: Select all

9. nano /mnt/cmdline.txt and change the root= to /dev/mmcblk0p1

Currently/old content of cmdline.txt is: root=/dev/mmcblk0p10 --> which was the DATA partition (linux)! According the above information changing it to root=/dev/mmcblk0p1 it would then point to the BOOT partition.

What is correct here? Point to the BOOT or the DATA partition? I guess it´s the DATA partition, so mmcblk0p2 would be the correct value here. Makes 3rd error by the way

[rpdom]

Ok next (hopefully last) problem:

Code: Select all

9. nano /mnt/cmdline.txt and change the root= to /dev/mmcblk0p1

Currently/old content of cmdline.txt is: root=/dev/mmcblk0p10 --> which was the DATA partition (linux)! According the above information changing it to root=/dev/mmcblk0p1 it would then point to the BOOT partition.

What is correct here? Point to the BOOT or the DATA partition? I guess it´s the DATA partition, so mmcblk0p2 would be the correct value here. Makes 3rd error by the way

I'm not doing very well tonight, am I

Yes, mmcblk0p2 is correct for that entry.

I really hope this works now...

[e-raser]

PERFECT! Raspbian is already running on the new microSD card in the new Pi 2 with loooooooooots of free disk space

Really a BIIIIIIIIIG THANK YOU all here! Wish I had created this topic a few weeks earlier, but at least I learned a bit on partitioning etc.

[rpdom]

Thanks, I'm glad that worked. I'll go back and fix the mistakes you found so that people who find this post in future will have the right instructions

[electronicsguy]

PERFECT! Raspbian is already running on the new microSD card in the new Pi 2 with loooooooooots of free disk space

Really a BIIIIIIIIIG THANK YOU all here! Wish I had created this topic a few weeks earlier, but at least I learned a bit on partitioning etc.

you could also consider buying a usb flash drive and transferring the root partition on that. the RPi will then use that for all its need, except initial boot. the USB drives easily come in sizes of 64GB, 128GB. and are much more reliable than micro-sd cards.

[Amarand]

you could also consider buying a usb flash drive and transferring the root partition on that. the RPi will then use that for all its need, except initial boot. the USB drives easily come in sizes of 64GB, 128GB. and are much more reliable than micro-sd cards.

I found my way to this (great!) post, and I'm sure at some point I will follow these directions to free up the space and partitions that NOOBS uses - but I also just migrated from the 8GB branded card to a SanDisk Ultra 64GB UHS-I/Class 10 Micro SDXC Memory Card. I just realized that allocating 1GB on a 8GB [12.5%] (or, yikes, a 4GB [25%]) memory card is a lot of overhead, it's really not that much for a 64GB card [1.56%].

What size card are you using? 8GB I'm assuming? Have you considered purchasing a huge (64GB) card? The migration over to the new card was super simple - just two dd commands, and a gparted to wiggle things around.

[Amarand]

...the USB drives easily come in sizes of 64GB, 128GB. and are much more reliable than micro-sd cards.

Are microSD cards really less reliable than USB drives?

[electronicsguy]

...the USB drives easily come in sizes of 64GB, 128GB. and are much more reliable than micro-sd cards.

Are microSD cards really less reliable than USB drives?

Yes, very much so. They're prone to being corrupted - at the data and physical level by ESD, bad connections, etc (I had one which gave out within 2 days). Try plugging off/on the power the to Pi with SDcard in it a few times and you'll damage it. But the flash drive will still be ok.

Plus flash drives are much faster. I have raspbian, arch linux and google coder on my RPi. My SD card only has NOOBS and 3 boot partitions. All the root filesystems reside on my USB flash drive. Plus, I capture data from my Pi Camera, and store the images/videos on my flash drive itself. So its so much easier to take the flash drive out, put it in any usb slot on my laptop and run mencoder for doing a time-lapse from the photos.

In general, I feel, its much better to use a flash drive as the main file system in the RPi. But whatever works best for you!

[BartmanEH]

I found this information helpful in my quest to rid myself of NOOBs and run a pure Raspbian environment.

I don't believe all the malarkey about USB drives being more reliable than SD Cards - they use basically exactly the same technology.

I suggest that the rsync command "rsync -ax --progress / /mnt": would be better as "rsync -axH --progress / /mnt" to preserve hard links. I've run rsync as suggested but was worried I don't have an exact copy of my original file system since hard links have been expanded to copies of the files instead of links so I added a --delete-after directive and reran it with the H directive added.