HOWTO: Move the filesystem to a USB stick/Drive


263 posts   Page 1 of 11   1, 2, 3, 4, 5 ... 11
by paulv » Sat May 18, 2013 1:26 pm
I use a Raspberry Pi in a more or less Mission Critical Application on another continent, and was getting concerned with the reports about SD cards getting corrupted. Apparently, these cards are not really made for heavy read/write applications, although they are nice and cheap in educational and hobby environments.

If you want to offload the main filesystem to another medium, there are several examples if you Google around enough. Unfortunately, some are not really applicable to the Raspberry Pi, and others are for super-super users, who know what they are doing.

I created this description to make it also usable for power-users, although I am by no means a super-power-user myself. Beware!

To make it simple and easier, I created two parts to the procedure.
1. A simple version if you are using Pi with only 1 extra USB drive or stick
2. An extended version if you need more than one drive attached to your Pi. This may be applicable to RaspBMC users? Note that this version is a little more complicated.

We will start off with a brand new SD card, and a brand new USB stick.
The size and speed of the SD card is no longer important and any card over 2G will do (it has to be at least 2G because we're going to load Debian onto it in its entirety).

In this tutorial, I will use a micro SDHC memory card with 4GM, class 10, because I use the cards for other applications as well and I don't like the SD card sticking outside of the Pi cover. I also use an 8GB micro USB stick.

First of all, you need to load the Wheezy Raspbian image (I used 2013-02-09) on the SD card. There are enough procedures to tell you how to do that. I use win32diskimager on my W7 laptop. You can then put it in your Pi and boot it up.
If you use a LAN cable, you're good to go. If you run a headless Pi (like I do) over WIFI, you need to configure that first: again, there are many procedures on how to do that for your particular dongle. I use an EDIMAX in this particular setup. Obviously, I'm using ssh, but that does not matter.

First of all, you should run raspi-config and set up ONLY the following:
Code: Select all
sudo raspi-config

configure_keyboard (not needed for a headless Pi)
change_pass (change the raspberry password if you wish at this moment, can also be done later)
change_locale
change_timezone
memory_split (a headless can be set to 16MB)
Do not update/upgrade at this moment! That will cause a lot of thrashing on the SD card, which is what we want to avoid.

After a reboot to load the new parameters, we're good to go.

Check the filesystem:
Code: Select all
pi@raspberrypi ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          1.8G  1.4G  247M  86% /
/dev/root       1.8G  1.4G  247M  86% /
devtmpfs        235M     0  235M   0% /dev
tmpfs            49M  248K   49M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            98M     0   98M   0% /run/shm
/dev/mmcblk0p1   56M   19M   38M  34% /boot
pi@raspberrypi ~ $
Notice that although I use an 8GB SD card, the default is 1.8G in size, because we did not use expand_rootfs.
The entry /dev/mmcblk0p1 is the boot partition (p1) on device mmcblk0, the SD card.

The boot partition will be the only piece on the SD card used when we're done, and that is because of the particular way the Pi goes through the boot process. It actually starts with the graphics chip loading this data first, and it then releases the reset pin of the CPU which continues and takes over.

Now plug in your USB drive/stick if you have not done that yet (I'll refer to stick from now on)

Check to see if it's recognized:
Code: Select all
pi@raspberrypi ~ $ lsusb
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 7392:7811 Edimax Technology Co., Ltd EW-7811Un 802.11n Wireless Adapter [Realtek RTL8188CUS]
Bus 001 Device 006: ID 8644:800b
pi@raspberrypi ~ $
Device 006 is the USB adapter. The ID shows a unique manufacturer identifier.
Also check to see if the OS recognized the device: (you can also use the command dmesg)
Code: Select all
pi@raspberrypi ~ $ tail /var/log/messages
May 18 11:44:31 raspberrypi kernel: [   64.093866] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
May 18 11:44:31 raspberrypi kernel: [   64.093883] usb 1-1.3: Product: Intenso Micro Line
May 18 11:44:31 raspberrypi kernel: [   64.093898] usb 1-1.3: Manufacturer: Intenso
May 18 11:44:31 raspberrypi kernel: [   64.093912] usb 1-1.3: SerialNumber: 12390000000004E2
May 18 11:44:31 raspberrypi kernel: [   64.112228] scsi1 : usb-storage 1-1.3:1.0
May 18 11:44:32 raspberrypi kernel: [   65.113335] scsi 1:0:0:0: Direct-Access     Intenso  Micro Line       1.01 PQ: 0 ANSI: 2
May 18 11:44:32 raspberrypi kernel: [   65.114862] sd 1:0:0:0: [sda] 15669248 512-byte logical blocks: (8.02 GB/7.47 GiB)
May 18 11:44:32 raspberrypi kernel: [   65.115607] sd 1:0:0:0: [sda] Write Protect is off
May 18 11:44:32 raspberrypi kernel: [   65.126062]  sda: sda1
May 18 11:44:32 raspberrypi kernel: [   65.129907] sd 1:0:0:0: [sda] Attached SCSI removable disk
pi@raspberrypi ~ $
Yes, the USB stick can be referenced as /dev/sda

There is no way of knowing what name the OS assigns to additional USB drives. If that's what you want to do, you need to follow the extended version of the procedure down below.

If you are going to use a single USB stick, start here:

Create a filesystem on the USB stick; use "p" to list the partitions
Code: Select all
pi@raspberrypi ~ $ sudo fdisk /dev/sda

Command (m for help): p

Disk /dev/sda: 8022 MB, 8022654976 bytes
94 heads, 14 sectors/track, 11906 cylinders, total 15669248 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              32    15669247     7834608    b  W95 FAT32

Command (m for help):
There is a partition, formatted for Windows. We need to reformat the drive for Linux.
Enter d to delete partion(s).
Note
If you are only going to use the stick for the Debian filesystem, go ahead. If you have a large stick or a USB drive, you may want to consider making one partition for Debian, and other ones for whatever you like. This is beyond the scope of this article, but I'm sure you can figure it out.
We will continue with the simple solution, one partition.

Enter n to create a new primary partition (number 1) and use the full capacity by hitting return a few times until done:

Code: Select all
Command (m for help): d
Selected partition 1

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-15669247, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-15669247, default 15669247):
Using default value 15669247

Command (m for help):
Now write the new information to the stick by entering "w":

Code: Select all
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
pi@raspberrypi ~ $
Done. Now we can format the partition:
Code: Select all
pi@raspberrypi ~ $ sudo mke2fs -t ext4 -L rootfs /dev/sda1
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=rootfs
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
490560 inodes, 1958652 blocks
97932 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2009071616
60 block groups
32768 blocks per group, 32768 fragments per group
8176 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

pi@raspberrypi ~ $
And we need to mount the partition to start using it:
Code: Select all
pi@raspberrypi ~ $ sudo mount /dev/sda1 /mnt
pi@raspberrypi ~ $
Let's see what we've done so far:
Code: Select all
pi@raspberrypi ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          1.8G  1.4G  247M  86% /
/dev/root       1.8G  1.4G  247M  86% /
devtmpfs        235M     0  235M   0% /dev
tmpfs            49M  248K   49M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            98M     0   98M   0% /run/shm
/dev/mmcblk0p1   56M   19M   38M  34% /boot
/dev/sda1       7.4G  145M  6.9G   3% /mnt
pi@raspberrypi ~ $
We made the stick /dev/sda1 available as /mnt for Linux, so now we can start to use it.

We can copy (mirror) the data in /dev/root to the new drive.
We can use dd, but that is a bit copy program and, because it also copies the empty space on the SD card, it is very, very slow.
It is better to use rsync.
Install that first:
Code: Select all
pi@raspberrypi ~ $ sudo apt-get install rsync
...
pi@raspberrypi ~ $
And run it. Although much faster, it will still take some time to create a mirror image of the file system on the USB drive. We will copy everything from "/" to the just mounted drive /mnt (which is /dev/sda1)

Code: Select all
pi@raspberrypi ~ $ sudo rsync -axv / /mnt
...
On my system it took about 15 minutes. Make sure this process does not get interrupted!
If you want, you can open another ssh session and continue with the process, so you don't have to wait for the rsync process to complete.

We're going to tell the Pi where to find the new data when it boots Debian.
The file we need to change is /boot/cmdline.txt
Before we do that, we're going to create a backup copy so we can change things back if there is a problem. Be careful: if you mess up this file, you will almost certainly stop the Pi from booting, and you may have to start all over again by reloading the image.

Code: Select all
sudo cp /boot/cmdline.txt /boot/cmdline.orig

The contents of this file is one very long command string, and we need to replace the reference to the partition on the SD card (root=/dev/mmcblk0p2) to the new USB stick (root=/dev/sda1)
Here is a copy of the original file:
Code: Select all
pi@raspberrypi ~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
pi@raspberrypi ~ $
To be on the safe side, there is one more command we should add. During the power-up/boot sequence, it may take the USB stick a little longer to be ready, than the OS expects. If we add a delay of 5 seconds (rootdelay=5), the USB stick will be ready when it's needed. When my remote system has to reboot, I want it to be safe: a few seconds more is not an issue.

Use your favorite editor to add these two changes like so:
Code: Select all
sudo nano /boot/cmdline.txt

It should look like this:
Code: Select all
pi@raspberrypi ~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait rootdelay=5
pi@raspberrypi ~ $

Depending on the power consumption of your stick or drive, you may need to edit /boot/config.txt and add:
max_usb_current=1

Next we need to enter a few things into the filesystem table, /etc/fstab. (on the stick, not the SD card)
Code: Select all
sudo nano /mnt/etc/fstab
Add the following line :
Code: Select all
/dev/sda1    /   ext4    defaults,noatime  0       1
And put a comment in front of the reference to the SD card:
Code: Select all
#/dev/mmcblk0p2  /               ext4    defaults,noatime       0       1

It should now look like this:
Code: Select all
proc            /proc           proc    defaults          0       0
/dev/sda1       /               ext4    defaults,noatime  0       1
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
#/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that

This is it for the simple version. you can now reboot the system:
Code: Select all
sudo reboot
Note that you will not see much activity on the ACT LED anymore. This LED only shows R/W activity on the SD card.
After the Pi has booted, check the filesystem:
Code: Select all
pi@raspberrypi ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.4G  1.6G  5.5G  22% /
/dev/root       7.4G  1.6G  5.5G  22% /
devtmpfs        235M     0  235M   0% /dev
tmpfs            49M  248K   49M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            98M     0   98M   0% /run/shm
/dev/mmcblk0p1   56M   19M   38M  34% /boot
pi@raspberrypi ~ $
The rootfs is now on the stick, notice the size of 7.5G

Changing the swap file size
There is no need to mess with swap partitions on the Pi, this is implemented differently from other Linux systems. Raspbian uses dphys-swapfile, which is a swap-file based solution instead of the "standard" swap-partition solution. It is also much easier to change the size of the swap.

If you need to change the swap file size, proceed as follows:

Look at the swap data first by using the top command:
Code: Select all
pi@raspberrypi ~ $ top
top - 13:28:20 up 3 min,  1 user,  load average: 0.18, 0.37, 0.18
Tasks:  66 total,   1 running,  65 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.0 us,  0.0 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:    497544 total,    55700 used,   441844 free,     5432 buffers
KiB Swap:   102396 total,        0 used,   102396 free,    27592 cached

102396 is 100MB of swap space

The configuration file for the swap space is: /etc/dphys-swapfile

The content is very simple. By default, Raspbian has 100MB of swap space:
Code: Select all
pi@raspberrypi ~ $ cat /etc/dphys-swapfile
CONF_SWAPSIZE=100
pi@raspberrypi ~ $
If you want to change the size, you need to modify the number and then resize and restart the dphys-swapfile:

Code: Select all
pi@raspberrypi ~ $ sudo nano /etc/dphys-swapfile
pi@raspberrypi ~ $
I changed the size from 100 MB to 512MB, the full amount of RAM.
After the parameter in the file has been changed run:
Code: Select all
pi@raspberrypi ~ $ sudo dphys-swapfile setup
want /var/swap=512MByte, checking existing: deleting wrong size file (104857600), generating  swapfile ... of 512MBytes
pi@raspberrypi ~ $
To load the new size, we need to stop the swapfile process:
Code: Select all
pi@raspberrypi ~ $ sudo /etc/init.d/dphys-swapfile stop
Stopping dphys-swapfile swapfile setup ..., done.
pi@raspberrypi ~ $
And restart it:
Code: Select all
pi@raspberrypi ~ $ sudo /etc/init.d/dphys-swapfile start
Starting dphys-swapfile swapfile setup ...
want /var/swap=512MByte, checking existing: keeping it
done.
pi@raspberrypi ~ $
And now we can check it again with top:
Code: Select all
top - 13:37:22 up 12 min,  1 user,  load average: 0.01, 0.08, 0.12
Tasks:  64 total,   1 running,  63 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    497544 total,    57016 used,   440528 free,     5556 buffers
KiB Swap:   524284 total,        0 used,   524284 free,    28604 cached
Done!
Now that you have everything running on the USB stick, it's time to update and upgrade Debian:
Code: Select all
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get autoremove
And off you go...

Changing the process back to the SD card
If you ever run into problems booting from the USB stick, you can try to move the original /boot/cmdline.txt back by putting the SD card back into your PC, and use the Filemanager to copy the file /boot/cmdline.txt to something else (/boot/cmdline.usb) and copy the name of /boot/cmdline.orig to /boot/cmdline.txt. (watch out: do not use rename, because Windows may rename the file called cmdline.txt to cmdline.txt.txt !)
If your system still works with the USB stick, you can also mount the windows partition on the SD card and make the changes to the cmdline files from within Debian.

Before you do that however, remember that we modified the /etc/fstab file on the SD card and then moved it to the USB stick. We need to change the contends back on the SD card to the original settings before we can easily change the filesystem back to the SD card.
If you followed the procedure above, do the following:
Code: Select all
pi@raspberrypi ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.4G  2.5G  4.6G  36% /
/dev/root       7.4G  2.5G  4.6G  36% /
devtmpfs        235M     0  235M   0% /dev
tmpfs            49M  240K   49M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            98M     0   98M   0% /run/shm
/dev/mmcblk0p1   56M   19M   38M  33% /boot
Use the device name without the partition (p1)
Code: Select all
pi@raspberrypi ~ $ sudo fdisk /dev/mmcblk0

Command (m for help): p

Disk /dev/mmcblk0: 4035 MB, 4035969024 bytes
4 heads, 16 sectors/track, 123168 cylinders, total 7882752 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00014d34

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1            8192      122879       57344    c  W95 FAT32 (LBA)
/dev/mmcblk0p2          122880     7882751     3879936   83  Linux

Command (m for help): q
So now that we know what the Linux file system partition is (p2), we can mount it.
Code: Select all
pi@raspberrypi ~ $ sudo mount /dev/mmcblk0p2 /mnt
And check it
Code: Select all
pi@raspberrypi ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.4G  2.5G  4.6G  36% /
/dev/root       7.4G  2.5G  4.6G  36% /
devtmpfs        235M     0  235M   0% /dev
tmpfs            49M  240K   49M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            98M     0   98M   0% /run/shm
/dev/mmcblk0p1   56M   19M   38M  33% /boot
/dev/mmcblk0p2  3.7G  1.7G  1.9G  47% /mnt
OK, now we can edit the fstab file on the SD card so it looks like the original version again.
Code: Select all
pi@raspberrypi ~ $ sudo nano /mnt/etc/fstab
And check it.
Code: Select all
pi@raspberrypi ~ $ sudo cat /mnt/etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that
#filesystem on USB stick

pi@raspberrypi ~ $

With a bit of luck, by just doing a copy of the right /boot/cmdline version on the Windows partition of the SD card, you will be able to change the booting process from the SD card. If not, there is enough information in this procedure to help you out I hope. ;)


Extended procedure:

Because we must force Debian to boot from the right USB device, we need to assign unique identifiers for the partition and the filesystem, rather than the names the system gives to the USB devices, like /dev/sda.

I'm assuming that after reading the above, we can be a little more terse.
Instead of using the standard fdisk command, we will use the more modern version gdisk.
We can install gdisk with:
Code: Select all
sudo apt-get install gdisk

We start the partitioning with:
Code: Select all
sudo gdisk /dev/sda

And we actually use the same sequence as with fdisk. First "d" for delete, then "n" for new and "w" to write it all out. Again use the defaults by hitting return. If you want more partitions, now is the time.

When you use the "i" command (information), you can see the following details:
Code: Select all
pi@raspberrypi ~ $ sudo gdisk /dev/sda
GPT fdisk (gdisk) version 0.8.5

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): i
Using 1
Partition GUID code: 0FC63DAF-8483-4772-8E79-3D69D8477DE4 (Linux filesystem)
Partition unique GUID: 2C7FD5E5-5CC4-434C-BFF6-F45336F7140B
First sector: 2048 (at 1024.0 KiB)
Last sector: 15669214 (at 7.5 GiB)
Partition size: 15667167 sectors (7.5 GiB)
Attribute flags: 0000000000000000
Partition name: 'Linux filesystem'

Command (? for help):
And this is what we need: Partition unique GUID: 2C7FD5E5-5CC4-434C-BFF6-F45336F7140B

This information (PARTUUID) goes into the /boot/cmdline.txt like so:
Code: Select all
pi@raspberrypi ~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=PARTUUID=2C7FD5E5-5CC4-434C-BFF6-F45336F7140B rootfstype=ext4 elevator=deadline rootwait rootdelay=5
pi@raspberrypi ~ $
Don't forget to add the rootdelay!

[UPDATE]
With newer versions of the kernel, the PARTUUID should now be obtained from executing the following :
Code: Select all
sudo blkid
It will return something like :
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="947B-B89A" TYPE="vfat" PARTUUID="6f92008e-01"
/dev/mmcblk0p2: UUID="eda95d8a-9cdd-4224-96d0-890b4791600c" TYPE="ext4" PARTUUID="6f92008e-02"
/dev/mmcblk0: PTUUID="6f92008e" PTTYPE="dos"


Depending on the power consumption of your stick or drive, you may need to edit /boot/config.txt and add:
max_usb_current=1

We now continue with formatting, mounting, installing rsync and the mirroring:
Code: Select all
sudo mke2fs -t ext4 -L rootfs /dev/sda1
sudo mount /dev/sda1 /mnt
sudo apt-get install rsync
sudo rsync -axv / /mnt
Now we need to get a unique identifier for the fstab drive information:

Code: Select all
pi@raspberrypi ~ $ sudo tune2fs -l /dev/sda1
tune2fs 1.42.5 (29-Jul-2012)
Filesystem volume name:   rootfs
Last mounted on:          /
Filesystem UUID:          3d2cb3c9-7e2b-4255-ac63-863bd3ad7993
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
...
pi@raspberrypi ~ $
And this is what we need: Filesystem UUID: 3d2cb3c9-7e2b-4255-ac63-863bd3ad7993

Next we can enter the information into the filesystem table, /etc/fstab (on the stick, not the SD card). Add the following line:
Code: Select all
/dev/disk/by-uuid/3d2cb3c9-7e2b-4255-ac63-863bd3ad7993    /   ext4    defaults,noatime  0       1

And put a comment in front of the reference on the SD card:
Code: Select all
#/dev/mmcblk0p2  /               ext4    defaults,noatime       0       1

And that's it. If you have checked everything ([d]don't mix up the partition GUID and the filesystem UUID![/d]) , you can reboot now and check if all is well.

Have fun! :)

[edit] I have made the changes based on the input from violagirl23 on sept-26-2013. Thank you girl!
[edit] This is a post that will be useful in relation to this one:
How to force a file check on both partitions during the boot process: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=113669
[edit] Added the edit to /boot/config.txt to add "max_usb_current=1" to allocate more juice. (tks DougyLawson)
Last edited by paulv on Fri May 13, 2016 6:12 am, edited 16 times in total.
Posts: 478
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands
by evansg515 » Sun May 19, 2013 7:00 am
Amazing thank you.
Gareth Evans

@pukkapi
www.pukkapi.co.uk
Posts: 7
Joined: Wed May 15, 2013 10:24 am
by bugloss » Thu Jun 20, 2013 7:44 pm
This is very good. It has solved the problem that I was having with gphoto2 writing to both the hard drive and the sd card.

Just a thought though. I made a copy of my sd card image with pi cloner before I started, as I had been working on it for a week and didn't want to loose what I had.
http://www.adamwilson.co
Posts: 22
Joined: Fri Jan 04, 2013 2:44 pm
by paulv » Thu Jun 20, 2013 8:33 pm
Great, I'm glad this was useful for you.

BTW, I had never heard of PiCloner, had to Google it.
For others, here is the link : http://sourceforge.net/projects/picloner/
Posts: 478
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands
by bugloss » Sat Jun 22, 2013 12:05 pm
paulv wrote:Great, I'm glad this was useful for you.

BTW, I had never heard of PiCloner, had to Google it.
For others, here is the link : http://sourceforge.net/projects/picloner/

It's best to run PiCloner overnight as it takes sooooooooooo looooooooooooooooong to do it's thing, but it works well nevertheless.
http://www.adamwilson.co
Posts: 22
Joined: Fri Jan 04, 2013 2:44 pm
by minkowski » Sat Jun 22, 2013 2:33 pm
There's something I don't quite understand and that bothers me for some time now.
Why is it good to move the OS to a USB stick (I'm not speaking about a proper hard drive attached via USB). They also use flash storage so they too have a limited number of read/write cycles. So what's the advantage?
Posts: 52
Joined: Tue Jan 29, 2013 12:10 pm
by portets » Sat Jun 22, 2013 7:19 pm
The right USB stick can be much faster than an SD card. Especially USB3.0 sticks, even in a 2.0 port.

But more importantly(for some), overclocking core_freq causes SD card corruption, but doesn't corrupt USB drives.
Posts: 185
Joined: Sat Oct 29, 2011 6:24 am
by Joaogl » Sun Jul 07, 2013 9:52 am
I've no Idea why but yesterday it worked and this morning when I woke up it had stoped working....

I've rebooted the pi and it worked again but soon like 1 or 2 minutes later it has sent like 100 messages saying this:

Message from syslog@raspberrypi at Jul 7 10:32:27 ...
kernel:[84.503122] journal commit I/O error

and them

[119.737956] EXT4-fs error (device sda1): ext4_find_entry1209: imode #19136519: comn crom: reading firectory Iblock 0

and I just cant do any thing... cant use any bash command and login doest work on ssh...

Why? How to solve this?
Posts: 16
Joined: Mon Jun 17, 2013 11:18 pm
by Joaogl » Sun Jul 07, 2013 10:11 am
I got I few lines now through ssh:

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.896008] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.945980] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.974871] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.997791] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 613.031156] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 613.064856] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.093393] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.102644] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.149364] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.180717] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.213619] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.222739] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.253678] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.286152] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.313373] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.322353] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.373398] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.382071] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.417307] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.453529] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.461924] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.499471] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.499481] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500010] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500314] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500753] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500791] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.501062] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.501333] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.501740] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.502662] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.502865] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.502921] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503072] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503098] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503137] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503433] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503605] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503616] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504178] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504355] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504517] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504810] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504984] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505070] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505077] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505192] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505199] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505279] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505310] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505317] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505439] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505728] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505760] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505853] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505859] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505887] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505955] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.506074] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.506268] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.506295] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.510140] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.512039] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.512103] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.512225] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.513000] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.552007] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.552412] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.710757] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.710767] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.712624] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.712752] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.712776] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.721582] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.722073] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.726739] journal commit I/O error
Posts: 16
Joined: Mon Jun 17, 2013 11:18 pm
by wheezer » Tue Jul 09, 2013 11:17 pm
Thanks for taking the time. I had tried two other procedures and while I now know that too were ok they missed one "small" point. Fortunately for me you did not miss this point. It was the "rootdelay=5" on the /boot/cmdline.txt file. I new that the boot process was not loading the correct file system but why. Now I know. Another day gone but it did end on a happy note.

Great effort.
Posts: 2
Joined: Tue Jul 09, 2013 11:10 pm
by ejsuncy » Thu Jul 11, 2013 4:42 am
Great article! The only one that worked for me...after several tries. One question though, is it safe to enable write-lock on the SD card now, so that it won't write (read corrupt) anything to it? I also understand that the pi must read from the SD card upon booting so it will still count toward the read/write limit, correct? And we've at LEAST doubled the life of the SD card (and then some!).

One more question: when the SD card eventually DOES get corrupted and it's time to replace it, is the process simple for replacing it? should I just make a backup of the SD card now (minus rootfs) and will i be able to just restore that backup to the new SD card? or is the process more involved?

Thanks for all your help!
Posts: 2
Joined: Wed Jan 02, 2013 8:57 pm
by FM81 » Thu Jul 11, 2013 7:14 am
ejsuncy wrote:One question though, is it safe to enable write-lock on the SD card now, so that it won't write (read corrupt) anything to it? I also understand that the pi must read from the SD card upon booting so it will still count toward the read/write limit, correct? And we've at LEAST doubled the life of the SD card (and then some!).
As far as I know: the Write-Protect-Switch is on the PCB not connected - so this doesn't matter! But after booting the SD-card is not needed any more - means also you can (for example) mount it only on demand.

If you plan to put the ROOTFS in every case on USB: there is no need to have more than one partition on SD-card. The FAT32 is enough - and can be easy restored on every PC.

MfG, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!
Posts: 411
Joined: Wed Apr 17, 2013 4:33 pm
by JustThisGuy » Sun Jul 21, 2013 11:53 pm
Just got around to doing this for my stay-at-home Pi.

Thanks Paul, you saved me a lot of investigative work...

One nit. I nearly got off track when updating the /etc/fstab file. Perhaps the instructions could specify that it is the fstab on the stick/hd and not the one on the SD.

Thanks again.
Any conversation about a sufficiently complex subject is indistinguishable from babble.
Posts: 114
Joined: Thu Jan 05, 2012 11:22 pm
by rpdom » Mon Jul 22, 2013 3:29 am
ejsuncy wrote:One more question: when the SD card eventually DOES get corrupted and it's time to replace it, is the process simple for replacing it? should I just make a backup of the SD card now (minus rootfs) and will i be able to just restore that backup to the new SD card? or is the process more involved?


You could zip up the files that are in /boot and copy the zipfile to another computer - even one that runs windows. Do that after any rpi-update or any time the files in /boot get updated.

If your SD card fails, you get a new one - lowest capacity/cheapest you can find, shouldn't be any need to format it, as they come ready formatted for windows in most cases, then just unzip the files onto it. Easy :)
User avatar
Posts: 9439
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK
by smartroad » Sun Aug 04, 2013 8:53 am
Just done this with an old harddrive I have laying about. Works a treat! Pi also seems to boot a little faster (although aware that could be me wanting it to be faster lol).

Many thanks :)
by derailleur » Mon Aug 05, 2013 6:49 am
This most clearest, and cleaniest instruction so far. Had it working the first go. Next for me is to mount the old / to something else. Thanks heaps.
Posts: 1
Joined: Mon Aug 05, 2013 6:41 am
by kalehrl » Mon Aug 05, 2013 8:04 am
The instructions are great!
Worked for me as well.
I'm also thinking of using the old / for something else.
It should be easy to reformat it to ext4 and readd it in the fstab file.
Posts: 350
Joined: Tue Jul 24, 2012 10:49 am
by JustinRSharp » Sat Aug 17, 2013 2:30 am
Just wondering, ive heard people say that moving the filesystem to the usb is faster but what about compared to a UHS-1 / Class 10 16gb sandisk sd card? as far as read / write im pretty sure that the sd card is faster.... its rated at 95Mbps....

Any comments?
(excuse my ignorance of what might be the obvious)
Posts: 8
Joined: Thu Aug 15, 2013 6:23 am
by itimpi » Sat Aug 17, 2013 9:03 am
There are two things that are relevant:
- Card ratings are based on serial writes. They do not necessarily tell you about random read and write performance which is important for the pi.
- You are constrained by the interface speeds supported by the hardware. The SD interface is reported as peaking at a bit above 20Mbps.

My experience is that a USB stick normally gives performance that is comparable to the SD card but seems less prone to corruption. Performance using a USB connected 2.5" HD is definitely snappier.
Posts: 1049
Joined: Sun Sep 25, 2011 11:44 am
Location: Potters Bar, United Kingdom
by deechster » Sun Aug 25, 2013 11:53 pm
itimpi,

Excellent HOW-to document. Followed it with out a hitch. I now have my pi running off an old 80GB 10,000RPM WD Raptor drive I had laying around.

Recommend pinning this one or someone should add it to the wiki. Great job!

Deechster
Posts: 10
Joined: Tue Nov 29, 2011 9:35 pm
by JPB-HK » Wed Aug 28, 2013 6:44 am
Thanks for the great instructions, worked perfectly the first time, and easy to put back if needed.

I think this is going to be a great development setup as you can leave the SD card alone, and plug in whatever USB flash drive you are working on. That way you could have several projects going, one on each USB flash drive, and plug in the one you are currently working on. Saves wear and tear on the SD card slot.

Thanks,
John
NCR 310 - ZX81 - Apple II+ - Dell/Intel - Mac mini... Raspberry Pi!
Posts: 54
Joined: Thu Nov 22, 2012 5:00 pm
by violagirl23 » Fri Sep 27, 2013 5:35 am
Shouldn't the pass number for / be 1? Otherwise fsck would never run on the root filesystem, which probably isn't good, right? So it would be either
Code: Select all
/dev/sda1    /   ext4    defaults,noatime,async  0       1
for the standard tutorial, or
Code: Select all
UUID=3d2cb3c9-7e2b-4255-ac63-863bd3ad7993    /   ext4    defaults,noatime,async  0       1
for the extended version? Also, I THINK the async option is already included in defaults for ext4, so you might not need to explicitly state it. [https://wiki.debian.org/fstab: defaults - default mount settings (equivalent to rw,suid,dev,exec,auto,nouser,async).]

As a side note, I was getting harmless errors at boot-time that said findfs: unable to resolve 'UUID="myUUID"', where myUUID is the UUID of my flash drive. Per the suggestion here: http://lists.debian.org/debian-user/2011/05/msg00721.html, changing my fstab format from UUID=myUUID to /dev/disk/by-uuid/myUUID fixed the problem. So for your tutorial, it would be the following:
Code: Select all
/dev/disk/by-uuid/3d2cb3c9-7e2b-4255-ac63-863bd3ad7993    /   ext4    defaults,noatime,async  0       1
Might be a nice note to include for users who are bothered by that harmless error message!
Posts: 7
Joined: Tue Sep 24, 2013 5:47 pm
by gkreidl » Fri Sep 27, 2013 9:03 am
Thanks, violagirl23,

I've used this fine tutorial (the complex variant) to transfer raspbian to an usb HD and was a bit irritated by the error message, although it was working fine. Now it's gone (and fsck is run at boot time again).
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer
Posts: 4462
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany
by paulv » Fri Sep 27, 2013 11:32 am
Thank you for the valuable inputs. I have changed the original post to include all your suggestions.
Well done girl!
Paul
Posts: 478
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands
by NaCaSi » Sun Nov 03, 2013 6:41 pm
Thanks for the excellent tutorial, it worked flawlessly for me. I have some questions:

1. In your section on changing back to the SD card you say that fstab on the SD card must be changed back, but in the original procedure you changed fstab on the USB drive. It seems to me that fstab on the SD card is still the same, and no changes are necessary. Is this right, or was fstab on the SD card changed implicitly somehow?

2. You change the swapfile to 512MB, but the Pi only uses part of the ram for the CPU. It seems to me that a more appropriate size would be equal to the memory allocated for the CPU. Does the OS also swap out memory used by the GPU? Is there value in having a swapfile larger than usable memory?
Posts: 1
Joined: Sun Nov 03, 2013 6:22 pm