HOWTO: Move the filesystem to a USB stick/Drive [updated]


311 posts   Page 1 of 13   1, 2, 3, 4, 5 ... 13
by paulv » Sat May 18, 2013 1:26 pm
This procedure will allow you to run Debian from a USB connected drive, instead of the SD card.

The RPi will still boot from the SD card, but in essence, will no longer write to it. The original idea was to keep the SD card from failing due to excessive writing to it. It will NOT protect you from disk corruption when you pull the power plug without first properly shutting down or halting the RPi. Depending on your setup, this procedure can provide some overall speed improvements, but greatly removes the speed, size and reliability requirements for the SD card.

Starting with the 2017-04-10 kernel version, the Foundation changed to a more reliable way of using drive identifiers. This post still shows both methods, but note that using /dev/sd* for the filesystem is no longer recommended. Please implement the procedure for "Multiple Drives" further down. to stay compatible.


This method is not for NOOBS installations!
You can use Jessie or Jessie Lite, other versions may work but are not tested.

Using a USB "stick"
If you're looking for a speed improvement, this procedure will be a disappointment for you when you are done. The speed advantage, if at all, is at the theoretical very best, only 2x.

Using a "real" disk drive
Things change dramatically however when you use a "real" USB disc or even better, an SSD drive, although the maximum speed is still limited due to the architecture of the RPi hardware.

If you have an RPi Model 3, you can also try out the latest method of displacing the SD card completely, and really boot from your USB drive. Have a look here, but remember, this is still under development at this very moment and may not work with all drives or in all situations.
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md

1. Installation
Power down your RPi or leave it powerless.
Connect your USB stick or drive to the USB slot of the RPI now. Only use one stick or drive for this procedure! If you have a real hard disk, it is even more important that you connect that to the USB port before you boot. Older model Pi's cannot handle the power surge when they are running, and will reboot abruptly and can damage the SD card.

After a reboot, check to see if your USB drive is recognized:
Code: Select all
pi@raspberrypi:~ $ lsusb
Bus 001 Device 006: ID 413c:2002 Dell Computer Corp. SK-8125 Keyboard
Bus 001 Device 005: ID 413c:1002 Dell Computer Corp. Keyboard Hub
Bus 001 Device 004: ID 0781:5571 SanDisk Corp. Cruzer Fit
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. LAN9500 Ethernet 10/100 Adapter / SMSC9512/9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device 004 is the USB stick I'm using. The ID shows a unique manufacturer identifier.
Also check to see if the OS recognized the device: (you can also use the command dmesg | grep sda)
Code: Select all
pi@raspberrypi:~ $ cat /var/log/messages | grep sda
Nov 25 17:56:35 raspberrypi kernel: [    4.024472] sd 0:0:0:0: [sda] 61489152 512-byte logical blocks: (31.5 GB/29.3 GiB)
Nov 25 17:56:35 raspberrypi kernel: [    4.025804] sd 0:0:0:0: [sda] Write Protect is off
Nov 25 17:56:35 raspberrypi kernel: [    4.026652] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Nov 25 17:56:35 raspberrypi kernel: [    4.063316]  sda: sda1 sda2
Nov 25 17:56:35 raspberrypi kernel: [    4.081595] sd 0:0:0:0: [sda] Attached SCSI removable disk
Jan 13 14:25:51 raspberrypi kernel: [    3.993300] sd 0:0:0:0: [sda] 61489152 512-byte logical blocks: (31.5 GB/29.3 GiB)
Jan 13 14:25:51 raspberrypi kernel: [    3.995088] sd 0:0:0:0: [sda] Write Protect is off
Jan 13 14:25:51 raspberrypi kernel: [    3.996142] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:25:51 raspberrypi kernel: [    4.017894]  sda: sda1 sda2
Jan 13 14:25:51 raspberrypi kernel: [    4.022524] sd 0:0:0:0: [sda] Attached SCSI removable disk
Jan 13 14:36:48 raspberrypi kernel: [  691.357635] sd 1:0:0:0: [sda] 61489152 512-byte logical blocks: (31.5 GB/29.3 GiB)
Jan 13 14:36:48 raspberrypi kernel: [  691.358933] sd 1:0:0:0: [sda] Write Protect is off
Jan 13 14:36:48 raspberrypi kernel: [  691.359948] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:36:48 raspberrypi kernel: [  691.379078]  sda: sda1
Jan 13 14:36:48 raspberrypi kernel: [  691.384786] sd 1:0:0:0: [sda] Attached SCSI removable disk
Jan 13 14:37:05 raspberrypi kernel: [    3.993309] sd 0:0:0:0: [sda] 61489152 512-byte logical blocks: (31.5 GB/29.3 GiB)
Jan 13 14:37:05 raspberrypi kernel: [    3.995048] sd 0:0:0:0: [sda] Write Protect is off
Jan 13 14:37:05 raspberrypi kernel: [    3.996068] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 13 14:37:05 raspberrypi kernel: [    4.002434]  sda: sda1
Jan 13 14:37:05 raspberrypi kernel: [    4.011623] sd 0:0:0:0: [sda] Attached SCSI removable disk
[sda] Attached SCSI removable disk < ===

Yes, the USB stick can be referenced as /dev/sda

There is no way of knowing what name the OS assigns to additional USB drives. It could also be /dev/sdb. To prepare for the eventuality of you adding another USB drive or stick, we are going to prepare the RPI for that now.

You need to install some programs that are needed for this procedure.
Run:
Code: Select all
sudo apt-get update && sudo apt-get install rsync gdisk

To prepare the drive to hold a file system, we are going to create one or more partitions, and then format them.
You should not use fdisk or parted, because that stops you from adding drives reliably in the future.
If you know fdisk/parted, gdisk works the same, but adds unique partition identifiers we need later on.

Creating Partition(s)
Code: Select all
pi@raspberrypi:~ $ sudo gdisk /dev/sda
GPT fdisk (gdisk) version 0.8.10

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************

Warning! Main partition table overlaps the first partition by 2 blocks!
Try reducing the partition table size by 8 entries.
(Use the 's' item on the experts' menu.)

Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help):
The error message is because there is a partition already, formatted for Windows that was installed by ScanDisk by default. We're going to re-partition and re-format the drive for Linux.
Enter d to delete partion(s). It will say "Using (partition) 1", that's the default and OK.

Note
I'm now recommending that you pause for a moment, and consider to create two partitions as a minimum. If you have a large USB disk, you may even want more, and now is the time to do it. I will be creating two partitions on my stick. Partition 1 will hold the Raspbian filesystem, and partition 2 will hold my data. I like to keep those two separate as much as I can.

The first partition, for Raspian, can be any size you want (but larger than 2 GB). I'm going to create an 8GB partition, and use the rest for data. The size of the Raspian partition does matter, because the larger it is, the more room the "internal" file system on the card/stick has to move blocks around and optimize the "wear" leveling on these flash drives. If you have a "real" disk, this does not matter.
If you want more partitions, do as I do with the second, just keep track of the size. You may want to put that disk structure on paper first because backtracking is very difficult unless you are an expert.

Create the partitions:
Enter n to create a new partition, and select number 1:
Select the starting sector by hitting Return, then select +8G for the size 0f 8GByte. Now select the default filesystem ('Linux filesystem') by hitting Enter again.
Code: Select all
Command (? for help): d
Using 1

Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-61489118, default = 64) or {+-}size{KMGTP}:
Last sector (64-61489118, default = 61489118) or {+-}size{KMGTP}: +8G
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help):
OK, now the second partition. I will use one that fills the rest of the disk. If you want more, use the default for the start, and give the size again, and continue until you are done.
Code: Select all
Command (? for help): n
Partition number (2-128, default 2):
First sector (34-61489118, default = 16777280) or {+-}size{KMGTP}:
Last sector (16777280-61489118, default = 61489118) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help):
Now use v for verify to see if everything worked. If you don't like what you see, of if you changed your mind, or made a mistake, no sweat, just hit Ctrl-C and start again. If you are happy, hit w for write to make it permanent. You get one more chance to make up your mind.
Code: Select all
Command (? for help): v

No problems found. 30 free sectors (15.0 KiB) available in 1
segments, the largest of which is 30 (15.0 KiB) in size.

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.
Phew, take a deep breath again, you need the oxygen for the next step.

Formatting the partition(s)

You need to format every partition you created in the previous step. After the partitioning, we now have more devices. device /dev/sda now has two children, called sda1 and sda2 or even more if you were (re)productive.

Code: Select all
pi@raspberrypi:~ $  sudo mke2fs -t ext4 -L rootfs /dev/sda1
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: ecd95e15-0cdb-46d6-b705-b03b1ad5cb09
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Do the same for the second partition :
Code: Select all
sudo mke2fs -t ext4 -L my_data /dev/sda2
-L is for label, you can call the additional partitions anything you want.

Mounting the root partition

In order to use the new partitions, we need to mount them. Initially we're only going to mount the first partition that will hold the Raspian files (also called rootfs):
Code: Select all
pi@raspberrypi ~ $ sudo mount /dev/sda1 /mnt
Let's see what we've done so far:
Code: Select all
pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.2G  885M  244M  79% /
devtmpfs        237M     0  237M   0% /dev
tmpfs           242M     0  242M   0% /dev/shm
tmpfs           242M  4.5M  237M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           242M     0  242M   0% /sys/fs/cgroup
/dev/mmcblk0p1   63M   21M   42M  33% /boot
/dev/sda1       7.8G   18M  7.4G   1% /mnt
We made the 8GB drive /dev/sda1 available as /mnt for Raspian, so now we can start to use it.

We can copy (mirror) the Debian filesystem data currently on the SD card in /dev/root to the new drive. You can use dd, but that is a bit copy program and, because it also copies the empty space on the SD card, it is very, very slow.
It is better to use rsync that we already installed earlier.
Rsync is a lot faster, but it will still take some time to create a mirror image of the complete file system on the USB drive. We will copy everything from the root "/" to the just mounted drive /mnt (which is /dev/sda1 at the moment)

Code: Select all
pi@raspberrypi ~ $ sudo rsync -axv / /mnt
...
Because of the -v flag, you can see the files flashing by on the screen, so you can see the progress. You also get an appreciation about the complexity of a complete system.
On my Model B system it took only 5 minutes. Make sure this process does not get interrupted! If there are errors, have a look to see if it is something obvious, or just try the rsync again, or even the whole process. Don't continue unless this worked without errors.
Code: Select all
var/tmp/ <== this is the last file

sent 858,620,180 bytes  received 556,875 bytes  3,141,415.19 bytes/sec
total size is 856,440,892  speedup is 1.00


Setting up the new boot procedure

We're now going to tell the system where to find the new filesystem when we are going to boot from the new drive.
The first file we need to change is /boot/cmdline.txt on the SD card.

Before we do that, we're going to create backup copies of that important file so we can easily switch back and forth to booting from the SD card or from the USB disk.
Code: Select all
sudo cp /boot/cmdline.txt /boot/cmdline.sd
This will hold the original boot instructions.
The contents of this file is one very long command string, and we need to replace the reference to the partition on the SD card (root=/dev/mmcblk0p2) to the new USB drive (/dev/sda1)
Here is a copy of the original file on my SD card:
Code: Select all
pi@raspberrypi:~ $ sudo nano /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet
We are going to change this portion :
root=/dev/mmcblk0p2
Here is where we are going to decide what we're going to use. If you only use one extra drive -at this moment- (it does not matter how many partitions are on it), you can safely use:
Code: Select all
root=/dev/sda1
And that's it.

NOTE 1
To be on the safe side, there is one more command we should add to that single line. During the power-up/boot sequence, it may take the USB drive a little longer to spin up to speed and be ready, than the OS expects. The default waiting period seems to be 2 seconds for USB devices. If we add a delay of 5 seconds (rootdelay=5), the USB drive should be ready when it's needed. If not, use an even longer period. When you use a stick or SSD drive this will most likely not be needed.
[update] The more up-to-date method now seems go towards adding
Code: Select all
program_usb_timeout=1
to the /boot/config.txt file. I'm not sure how much longer the rootdelay option for cmdline.txt will be usable in the future.

NOTE 2
Depending on the power consumption of your stick or drive, and if you use an RPI that supports it, you can add:
Code: Select all
max_usb_current=1
in /boot/config.txt to let the RPI send more juice to the drive.

Make sure again the command in /boot/cmdline.txt is still only one line! Save the file and close the editor. Also save this version of the file so we can switch back and forth.
Code: Select all
sudo cp /boot/cmdline.txt /boot/cmdline.usb



Mounting the new drive during boot

The next step is to actually mount the rootfs system during the boot process. This is handled by the file system during the later stages of the boot process itself. It uses the information located in the file /etc/fstab.
NOTE We are only going to change the data on the new drive, so we keep the SD card unchanged. This will make it easier to go back and forth with only changing the /boot/cmdline.txt file on the SD card.

Code: Select all
sudo nano /mnt/etc/fstab
It will show something like this:
Code: Select all
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
We will now change the reference to /dev/mmcblk0p2 (on the SD card) and use the new USB drive instead. You can copy the whole line starting with /dev/mmcblk0p2 and comment one out, and the change the other, or just change /dev/mmcblk0p2 to
Code: Select all
/dev/sda1
Leave the rest of the line the same. Save the file and close the editor.
We are ready to reboot the RPI now. Even when you will need to go to the extended version, try this out first.

Code: Select all
sudo reboot
It helps if you still have a console connected so you can see what is going on during the boot process. And hold your breath again! It will most likely work first time.
If something goes wrong however, you need to look at the error messages and back-track carefully. This procedure really works, if you follow it carefully and not make short-cuts or changes. Make it work first, then change whatever you want later.

Adding the second Partition
If you now want to add the second partition on the USB drive, the one I designated for data, do as follows. First create a place holder to mount to.
Code: Select all
sudo mkdir /mnt/my_data
sudo chown pi:pi /mnt/my_data
The last statement will set the ownership rights so you don't need to "sudo" to read/write to this drive. To make it permanently available, you can add one more line to /etc/fstab, like so :
Code: Select all
/dev/sda2  /mnt/my_data               ext4    defaults  0       1


Mine looks like this now:
Code: Select all
pi@raspberrypi:~ $ cat /etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/sda1       /               ext4    defaults,noatime  0       1
/dev/sda2       /mnt/my_data    ext4    defaults        0       1

#/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

Reboot, or use "sudo mount -a" and see if you can access that partition too (ls -al /mnt/my_data), it should only have a "lost+found" file.

Multiple Drives:
If you will use, or already have more disks that will be directly connected to the USB ports on the RPI, you need to make sure that the kernel always assigns what we now have in /dev/sda1 for the rootfs system. You can never be sure if the drive will be assigned sda or sdb when the system sees multiple drives of whatever kind.

Here's how you do that. I may oversimplify this a bit, but every drive, and in addition, every partition we just created now has a unique identifier (thanks to gdisk). There is a UUID for the drive itself, and a PARTUUID for every partition. Here is where you can see that :
Code: Select all
pi@raspberrypi:~ $ sudo blkid
/dev/mmcblk0p1: LABEL="boot" UUID="0F5F-3CD8" TYPE="vfat" PARTUUID="244b8248-01"

/dev/mmcblk0p2: UUID="0aed834e-8c8f-412d-a276-a265dc676112" TYPE="ext4" PARTUUID="244b8248-02"

/dev/sda1: LABEL="rootfs" UUID="ecd95e15-0cdb-46d6-b705-b03b1ad5cb09" TYPE="ext4" PARTLABEL="Linux filesystem" PARTUUID="72d9a26d-1064-4ff0-8e0a-c8ee1102b296"

/dev/mmcblk0: PTUUID="244b8248" PTTYPE="dos"

/dev/sda2: LABEL="my_data" UUID="5cf4fde0-8c39-4b3d-be7a-0236b2e217e0" TYPE="ext4" PARTLABEL="Linux filesystem" PARTUUID="049e6bac-b5d8-4721-9e3e-e8b03f730067"
(empty lines added for clarity)

For the cmdline.txt file in /boot, we're going to tell the system what partition to boot from, so we need the PARTUUID for that. Instead of "root=/dev/sda1", we now need to use from blkid:
Code: Select all
root=PARTUUID=72d9a26d-1064-4ff0-8e0a-c8ee1102b296
First highlight that string with your mouse to copy it. Open the editor
Code: Select all
sudo nano /boot/cmdline.txt
and past that into the open editor instead of "root=/dev/mmcblk0p2" or "/dev/sda1" as
Code: Select all
root=PARTUUID=72d9a26d-1064-4ff0-8e0a-c8ee1102b296
Make sure that the one line still stays one line, or it may not boot. You may need to add the same rootwait=5 to this line, or max_usb_current=1 to /boot/config.txt as in the simple method, so add that if you do. Also save this /boot/cmdline.txt file so we can switch back and forth later on.
Code: Select all
sudo cp /boot/cmdline.txt /boot/cmdline.usb
Check that you have three versions of this file before you move on. One .txt, the current version, one .sd the original one, and one .usb, the new one.

Fstab also needs to know about the actual drive it needs to mount, so we need to give it the UUID of what is currently /dev/sda1.
That information was also in the "sudo blkid" we ran before, so just highlight and paste the UUID code to replace "/dev/mmcblk0p2" or "/dev/sda1" with
Code: Select all
/dev/disk/by-uuid/ecd95e15-0cdb-46d6-b705-b03b1ad5cb09
and leave the rest of the line the same, or duplicate the original line and comment it out with "#".

With this you are again ready to reboot to see the results.
If something goes wrong, you can revert back to the easy method and replace the UUID's one by one to see where the error is. You cannot really break anything so just backtrack your steps and carefully make note of what I did. It should work!

If it does work, and you have everything running on the USB stick or drive, you can also add the second or more partitions to /etc/fstab, similar to what I showed earlier. My /etc/fstab looks like this with both drives added:
Code: Select all
pi@raspberrypi:~ $ cat /etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/disk/by-uuid/ecd95e15-0cdb-46d6-b705-b03b1ad5cb09  /       ext4    defaults,noatime        0       1
/dev/disk/by-uuid/5cf4fde0-8c39-4b3d-be7a-0236b2e217e0  /       ext4    defaults        0       1
# alternatives and backup:
#/dev/sda1       /               ext4    defaults,noatime  0       1
#/dev/sda2      /mnt/my_data    ext4    defaults        0       1
#/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that


If that also works, you can now update and upgrade Debian:
Code: Select all
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
sudo apt-get autoremove
And off you go...

Changing the process back to the SD card
If you ever run into problems booting from the USB drive, you can move the original /boot/cmdline.txt back by putting the SD card back into your PC, and use the Filemanager to copy the file /boot/cmdline.sd back to /boot/cmdline.txt. You don't need to do anything to the /etc/fstab file on either the SD card or the USB drive if you are going to boot from the SD card. (watch out: if you want to do this while the card is in your PC, do not use rename, because Windows may rename the file called cmdline.txt to cmdline.txt.txt !)
If your system still works with the USB stick, you can also make the changes to the cmdline files from within Debian (in /boot). You can also run a disk check in the USB drive when you have dismounted that drive and run from the SD card.

Note
One thing you need to keep track off is the difference in kernel versions between the boot code on the SD card and the file system on the USB drive when you apt-get upgrade. The original kernel software on the SD card may get out of synchronization with the newer boot files on the DOS partition. One way to fix that is to revert back to the SD card and also update that file system with apt-get upgrade.


This updated version/revision was made possible by the contributions of many users. Their posts are below, just be aware that many inputs or comments are now outdated, or hopefully integrated. It was the only way for me to keep new users from going through 12 pages (currently) of comments, discussions and other contributions.

Enjoy and stay tuned!
Paul
Last edited by paulv on Thu Apr 27, 2017 6:55 am, edited 51 times in total.
Posts: 520
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands
by evansg515 » Sun May 19, 2013 7:00 am
Amazing thank you.
Gareth Evans

@pukkapi
www.pukkapi.co.uk
Posts: 7
Joined: Wed May 15, 2013 10:24 am
by bugloss » Thu Jun 20, 2013 7:44 pm
This is very good. It has solved the problem that I was having with gphoto2 writing to both the hard drive and the sd card.

Just a thought though. I made a copy of my sd card image with pi cloner before I started, as I had been working on it for a week and didn't want to loose what I had.
http://www.adamwilson.co
Posts: 22
Joined: Fri Jan 04, 2013 2:44 pm
by paulv » Thu Jun 20, 2013 8:33 pm
Great, I'm glad this was useful for you.

BTW, I had never heard of PiCloner, had to Google it.
For others, here is the link : http://sourceforge.net/projects/picloner/
Posts: 520
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands
by bugloss » Sat Jun 22, 2013 12:05 pm
paulv wrote:Great, I'm glad this was useful for you.

BTW, I had never heard of PiCloner, had to Google it.
For others, here is the link : http://sourceforge.net/projects/picloner/

It's best to run PiCloner overnight as it takes sooooooooooo looooooooooooooooong to do it's thing, but it works well nevertheless.
http://www.adamwilson.co
Posts: 22
Joined: Fri Jan 04, 2013 2:44 pm
by minkowski » Sat Jun 22, 2013 2:33 pm
There's something I don't quite understand and that bothers me for some time now.
Why is it good to move the OS to a USB stick (I'm not speaking about a proper hard drive attached via USB). They also use flash storage so they too have a limited number of read/write cycles. So what's the advantage?
Posts: 52
Joined: Tue Jan 29, 2013 12:10 pm
by portets » Sat Jun 22, 2013 7:19 pm
The right USB stick can be much faster than an SD card. Especially USB3.0 sticks, even in a 2.0 port.

But more importantly(for some), overclocking core_freq causes SD card corruption, but doesn't corrupt USB drives.
Posts: 186
Joined: Sat Oct 29, 2011 6:24 am
by Joaogl » Sun Jul 07, 2013 9:52 am
I've no Idea why but yesterday it worked and this morning when I woke up it had stoped working....

I've rebooted the pi and it worked again but soon like 1 or 2 minutes later it has sent like 100 messages saying this:

Message from syslog@raspberrypi at Jul 7 10:32:27 ...
kernel:[84.503122] journal commit I/O error

and them

[119.737956] EXT4-fs error (device sda1): ext4_find_entry1209: imode #19136519: comn crom: reading firectory Iblock 0

and I just cant do any thing... cant use any bash command and login doest work on ssh...

Why? How to solve this?
Posts: 16
Joined: Mon Jun 17, 2013 11:18 pm
by Joaogl » Sun Jul 07, 2013 10:11 am
I got I few lines now through ssh:

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.896008] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.945980] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.974871] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 612.997791] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 613.031156] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:46 ...
kernel:[ 613.064856] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.093393] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.102644] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.149364] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.180717] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.213619] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.222739] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.253678] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.286152] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.313373] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.322353] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.373398] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.382071] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.417307] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.453529] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:47 ...
kernel:[ 613.461924] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.499471] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.499481] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500010] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500314] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500753] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.500791] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.501062] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.501333] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.501740] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.502662] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.502865] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.502921] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503072] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503098] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503137] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503433] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503605] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.503616] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504178] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504355] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504517] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504810] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.504984] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505070] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505077] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505192] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505199] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505279] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505310] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505317] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505439] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505728] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505760] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505853] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505859] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505887] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.505955] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.506074] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.506268] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.506295] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.510140] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.512039] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.512103] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.512225] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.513000] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.552007] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.552412] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.710757] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.710767] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.712624] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.712752] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.712776] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.721582] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.722073] journal commit I/O error

Message from syslogd@raspberrypi at Jul 7 11:09:48 ...
kernel:[ 613.726739] journal commit I/O error
Posts: 16
Joined: Mon Jun 17, 2013 11:18 pm
by wheezer » Tue Jul 09, 2013 11:17 pm
Thanks for taking the time. I had tried two other procedures and while I now know that too were ok they missed one "small" point. Fortunately for me you did not miss this point. It was the "rootdelay=5" on the /boot/cmdline.txt file. I new that the boot process was not loading the correct file system but why. Now I know. Another day gone but it did end on a happy note.

Great effort.
Posts: 2
Joined: Tue Jul 09, 2013 11:10 pm
by ejsuncy » Thu Jul 11, 2013 4:42 am
Great article! The only one that worked for me...after several tries. One question though, is it safe to enable write-lock on the SD card now, so that it won't write (read corrupt) anything to it? I also understand that the pi must read from the SD card upon booting so it will still count toward the read/write limit, correct? And we've at LEAST doubled the life of the SD card (and then some!).

One more question: when the SD card eventually DOES get corrupted and it's time to replace it, is the process simple for replacing it? should I just make a backup of the SD card now (minus rootfs) and will i be able to just restore that backup to the new SD card? or is the process more involved?

Thanks for all your help!
Posts: 2
Joined: Wed Jan 02, 2013 8:57 pm
by FM81 » Thu Jul 11, 2013 7:14 am
ejsuncy wrote:One question though, is it safe to enable write-lock on the SD card now, so that it won't write (read corrupt) anything to it? I also understand that the pi must read from the SD card upon booting so it will still count toward the read/write limit, correct? And we've at LEAST doubled the life of the SD card (and then some!).
As far as I know: the Write-Protect-Switch is on the PCB not connected - so this doesn't matter! But after booting the SD-card is not needed any more - means also you can (for example) mount it only on demand.

If you plan to put the ROOTFS in every case on USB: there is no need to have more than one partition on SD-card. The FAT32 is enough - and can be easy restored on every PC.

MfG, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!
Posts: 490
Joined: Wed Apr 17, 2013 4:33 pm
by JustThisGuy » Sun Jul 21, 2013 11:53 pm
Just got around to doing this for my stay-at-home Pi.

Thanks Paul, you saved me a lot of investigative work...

One nit. I nearly got off track when updating the /etc/fstab file. Perhaps the instructions could specify that it is the fstab on the stick/hd and not the one on the SD.

Thanks again.
Any conversation about a sufficiently complex subject is indistinguishable from babble.
Posts: 114
Joined: Thu Jan 05, 2012 11:22 pm
by rpdom » Mon Jul 22, 2013 3:29 am
ejsuncy wrote:One more question: when the SD card eventually DOES get corrupted and it's time to replace it, is the process simple for replacing it? should I just make a backup of the SD card now (minus rootfs) and will i be able to just restore that backup to the new SD card? or is the process more involved?


You could zip up the files that are in /boot and copy the zipfile to another computer - even one that runs windows. Do that after any rpi-update or any time the files in /boot get updated.

If your SD card fails, you get a new one - lowest capacity/cheapest you can find, shouldn't be any need to format it, as they come ready formatted for windows in most cases, then just unzip the files onto it. Easy :)
User avatar
Posts: 11019
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK
by smartroad » Sun Aug 04, 2013 8:53 am
Just done this with an old harddrive I have laying about. Works a treat! Pi also seems to boot a little faster (although aware that could be me wanting it to be faster lol).

Many thanks :)
by derailleur » Mon Aug 05, 2013 6:49 am
This most clearest, and cleaniest instruction so far. Had it working the first go. Next for me is to mount the old / to something else. Thanks heaps.
Posts: 1
Joined: Mon Aug 05, 2013 6:41 am
by kalehrl » Mon Aug 05, 2013 8:04 am
The instructions are great!
Worked for me as well.
I'm also thinking of using the old / for something else.
It should be easy to reformat it to ext4 and readd it in the fstab file.
Posts: 350
Joined: Tue Jul 24, 2012 10:49 am
by JustinRSharp » Sat Aug 17, 2013 2:30 am
Just wondering, ive heard people say that moving the filesystem to the usb is faster but what about compared to a UHS-1 / Class 10 16gb sandisk sd card? as far as read / write im pretty sure that the sd card is faster.... its rated at 95Mbps....

Any comments?
(excuse my ignorance of what might be the obvious)
Posts: 8
Joined: Thu Aug 15, 2013 6:23 am
by itimpi » Sat Aug 17, 2013 9:03 am
There are two things that are relevant:
- Card ratings are based on serial writes. They do not necessarily tell you about random read and write performance which is important for the pi.
- You are constrained by the interface speeds supported by the hardware. The SD interface is reported as peaking at a bit above 20Mbps.

My experience is that a USB stick normally gives performance that is comparable to the SD card but seems less prone to corruption. Performance using a USB connected 2.5" HD is definitely snappier.
Posts: 1069
Joined: Sun Sep 25, 2011 11:44 am
Location: Potters Bar, United Kingdom
by deechster » Sun Aug 25, 2013 11:53 pm
itimpi,

Excellent HOW-to document. Followed it with out a hitch. I now have my pi running off an old 80GB 10,000RPM WD Raptor drive I had laying around.

Recommend pinning this one or someone should add it to the wiki. Great job!

Deechster
Posts: 10
Joined: Tue Nov 29, 2011 9:35 pm
by JPB-HK » Wed Aug 28, 2013 6:44 am
Thanks for the great instructions, worked perfectly the first time, and easy to put back if needed.

I think this is going to be a great development setup as you can leave the SD card alone, and plug in whatever USB flash drive you are working on. That way you could have several projects going, one on each USB flash drive, and plug in the one you are currently working on. Saves wear and tear on the SD card slot.

Thanks,
John
NCR 310 - ZX81 - Apple II+ - Dell/Intel - Mac mini... Raspberry Pi!
Posts: 54
Joined: Thu Nov 22, 2012 5:00 pm
by violagirl23 » Fri Sep 27, 2013 5:35 am
Shouldn't the pass number for / be 1? Otherwise fsck would never run on the root filesystem, which probably isn't good, right? So it would be either
Code: Select all
/dev/sda1    /   ext4    defaults,noatime,async  0       1
for the standard tutorial, or
Code: Select all
UUID=3d2cb3c9-7e2b-4255-ac63-863bd3ad7993    /   ext4    defaults,noatime,async  0       1
for the extended version? Also, I THINK the async option is already included in defaults for ext4, so you might not need to explicitly state it. [https://wiki.debian.org/fstab: defaults - default mount settings (equivalent to rw,suid,dev,exec,auto,nouser,async).]

As a side note, I was getting harmless errors at boot-time that said findfs: unable to resolve 'UUID="myUUID"', where myUUID is the UUID of my flash drive. Per the suggestion here: http://lists.debian.org/debian-user/2011/05/msg00721.html, changing my fstab format from UUID=myUUID to /dev/disk/by-uuid/myUUID fixed the problem. So for your tutorial, it would be the following:
Code: Select all
/dev/disk/by-uuid/3d2cb3c9-7e2b-4255-ac63-863bd3ad7993    /   ext4    defaults,noatime,async  0       1
Might be a nice note to include for users who are bothered by that harmless error message!
Posts: 7
Joined: Tue Sep 24, 2013 5:47 pm
by gkreidl » Fri Sep 27, 2013 9:03 am
Thanks, violagirl23,

I've used this fine tutorial (the complex variant) to transfer raspbian to an usb HD and was a bit irritated by the error message, although it was working fine. Now it's gone (and fsck is run at boot time again).
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer
Posts: 5117
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany
by paulv » Fri Sep 27, 2013 11:32 am
Thank you for the valuable inputs. I have changed the original post to include all your suggestions.
Well done girl!
Paul
Posts: 520
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands
by NaCaSi » Sun Nov 03, 2013 6:41 pm
Thanks for the excellent tutorial, it worked flawlessly for me. I have some questions:

1. In your section on changing back to the SD card you say that fstab on the SD card must be changed back, but in the original procedure you changed fstab on the USB drive. It seems to me that fstab on the SD card is still the same, and no changes are necessary. Is this right, or was fstab on the SD card changed implicitly somehow?

2. You change the swapfile to 512MB, but the Pi only uses part of the ram for the CPU. It seems to me that a more appropriate size would be equal to the memory allocated for the CPU. Does the OS also swap out memory used by the GPU? Is there value in having a swapfile larger than usable memory?
Posts: 1
Joined: Sun Nov 03, 2013 6:22 pm