Raspbian on "external" USB hard drive


54 posts   Page 1 of 3   1, 2, 3
by ian.macarthur123 » Tue May 29, 2012 9:17 pm
(Note; I'm downloading the "pisces" image as I type this so...)

Has anyone tried a Raspbian image on an external drive?

Reports elsewhere suggest that this is a Big Win in performance terms for the regular Debian image, so I guess I'm hoping for a double boost from the armhf stuff combined with the "faster" external drive...

But may be sensitive to which kernel is in the image (though the pisces kernel is "fresh" AIUI so should be OK?)

Anyway, only had the R-Pi for a bit over a day now; works nicely with the stock Debian image, but Reeaallllyyy Slooooowwww - haven't tried the external disk with that yet either, but "morally" the hf work just seems to me to be the Right Thing To Do. (I work in embedded systems, and the thought of *not* leveraging the fp and vfp that are there just feels like a bad thing.)

And I suspect that the batch of SD cards I have are *particularly* slow...

Cheers,
--
Ian
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by mpthompson » Tue May 29, 2012 9:23 pm
Ian, I suspect that using a USB hard drive will indeed be a huge win with regards to performance. I'll be setting up a hard drive with the Raspbian build within the next day or so. I've heard concerns that there are still some USB stability issues in the kernel that may cause some grief, but I'm not certain about this. I guess I'll find out soon enough.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by dom » Tue May 29, 2012 9:51 pm
I think USB disks should be reliable with latest GitHub kernel. The last Debian image could kernel panic when doing ls of USB activity. You'll need a self powered drive, or a powered hub.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4030
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by dukla2000 » Tue May 29, 2012 9:59 pm
I swapped my rootfs to a USB stick the other day and got decent gains. Actually my first USB stick was the same Class 10 micro-SD card I did my first RasPi boot from but hdparm and bonnie made it clear the Pi USB ports are far faster than the SDcard slot.

Today went beserk and got a proper USB memory stick - went for USB 3 in the hope it has reasonably fast controller and chips (yes I know RasPi is USB 2). It murders my previous setup - will post hdparm & bonnie++ results when I get a bit further with Raspian but the one number I can remember is my stock Debian boot time dropped from 27 seconds to 20 seconds.
Posts: 109
Joined: Tue Jan 10, 2012 12:02 am
Location: Reading.UK.EU
by ian.macarthur123 » Tue May 29, 2012 10:09 pm
Yup - hardware-wise I'm probably OK (though not yet tested.)

I have two drives I'm considering using; a Samsung unit that draws (nominally) 850 mA, though in practice slightly less, or an old POS Phillips branded unit with its own PSU, that takes up more room than [a large number] of Pi's... The Samsung is favourite, since it is much more compact... And hooked to a powered hub, of course.

If that fails, the Phillips unit should be OK since it draws no power from the USB interface at all...

As an aside <thread drift alert!> - has anyone tried just using a beefy powered hub to power the Pi and all the peripherals?
I'm powering my Pi from a phone charger and that's fine, but I can't think of a good reason why I wouldn't just stick a USB-A to micro-USB lead between the hub and the Pi and be done with it (except that I couldn't find a suitable lead in the drawer... I think it maybe in the camera bag...)

Also, I like Mike's hat.
--
Ian
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by mpthompson » Tue May 29, 2012 10:55 pm
ian.macarthur123 wrote:As an aside <thread drift alert!> - has anyone tried just using a beefy powered hub to power the Pi and all the peripherals?
I'm powering my Pi from a phone charger and that's fine, but I can't think of a good reason why I wouldn't just stick a USB-A to micro-USB lead between the hub and the Pi and be done with it (except that I couldn't find a suitable lead in the drawer... I think it maybe in the camera bag...)


I'm using the following hub and giving it 5V form a good 5W switching power supply:

http://www.monoprice.com/products/produ ... 1&format=2

The Raspberry Pi is plugged into one of the front ports using the USB micro cable. The USB B port on the back is plugged into one of the USB ports on the Raspberry Pi. Works great for me.

ian.macarthur123 wrote:Also, I like Mike's hat.


Thanks... :P
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by mahjongg » Tue May 29, 2012 11:18 pm
ian.macarthur123 wrote:As an aside <thread drift alert!> - has anyone tried just using a beefy powered hub to power the Pi and all the peripherals?

wouldn't work, as polyfuses F1 and F2 only let through about 140 mA max to each USB port before "blowing"
User avatar
Forum Moderator
Forum Moderator
Posts: 5487
Joined: Sun Mar 11, 2012 12:19 am
by plugwash » Tue May 29, 2012 11:31 pm
nothing a soldering iron and some short bits of wire won't fix
Forum Moderator
Forum Moderator
Posts: 2266
Joined: Wed Dec 28, 2011 11:45 pm
by jbeale » Wed May 30, 2012 12:17 am
mahjongg wrote:
ian.macarthur123 wrote:As an aside <thread drift alert!> - has anyone tried just using a beefy powered hub to power the Pi and all the peripherals?

wouldn't work, as polyfuses F1 and F2 only let through about 140 mA max to each USB port before "blowing"


"[...] just stick a USB-A to micro-USB lead between the hub and the Pi..."
That is not back-feeding from a USB output, but normally powered from a hub using the microUSB power cable the Pi works fine as stated. In fact this configuration was suggested by one of the R-Pi foundation members. Just make sure your hub is good to provide the >500 mA current that the R-Pi needs- I think most are.
User avatar
Posts: 2039
Joined: Tue Nov 22, 2011 11:51 pm
by jerry.tk » Wed May 30, 2012 8:06 am
Just a little heads-up if you plan to use Raspbian installer for external drive installation: Raspbian Wiki: Installing to USB drive
Posts: 63
Joined: Sat Apr 07, 2012 6:26 am
Location: CZ
by ian.macarthur123 » Wed May 30, 2012 8:18 pm
mahjongg wrote:wouldn't work, as polyfuses F1 and F2 only let through about 140 mA max to each USB port before "blowing"


Ah, I think you misunderstand me... What I'm talking about is using a hub with a large current capacity as the PSU to drive the R-Pi itself, and all the peripherals. Little (i.e. no) power is drawn from the R-Pi USB sockets so I can't see the polyfuses coming into play here at all.
--
Ian
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by ian.macarthur123 » Wed May 30, 2012 8:22 pm
jerry.tk wrote:Just a little heads-up if you plan to use Raspbian installer for external drive installation: Raspbian Wiki: Installing to USB drive


Ah, thanks jerry - cool, that's useful stuff.

In other news, I got distracted yesterday by the SD card reader on my laptop going berserk and trashing my SD card when I dd'd in the img file. Took me longer than it should have to grok what had gone wrong, too... dd'ing the card again from the Mac worked OK, but I lost a lot of time to nothing there...!
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by ian.macarthur123 » Sat Jun 02, 2012 10:23 pm
Right, so I found a little time, and took the raspbian image I had and flashed it to my 8GB SD card.

(This proved to be more onerous than I'd expected: the SD card reader on my laptop has gone duff so I had to use an external one in the end, but it took me too long to figure that out... Initially I just thought the image was borked...)
Anyway, spin up gparted, resize the image to fill the 8GB and boot R-Pi, do the dist-upgrade, rpi-update...wait... and that seems OK.

When running the system (from the SD card) take a look in dmesg and it is apparent that it is spewing warnings about the SD card more or less constantly. I suspected the batch of 8GB SD cards I had were a bit dodgy, this seems to confirm that... Shut down the R-Pi and back to the laptop (and its external card reader.)

Anyway, find another SD card (old class 2 2GB card) FAT32 format this and copy on the "boot" partition files, edit cmdline.txt to point to /dev/sda2

Fire up gparted again and repartiton the Samsung hard disk with a few gigs of swap in sda1 and ext4 in sda2.

Then rsync the raspbian rootfs from p3 on the SD card to sda2 on the Samsung disk.

Plug the hard disk in, put the 2GB SD card in the Pi, power on.

And it boots, and runs. Blimey.

Doesn't find my swap partition though - rather it complains about *not* finding p2 on the SD card.

Hand edit fstab to add a swap entry for /dev/sda1 then "swapon -e -a" and I seem to be good to go.

Initial tests suggest strongly that this is A LOT faster than running from a suspect 8GB SD card was, and since I now have about 300 GB free space, I took the opportunity to apt-get a whole heap 'o packages I might need at some point, you know, just in case...

However, the one fly in the ointment is that it currently dies horribly (causing the ext4 rootfs partition to become inaccessible) if I try to invoke startx... and X does not start, of course.

Need to investigate what is happening there, but now is not the time... It is past my bedtime.

Ho hum...
--
Ian
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by Montala » Sun Jun 03, 2012 9:00 am
This sounds very interesting Ian, and I will look forward to reading the next 'instalment'... when you find some more 'free' time, and learning how you managed to overcome the 'startx' problem... which I am sure you will.

I didn't actually understand everything you wrote, which is not helped by the fact that my Raspberry Pi has not arrived yet, but I am sure that when it does things will have moved on with Raspbian anyway... especially if the current rate of progress in maintained. (Which I am sure it will be! :) )
User avatar
Posts: 638
Joined: Mon Mar 05, 2012 11:54 pm
Location: Herefordshire (U.K.)
by ian.macarthur123 » Sun Jun 03, 2012 9:41 am
[quote="Montala"]This sounds very interesting Ian, and I will look forward to reading the next 'instalment'... when you find some more 'free' time, and learning how you managed to overcome the 'startx' problem... which I am sure you will.[/quote]

Sure - but do not hold your breath waiting for me!

[quote]I didn't actually understand everything you wrote, which is not helped by the fact that my Raspberry Pi has not arrived yet, but I am sure that when it does things will have moved on with Raspbian anyway... especially if the current rate of progress in maintained. (Which I am sure it will be! :) )[/quote]

OK - which bits were too obscure? For the most part, I was just referring (perhaps too tangentially) to steps and tweaks that are described in the wiki (either at elinux or in the rasbian pages) so I guess I just assumed folks would know what I was talking about...

But if there are specific steps I've been too "hand wavy" over, then shout out and I'll try and fill in the missing details (to the extent that I actually know them!)
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by Montala » Sun Jun 03, 2012 10:41 am
Thanks for the reply Ian, and I guess that if I actually had a Pi in my hands and was able to experiment with it, I would learn things a lot quicker, but it is lines like 'Then rsync the raspbian rootfs from p3 on the SD card to sda2 on the Samsung disk.' and 'Hand edit fstab to add a swap entry for /dev/sda1 then "swapon -e -a....' which make me think "what on earth is he talking about!"

Having said that though, I am sure that once I start to learn a bit more about using the Linux command line (and have a Pi to play with) things will start to fall into place a bit more.

My current thoughts are to set up something like QEMU (maybe on my Linux netbook) in the meantime to familiarise myself a bit more with what to expect.... and to realise that I must learn to walk, before I try to run! ;)

Anyway, I realise that I am drifting a bit "off topic" here, but will certainly continue watching this, and other Raspbian related, threads.
User avatar
Posts: 638
Joined: Mon Mar 05, 2012 11:54 pm
Location: Herefordshire (U.K.)
by ian.macarthur123 » Sun Jun 03, 2012 2:02 pm
[quote="Montala"]
Thanks for the reply Ian, and I guess that if I actually had a Pi in my hands and was able to experiment with it, I would learn things a lot quicker, but it is lines like 'Then rsync the raspbian rootfs from p3 on the SD card to sda2 on the Samsung disk.' and 'Hand edit fstab to add a swap entry for /dev/sda1 then "swapon -e -a....' which make me think "what on earth is he talking about!"
[/quote]

Ah, ok... well for the sake of completeness then, I'll try and expand on that bit:

"rsync" is a kind of "copy", but a bit smarter and really meant for copying stuff from one machine to another, but you can also use it to efficiently "clone" filesystems and directory hierarchy's on the same machine, so I am using it here to make a clone of the rootfs partition on my SD card, into a partition on the USB disk.

So, what I did was, mont both the SD card and the USB drive on my (ubuntu) laptop and then:

[code]
rsync -avxS /media/mmcblk0p3/ /media/sda2/
[/code]
Where /media/mmcblk0p3 was where the laptop saw the rootfs partition of my SD card and /media/sda2/ was where it saw the partition I wanted to write to on the USB drive.
The options to rsync are pretty complex (and frequently pretty opaque!), but you say you have a linux netbook, so spend some time at the shell prompt, and try "man rsync" to figure out what the options are doing - it's god exercise!

Prior to that, I had used the gparted tool to repartition the USB drive into two chunks, small one (a few gigs) in sda1 for a linux swap partition, and the rest of the disk formatted as ext4 for the new rootfs.

Note that, with such a limited memory as the RPi, you are going to want some swap space or many programs just will not run...

Anyway, once the system booted, it did not find the big chunk of swap I'd allocated. And issuing a "swapon" command did not work - other than complaining that it could not find /dev/mmcblk0p2 (which is where the swap was on the stock raspbian SD card.)

Now, historical unix system used a file called "fstab" (file systems table) to record where the file systems were, but a lot of modern linux/unix system do not actually use that file much, though it still exists.

However, swapon does still look in fstab to see if there are any swap file systems listed that it can try to load, so editing fstab to add an entry for /dev/sda1 (which is where my "new" swap partition appears) and then issuing a:

[code]
swapon -a -e
[/code]
Did the trick. Again, try "man swapon" for details, but in brief the "-a" means "load all the swap partitions you can find" and the "-e" means "ignore any errors for partitions that do not actually exist"...

[quote]
Having said that though, I am sure that once I start to learn a bit more about using the Linux command line (and have a Pi to play with) things will start to fall into place a bit more.
[/quote]

You say you have a linux netbook? If so then just playing around with the shell in that will help, since the shell looks "the same" on all *nix systems, more or less. There's nothing particularly unique about the R-Pi in that regard.

[quote]
My current thoughts are to set up something like QEMU (maybe on my Linux netbook) in the meantime to familiarise myself a bit more with what to expect.... and to realise that I must learn to walk, before I try to run! ;)
[/quote]

As I said, just using the netbook should be enough - emulating the Pi is probably only useful if you want to try cross-compiling code for it or something. If all you want is to get to grips with the linux "experience" initially, then any linux system, whether x86, arm or PPC or whatever, will work in pretty much the same way, and the commands and applications are basically the same.
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by Montala » Sun Jun 03, 2012 11:05 pm
Thank you again Ian for a most comprehensive reply, which I do appreciate.

I have however decided (initially) to use QEMU to emulate the Raspberry Pi on my Linux netbook, and have got as far as downloading ver 1.1.0 of QEMU, configuring and compiling it to emulate ARM based architecture, but will probably start a separate thread with any further questions about the ongoing process.

Lots of fun still to come with 'rootfs', kernels etc! :)
User avatar
Posts: 638
Joined: Mon Mar 05, 2012 11:54 pm
Location: Herefordshire (U.K.)
by mpthompson » Mon Jun 04, 2012 8:25 am
Running into problems using a Western Digital My Passport 500GB USB Drive. This is a USB 3.0 drive, but it should be backwards compatible with USB 2.0 devices such as the Raspberry Pi. I can partition the drive, but if I attempt to format the partitions the system basically freezes with messages to the console every minute or so indicating the USB port is being reset. The system comes back to life once I disconnect the drive.

It would be useful if the Forums could contain specific sections on the kernel, USB devices, SD cards and other software/hardware/firmware related issues. Right now, as a Raspberry Pi user I'm not sure where the best place to go to discuss each of these issues which impact all Linux distributions on the device.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
by ian.macarthur123 » Mon Jun 04, 2012 8:51 am
mpthompson wrote:Running into problems using a Western Digital My Passport 500GB USB Drive. This is a USB 3.0 drive, but it should be backwards compatible with USB 2.0 devices such as the Raspberry Pi. I can partition the drive, but if I attempt to format the partitions the system basically freezes with messages to the console every minute or so indicating the USB port is being reset. The system comes back to life once I disconnect the drive..


Hmm, I didn't try to format the drive from the Pi - I had prepared the drive in advance from my laptop, and once running, dmesg et al look to be fairly clean - this in contrast to my SD card boots, that are full of chatter about mmc0 being duff (I suspect my SD cards are awry since no one else is reporting this effect...)

I'm using a 320 GB Samsung branded item with USB 2.0 interfaces thorugh a beefy powered hub.

The one problem I have is that I can't seem to get X to start in this configuration, but I have not had time to investigate. Maybe tomorrow (since street parties are not my thing!)

mpthompson wrote:It would be useful if the Forums could contain specific sections on the kernel, USB devices, SD cards and other software/hardware/firmware related issues. Right now, as a Raspberry Pi user I'm not sure where the best place to go to discuss each of these issues which impact all Linux distributions on the device.


Yup, that sounds like it would be handy...
--
Ian
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by ian.macarthur123 » Wed Jun 06, 2012 8:54 am
In case any one is following this...

I spent a chunk of Tuesday not having Jubilee parties but instead thrying to poke at my X11 issues.

This does not recount a fix - only describes what I have, and then asks for help...!

A recap: I have my R-Pi booting such that the SD card provides the initial boot, but the main rootfs and swap are on a USB hard drive (with actual spining platters and such!)
At the command line, this works well, and is noticeably swifter than my "pure-SD" boot.

But if I "startx" the whole thing borks horribly - I lose the USB, as far as I can tell, so my system disk, and the ethernet connection, all drop. Losing the ethernet also means that I can't SSH in to see what's happening. Losing the system disk means nothing useful gets written into the logs...

It is customary at this point to say "power supply issue" but I don't think it is; the Pi is powered directly from a phone charger rated at 1A. All the other peripherals (disk, mouse, kbd) are powered from a hub rated at 3.5 A.

I have tested with: 2 different kbd's, two different mice, 2 different hard disks (one powered by the hub, the other self-powered.)

All fail in an identical fashion.

But...

The failure is peculiar to the Raspbian-on-USB-disk configuration.

The same h/w connected (including leaving the hard disk plugged in) but booting and running from an SD card works fine - I even installed midori and browsed the interwebs with this set up in testing.

The same h/w connected but programming the hard disk with the stock armel distro from the foundation works fine and allows X11 to start and run fine. Indeed, this config (runing armel but from a hard disk) seems pretty snappy too, so for anyone watching, I'd suggest moving away from SD card based installs, is in, general a win anyway!

But - the case I actually want, the armhf-Raspbian on USB hard disk, dies horribly, and I can't figure out why.

I suspect, on the basis of inadequate diagnostics, that the USB stack is getting trashed, but I can't get inside of why...

Suggestions, answers, fixes... all welcome!

Cheers,
--
Ian
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by paaland » Wed Jun 06, 2012 7:29 pm
Just wanted to say that I'm now running raspbian with root and swap off a 2.5" USB 2.0 HDD.
The HDD enclosure has a separate power supply. I'm running "headless" so I've not tried startx, but I'm running xming on my PC and tunnel X via SSH. I can start several X programs (iceweasel and two xterms without any problems).
Dragon 32>BBC Model B>PC>PC>PC>PC>Raspberry PI (circle complete)
User avatar
Posts: 68
Joined: Mon Mar 12, 2012 2:30 pm
Location: Norway
by ian.macarthur123 » Wed Jun 06, 2012 8:16 pm
Yes, perhaps I should have been more specific - headless it is fine, I can "ssh -X" into it and all is well, X11 gui apps run fine - well, until I try and do something swoopy with OpenGL, at which point I get warnings about unsupported capabilities for the X-server, which is fair enough.

Given the Pi's fairly decent GPU capability, I expect that GL would be fine if I run it *native* on the R-Pi itself, of course.

But, I can't actually get X to start, *in the specific Raspbian+external drive* configuration.

Other configurations are fine though, so it must be something specific to the way we are doing things in Raspbian that cause the issue (or, as an alternative, the Raspbian+external drive configuration is exposing some latent issue that is not triggered in the other set-ups...)
In any case, something is awry... Be interesting to figure out just what!
Posts: 32
Joined: Thu Nov 17, 2011 10:58 pm
Location: Edinburgh
by marsman2020 » Wed Jun 06, 2012 9:51 pm
If you install the gpm console mouse service and start it, does that also cause your ethernet/usb hdd to drop out the same as when you are sttempting to start x?

Reference my github issue at https://github.com/raspberrypi/linux/issues/29 , powered hub + keyboard + mouse + (gpm or startx) -> networking drops out. Using tightvnc to log into x remotely also works fine.

(note installing gpm will set it up to start on boot by default, which might break this configuration on every boot, so you'll want to "update-rc.d -f gpm remove" after installing it and do all testing by starting/stopping the service manually)
Posts: 70
Joined: Sun May 20, 2012 11:26 pm
by mpthompson » Wed Jun 06, 2012 11:18 pm
ian.macarthur123 wrote:But, I can't actually get X to start, *in the specific Raspbian+external drive* configuration.


I would suspect this issue may be related to the fact what when I start up the X server in Raspbian, networking goes south -- described as dropping network packets by others who've had similar experiences. Since the ethernet is on the USB bus, I wouldn't be surprised if other high speed peripherals such as USB drives go south as well.

Sorry I don't have any suggestions for fixing this now. Something about the X server on Raspbian and the current kernel seems to really give the USB bus grief. We're working on creating a Raspbian kernel now so hopefully that will provide more of a base to investigate what may be going wrong.
User avatar
Forum Moderator
Forum Moderator
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA