adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Why running from USB better than from sd card?

Sun Dec 22, 2013 7:53 am

Hi all.

I've reached a stage when the routine of keeping the Pi alive (sd card corruptions) equals or exceeds the time spent playing with it.

I'm tempted to try to move to USB. I want to understand first if it makes sense.
  • Aren't USB sticks exactly like SD cards with some extra interface?
    adfasdf

Thanks!

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1489
Joined: Sat Sep 10, 2011 11:43 am

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 8:00 am

Have you tried using the most recent Raspbian fully updated / upgraded?

sudo apt-get update
sudo apt-get upgrade
sudo rpi-update

Will get you the latest changes, which includes a fix for SD card corruption (make sure you're not overclocking)

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 8:06 am

Newbie mistake, pressed Submit before finishing the post... I wanted just to preview. Preview and Submit buttons should be spaced apart or Submit should require a confirmation.

Going back to my question: why running from USB would solve sd card corruption?

Would like to understand:
- Aren't USB sticks just like SD cards but with a different interface?
- What makes running from USB more resilient then? Or is it just a myth?
- Where is the best recipe for moving to USB? I've googled and seen a few. Seen one well written on this site: http://www.raspberrypi.org/phpBB3/viewt ... 29&t=44177
- Would any USB stick do? Or is there a list with good and bad as for sd cards? Are we then going back to square one ;)?

Would appreciate a clarification! Thanks!

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 8:12 am

@gsh Yes, did the update/upgrade and all, went already through the usual battles... I have backups, I've been burnt. Have been on 2 SD cards. I have a list of suspects (the power supply) but I'm intrigued seeing quite a number of posts recommending running from USB and my question is: Why?

Once I understand the rationale I would go for it in no time.

Adrian

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1489
Joined: Sat Sep 10, 2011 11:43 am

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 8:20 am

Like I say, we've identified a bug which causes sd corruption... That's been fixed (in the last month or so), if you don't have that fix then that's why you're having trouble

As to why USB is more reliable, it's because there's a bug in the SD card software that causes corruption (didn't I say that already...)

What is your kernel version number (uname -a will tell you something like #545) You need at least #590 to have the fix (plus make sure you don't go playing with overclocking settings in your config.txt)
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 9:17 am

Thank you again. I'm on #610, so it should be something else. No overclocking, nothing fancy. I religiously do shutdown -h. I update the software often (too often?). Should this be the first report that there may be yet another bug?

My opinion is that there should be a big push to increase the resiliency. RPi is great but too fragile. Too many posts about corruption. Fortunately after a fsck one can revive the card and boot again (I was surprised) or at least mount it on a different computer and recover files.

I asked the question because I'm interested in the technical details - why so fragile when it's on ext4 that's journaled, why benign activity in user space (writing files) can corrupt the OS, etc. And, of course, the original question: is running from USB really better or an urban myth?

Adrian

Linux adipi1 3.10.24+ #610 PREEMPT Thu Dec 12 13:12:09 GMT 2013

gkreidl
Posts: 6219
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 9:38 am

Basically there is no no difference (except for driver bugs): both have rather limited write cycles and linux writes to disk all the time.

If you really want to have something more reliable, move the root file system to an USB HD.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

Joe Schmoe
Posts: 4277
Joined: Sun Jan 15, 2012 1:11 pm

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 10:05 am

sudo apt-get update
sudo apt-get upgrade
sudo rpi-update
I don't think it is good advice to recommend the 3rd item in the list above. The first two are sufficient.

Note that there have been horror stories posted in the past about people who used rpi-update and got a "too bleeding edge" version of the kernel, much to their chagrin. It's better to just to "upgrade" and get the latest stable/blessed version.

P.S. I just did (only) the first two on my Pi and am now running #614.
And some folks need to stop being fanboys and see the forest behind the trees.

(One of the best lines I've seen on this board lately)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25370
Joined: Sat Jul 30, 2011 7:41 pm

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 10:45 am

adrianidar wrote:Thank you again. I'm on #610, so it should be something else. No overclocking, nothing fancy. I religiously do shutdown -h. I update the software often (too often?). Should this be the first report that there may be yet another bug?

My opinion is that there should be a big push to increase the resiliency. RPi is great but too fragile. Too many posts about corruption. Fortunately after a fsck one can revive the card and boot again (I was surprised) or at least mount it on a different computer and recover files.

I asked the question because I'm interested in the technical details - why so fragile when it's on ext4 that's journaled, why benign activity in user space (writing files) can corrupt the OS, etc. And, of course, the original question: is running from USB really better or an urban myth?

Adrian

Linux adipi1 3.10.24+ #610 PREEMPT Thu Dec 12 13:12:09 GMT 2013
Well, there is work constantly being done to improve resiliency - it's the main development effort for the Raspi people themselves. And that resulted in the latest SD card fix (and previously many USB fixes), which should fix all *known* issues IIRC. It certainly has for most people. Remember that there are quite a few posts here about corruption, but that number is tiny compared with the number of Pi's sold (2M), so its actually quite rare. I've never seen an SD corruption for example.

How old is your SD card? I wonder if it is simply worn out.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 1:30 pm

Google flash memory and wear levelling for excruciating details on the "inner workings" of flash and you will soon realize why writing to file "A" can result in the OS, journal and a host of other files being trashed. Even files in a different, un-mounted partition.... And no, a USB stick is no less prone to these problems.
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 2:24 pm

Thank you all for the ideas.

@MaxK1 - This is what I thought as well, running from USB would not solve the issue, although in some places this is sold as a cure for corruption. It may have worked better for some. I would keep the backups warm ;). I have also no doubt it's tricky running from flash. "Even files in a different, un-mounted partition..." Well, this is scary!

@jamesh - The card is a 8GB Sandisk class 6, about 6 months old. Worked for a number of months in my first Pi then corruption (after abrupt power down). After fsck it booted again (amazing, huh!) and ran for another month. Then the Pi itself died. Maybe because I plugged the vaccum cleaner in the same socket? Didn't die that day, a few days after.
Got another Pi, put the September Raspbian, all updates, some somewhat heavy activities (compiled node.js from source, installed boost-dev, etc). And then some other heavy disk activity: time-lapse photography, more than 2000 pictures! After a week the card got corrupted. That may have been too much stress if Max is right!?
And 2M Pis ok, just a few bad apples? I think a poll would be required - from the ones that actively use the Pi.
But James, if you are a lucky one and no corruption at all, could you please let me know what power supply and sd card you use (Amazon links)?

So the culprits may be time-lapse photography, node.js, boost and a vacuum cleaner ;).

@gkreidl - I'd move to an USB but it negates some of the reasons to use the Pi - use powered USB hub. But this is what I may end up with eventually. Fine at home, but when I bring the gear to work for time-lapse photography I'd like a minimal setup.

Last thing: the Pi would not boot from the card after restoring with dd from 2 backup! Any ideas here other than throw the card and get a new one? Would dd insist on writing over bad sectors or are the bad ones are marked up and dd transparently uses the new good ones? But you see the problem: getting into too much details about sd cards - ideally one would do real work/play on the Pi instead of learning about sd cards, however interesting those may be.

Eager to follow the activity on making the Pi more bullet proof. Hoping to hear some good news eventually. Merry Christmas!

Thanks,

Adrian

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 3:16 pm

I have HDD's on 4 of my Pi's - I boot from the SD cards and that is it until I perform an rpi-update (at least once a week) If I want to be absolutely sure I don't have corruption, I umount /boot, physically remove the SD card and insert it in an external USB reader/writer, remount the sd card (mount /dev/sd? /boot) and perform the rpi-update. Is that really needed? Probably not. But when I don't have the time to restore a card, I'll take the time to do it that way. So far, so good. If nothing else, I now do regular backups as well!
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 5:50 pm

MaxK1 wrote:Google flash memory and wear levelling for excruciating details on the "inner workings" of flash and you will soon realize why writing to file "A" can result in the OS, journal and a host of other files being trashed. Even files in a different, un-mounted partition.... And no, a USB stick is no less prone to these problems.
Thinking now, it may be complicated, but there should be a solution. That's because there are millions of smartphones that work out of flash and don't get corrupted that often. Right, the battery is always connected, but otherwise there's a lot of writing when people update their mp3s or take photos or else. And from time to time people take out the battery without power off. Haven't heard of bricked phones because of that.

So, why cannot simply do the same thing as Android? It's Linux, runs on flash, no corruption... Opinions?

User avatar
rpdom
Posts: 16313
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 6:12 pm

adrianidar wrote:Thinking now, it may be complicated, but there should be a solution. That's because there are millions of smartphones that work out of flash and don't get corrupted that often. Right, the battery is always connected, but otherwise there's a lot of writing when people update their mp3s or take photos or else. And from time to time people take out the battery without power off. Haven't heard of bricked phones because of that.

So, why cannot simply do the same thing as Android? It's Linux, runs on flash, no corruption... Opinions?
Because most of the writes are infrequent, and to single largeish files. Also the files are written in one pass, not updated in small chunks. Flash is designed for this sort of thing.

Linux will be writing to many files of various sizes at the same time. Log files are a particular problem - they can get updated a few bytes at a time and each update might be written to the card (some will be caught by the write cache) updating the same block over and over again. The flash wear levelling will trigger and keep moving that block around but the card will still get a lot of writes. (Much of the logging is disabled on the Pi to save wear).

I assume Android either doesn't have much in the way of log files, or they are held in RAM, or caching writes to a greater extent.

Also on a smartphone or similar device there is battery monitoring hardware that will flush the buffers and do a clean shutdown before power gets too low.

danjperron
Posts: 3452
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Why running from USB better than from sd card?

Sun Dec 22, 2013 6:40 pm

Yes log files are problematic,

I did set the log file to a ramdisk on one of my first project. It is running fine since then. (Nov 2012).

Maybe try to kill the log file or do like I did , put them on ramdisk and see if you still have a problem with corrupted SD card.

I also have a problem with some sd card and specifically one from element14 (4G). I put it away and change it to a kingston card and the problem went away. After a while I just put it back and in less the a day the problem returned. Maybe check other SD card brand.

So if you are interested to my little method to set the /var/log into ramdisk check this post that I just did.

http://www.raspberrypi.org/forum/viewto ... 54#p473054

Using this method will limit the amount of log you could store and they will vanish on power off.You will need to set the log system with a smaller print.

B.T.W. Never power off an Raspberry Pi without a software shutdown first.

Code: Select all

sudo shutdown -h now
Daniel

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: Why running from USB better than from sd card?

Mon Dec 23, 2013 1:04 pm

danjperron wrote: So if you are interested to my little method to set the /var/log into ramdisk check this post that I just did.

http://www.raspberrypi.org/forum/viewto ... 54#p473054

Daniel
Yes, definitely! I wasn't a fan of moving swap to ramdisk ;) but moving logs there make sense.

I'm still puzzled why Raspberry so affected: not only Android runs out of flash, but Windows on Nokia Lumia as well. And heard of future Debian (or some distro) and even a Firefox OS for smartphones. So this should have been solved!
Should the Pi have an option to limit drastically or an on-off switch for the logging in raspi-config? Usually one doesn't watch them that much, only when developing or the Pi gets nuts. They could be enabled in that situation. Can of course, as you did, put them in ramdisk or /dev/null, we can do it ourselves.

Would like to hear the explanation why Pi so fragile and why Android ok (and then do the same on the Pi) from some of the heavyweight Pi developers, but I doubt they hang around in these places.

I'm impressed though by the level of support, nice that many people jumped in with suggestions.

Thanks Daniel!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25370
Joined: Sat Jul 30, 2011 7:41 pm

Re: Why running from USB better than from sd card?

Mon Dec 23, 2013 1:13 pm

The Raspi isnt 'fragile' as you put it. It's always been decent, but is even better with the latest firmware.

But you have to consider that generic SD cards running a general purpose file system have a limited life, whereas the custom flash and flash specific filesystems used in things like an Android phone have a longer life. The software on Android devices is also thoroughly customised to ensure stuff lasts as long as possible, by using ramdisks etc.

An android device is very different from a general purpose compute device like a Raspi.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

danjperron
Posts: 3452
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Why running from USB better than from sd card?

Mon Dec 23, 2013 1:32 pm

For me, it wasn't the SD card by itself the problem but really the power off.

I think that 95% of the people crash the SD card because they just pull the plug. They don't shut it down properly.

Mind you that I could be wrong.

The issue was that the RPi is outside controlling a heater on a pond and like last week the temperature outside was just below -30 Celsius. And now it is partially covert with snow and I don't want to open the box to rebuild the SD card.

Image

Now add 50 cm of snow on the picture and it is just the beginning.

Since I did that , I never had to rebuild the card.

B.T.W. I was running last winter with the original Raspian version and no problem. I just did an upgrade this summer when I switch from mjpg_streamer to motion to figure out which animal digs a hole under my shed.

Daniel

AJB2K3
Posts: 215
Joined: Sun Aug 11, 2013 3:25 pm

Re: Why running from USB better than from sd card?

Mon Dec 23, 2013 8:16 pm

I use a sandisk class 10 16GB and shutdown the raspy nearly 20 times in 5 hrs using sudo halt.
Never had a problem!
The only difference between halt and shutdown is that shutdown issues the acpi command to disconnect the power.
As this has to be done manually (from what I understand) both halt and shutdown are to be considered secure methods.

Mind you, I have had 3 sandisk sdhc micros and 1 MSPD corrupted by a camera but restored through fsck!
Back in the old days, we were smarter then the technology.
Nowadays technology makes us look like villages fools in front of the children!

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12653
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: Why running from USB better than from sd card?

Mon Dec 23, 2013 8:45 pm

IMHO the biggest problem is that the SD-card might be busy shuffling file blocks around to optimize wear leveling, and that may continue many seconds after the OS has "shut down". It explains the random nature of the corruptions.

Simply waiting longer before turning off the power might be all that is needed.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1489
Joined: Sat Sep 10, 2011 11:43 am

Re: Why running from USB better than from sd card?

Mon Dec 23, 2013 8:48 pm

No the SD card doesn't shuffle things around when it is not busy... Plus any shuffling it does is controlled atomically and therefore doesn't actually matter when you power it off as long as it is not signalling busy at the end of a write

Any corruption is perfectly happy to be blamed on the bug in the hardware... Occam's razor and all that

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

adrianidar
Posts: 20
Joined: Sun Dec 22, 2013 7:33 am
Location: London, UK

Re: Why running from USB better than from sd card?

Tue Dec 24, 2013 9:18 am

@danjperron - Thanks, followed your advice and moved the logs to RAM. I have also changed time-lapse to save the pictures to an USB stick.

Fingers crossed! A la prochaine! Merry Christmas!

Adrian

Said "A la prochaine" in case I would need further advice in case of failure :lol:

danjperron
Posts: 3452
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Why running from USB better than from sd card?

Tue Dec 24, 2013 1:41 pm

Don't forget to change the /etc/logrotate.conf to reduce the size of it

I did change the weekly to daily and the rotate from 4 to 7.

This mean that I only have 1 week of log and it rotate every day.

Joyeux Noël

Daniel

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: Why running from USB better than from sd card?

Tue Dec 24, 2013 1:52 pm

Plus any shuffling it does is controlled atomically
If that's the case, than why is Swissbit selling special sd cards for embedded systems with the guarantee that either the old data or the new data will be read back from the card after an interrupted write. They just are unpayable expensive. Isn't the truth that nobody (besides those who make the cards) knows the details how sd cards handle wear levelling? The methods might differ between different brands of sd cards as well. Some flash filing systems for flash memory handle the wear levelling themselfs. This is not possible with sd cards as you can't disable their buildin wear levelling. (Maybe you can but the method for it is not documented and not part of the sdhc communication protocol)
Is the Pi more fragile than an Android cellphone? Yes it is.
You need to properly shut it down before you remove the power....
The Android device has a battery and will only remove the power when the proper shutdown sequence is finished. If your phone is frozen and you need to remove it's battery, it likely isn't writing to it's flash memory at that moment.
Moving the log files to a ram tempfs might help a lot.
Disabling swap might help as well if you know what is running on the device and if that can deal with the available memory.
The next step is a complete read only root fs. Not all application like this, so some files need to be copied to tempfs as well during bootup. You can even further protect the sd card against writes with a software command making it really read only.
Obvious, this approach can't be used on every system. If you are experimenting a lot, you might need the log files and persistent changes to other files. In such occasions, you could consider a NFS root fs on a linux server as well.

danjperron
Posts: 3452
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Why running from USB better than from sd card?

Tue Dec 24, 2013 2:08 pm

Well for me , I never consider the SD card problem.

I did that trick long time ago on embedded linux system using Hard disk and CF card. The problem was not the storing device but the processor by itself when power is pulled. The processor was writing stuff into the peripheral and when power start to failed , it was writing into bad places. Problem is even worst with real hard drive.

For me has I said it before , The only way to reset the unit since it is outside in a plastic box cover with layer of isolation, I had to pull the plug to reset it. So I set the SD card to be mostly in read mode all the times.


Daniel

Return to “Troubleshooting”