How to protect the filesystem without a save shutdown button?

[slacer]

What happens if a student simply switch of the device by pulling the power plug?

I guess the file system will be corrupt at the next start, or after the next student wants to leave this classroom in order to meet his friends between lessons?

On a pc running linux you can press the power button once (short) and it will initiate the required shutdown.

The raspberry pi has no such button and there is no indicator the shutdown/halt is complete if you have no monitor connected.

What can be done to make sure the next lesson in this programming class can start without the need to recover 2-3 filesystems?

Maybe it was too expensive to add a way to shutdown a raspberry without the need to logon as root first, but maintaing filesystems during lessons might upset some teachers and students.

[abishur]

Well there actually are a couple LED indicators that I imagine might give you a clue of when it's safe to turn off, and if not you could always rig one of the GPIO to do the trick   Beyond that I bet if anyone was worried about such an event happening you could always design a breakout board that would feed the r-pi power as well as

a) Send kill signal to r-pi on button press

b) Receive back from r-pi signal that shut-down is complete (perhaps via a command that sets the GPIO high right before the shut-down process is finished.  The board would then wait 5-10 seconds and stop sending power to the board).  Alternatively you could always just have it send the kill signal then wait a minute and stop sending power

c) Send power to the r-pi next time the power button is pushed, power comes on so does r-pi

Also unless the filesystem is being written to sudden power loss shouldn't hurt anything.  And even then it would only damage the file being actively written to.  Generally speaking this is not an issue for critical files (such as the filesystem) or else every time a sever lost power the system would be corrupt and computers would suck It's certainly a possibility, but I wouldn't rank it as being a serious threat or point of concern.

Maybe the best solution is teaching the kids to be intelligent users and not just unplug computers all willy nilly

[thomas41546]

What you could do to prevent this from being a problem at-least with software, is to un-mount any connected disks before disconnecting the device from power.

If a student simply pulls the plug to turn off the device, then the file system "may" become corrupted, however ext3 partitions and most other modern file systems have mechanisms to prevent data corruption and typically recover from any potiential hard-disk corruption.

On most linux systems you can run the command:

"shutdown -h now" -- this will shut down the linux system gracefully, keeping the disk data intact and corrupt free.

[johnbeetem]

GUIs running on top of GNU/Linux generally have a shutdown menu item somewhere.

[Photon Peddler]

A common way with embedded Debian Linux-based devices is to run a mostly read-only filesystem with RAM disks mounted over the directories that have to be read-write. Unfortunately the project pages that I've used as inspiration in the past, Debian Router Project, are no longer available. Anyway, a 256 MB system with an X desktop is starved for RAM, so the idea of using overlay RAM disks might not be the best idea.

Debian also has an official howto for mounting most of the system read-only, at http://wiki.debian.org/ReadonlyRoot. Many directories are still left read-write: /etc, /home, /srv, /tmp and /var. Each of these remaining read-write directories can be handled differently:


/etc: Can be made read-only with certain precautions. See the Debian howto.
/home: Could be reconstructed on boot from a pre-made archive, if the students carry an USB stick for their personal files instead of saving them under /home.
/srv: This directory is empty most of the time (unless running servers).
/tmp: All files here can be deleted on boot.
/var: This is tricky, as it contains the critical directories /var/lib and /var/spool. Other directories in it are not as critical. /var/lib and /var/spool should be kept on a filesystem that survives a hard crash, such as ext3/ext4. In addition, a backup script such as rsnapshot could be run periodically to save any changes on another filesystem.


Also see the Emdebian project for information on running Debian on resource constrained systems: http://www.emdebian.org/

[patrickhwood]

Yeah, most of the journaling file systems (like ext3) are very robust.  I have yet to see one trashed by a reset, unplug of a USB drive w/o unmounting, etc., whereas I've seen a few FAT32 file systems destroyed this way.  In fact, I have yet to see an ext3 file system ever need anything recovered via e2fsck.

[kme]

Yes, the entire thing is a non-issue. The VFAT partition with the boot loader is essentially a RO file system. The core Linux system is on ext3/etx4 partitions which are journaled and very, very hard to break.

[Bakul Shah]

See this paper as to why removing power while writing to a disk is bad.

http://cseweb.ucsd.edu/users/s.....werCut.pdf

An excerpt:


If power fails during a write to a hard drive, the data being written may be irretrievable, but the other data on the disk remains intact. However, SSDs use complex flash translation layers (FTLs) to manage the mapping between logical block addresses and physical flash memory locations. FTLs must store metadata about this mapping in the flash memory itself, and, as a result, corruption of the storage array can potentially render the entire drive inoperable: Not only will the in-progress write not succeed, but all the data on the drive may become inaccessible.


The paper reports worrisome results such as data corruption in previously successfully completed operations! Now you may warn a user to always remove power safely but how do you protect against power failures, or worse, accidental removal of power (someone trips over the power cord)?

[kme]

That paper is poor work and highly speculative. First they don't state which file system they use or if it's journaled at all. Second they are playing with power cuts a few hundred microseconds after writing. This have no real world relevance.

[Bakul Shah]

The filesystem used is not relevant as they are talking about what happens at the FTL layer (it will be common to whatever FS you put on it). If power fails or if the miniusb plug is removed accidentally you have *no* control over when the power is cut out relative to any writes (or background erases).

But since you (kme) think this paper is not relevant, and since we don't seem to have any real data regarding what happens in a real world situation, try this experiment with a usb disk you are prepared to lose: put your favorite FS on it. Do lots and lots of back to back random access writes to it and *while* writing, remove it from your machine. Plug it back in, check that *everything* is fine (to make this checking possible write known but different patterns). And then repeat the process. Do this a few times. A similar experiment can be tried with the Raspi. It would be interesting to see if what the flash manufacturers warn against is really true & some data will be lost irretrievably or if you are right and everything is hunky dory! This will be a service to us all!

[kme]

Bakul said:


The filesystem used is not relevant as they are talking about what happens at the FTL layer (it will be common to whatever FS you put on it). If power fails or if the miniusb plug is removed accidentally you have *no* control over when the power is cut out relative to any writes (or background erases).


Would you be kind enough to enlight me why the FS journal wouldn't cover this? I assume you agree no one is using a non-journaling FS these days.

[bitplane]

In a nice clean hypothetical world where IO operations are atomic, journaling filesystems would offer real protection from a user pulling the plug.

In the real world where software works a bunch of levels of abstraction away from the filesystem, i.e. state is written across multiple writes and files, you'll likely screw something up if it's writing to disk when you turn it off.

[PianoMan2112]

Has anyone made/can anyone make a small USB plug that emulates a PC keyboard, with a momentary button that sends Ctrl-Alt-Del?  (It'll probably be easier to write a daemon to initiate shutdown if 2 GPIO wires are shorted, but the USB keyboard would have the advantage of working on most Linux distros.)

[Kaz]

In theory, since the part uses such limited power (as it's aimed at mobile phones), adding a sufficiently large capacitor or button cell might help mitigate such issues... a +5v sense GPIO could be there to say "oh, the +5v is gone... I'd better hurry and gracefully shut down."  Maybe in the future...

[Bakul Shah]

kme said:


Would you be kind enough to enlight me why the FS journal wouldn"t cover this? I assume you agree no one is using a non-journaling FS these days.


When there is embedded flash memory that s/w can directly access, without going through any FTL (flash translation layer), it has more control over how data is laid out and a journaling FS can do better. But we are talking about removable flash drive, that has a controller that implements the FTL. So two issues:

1. When your journaling FS writes to some block i, the FTL maps it to a physical block j (and this mapping can change over time, to even-out "wear" etc.). Now this mapping itself must also be stored on the same flash disk. If power fails right when the mapping is being updated, you can potentially lose the whole flash drive — and from above the FTL you have no way to fix whatever was corrupted. I don't think disk vendors provide fsck-FTL!

2. A journaling FS logs any changed data before it is written to the right places in the FS — this avoids the need to fsck the whole filesystem after a crash or power fail; but this only makes sense if one assumes the existing FS structure is consistent. The other failure described in the paper was when a *previously* successful operation now shows corruption! In effect the old FS structure can become corrupted on its own. This violates the most basic assumption of a journaling FS! You have to do a full fsck and you can potentially lose a lot of data depending on what part got corrupted.

[obarthelemy]

shutdown -h now ?

[jamesh]

I'm not sure why everyone is so het up about a problem that has existed with HD's and storage since they were invented! Power off at the wrong time can cause problems. Clever software a good HW design can mitigate it.

Sometimes sh*te happens.

[Paul7k]

i dont think this is going to be a of a problem for most people, its kind of the same for usb pens/hard drives at the moment, you plug them in when needed and when ur done u just unplug, very rarely will this cause an error,

errors are normally from while transfering files from/to mid way and is turned off which normally gets fixed after a quick scan.

ive never had a drive die on me doing this and i have been doing this on drives for years,

[Blub]

If you're evil you add the 'sync' mount-option, if you're nice you use a network filesystem in which case you might get incomplete files, but at least no corrupted filesystem

[bradburts]

My experience of embedded systems is that: 1) you can destroy the disk if power is lost during the write and 2) a lot of development effort can go into protecting that disk.

As stated FTL and wear leveling mean that you don't know where the data is.

Its important to minimise writes and/or control when writes are made.

Microsoft use the EWF mechanism to ensure reliable embedded device operation by blocking writes to disk (transparently holding the writes in RAM with an option to commit).

I am not aware of an equivelant Linux filter.

Your risk depends on the frequency of writes, the duration of a write and how often you pull the plug.

Its worth noting that Linux often has quite a lot going on in the log department, more so than Microsoft.

You can do a lot to minimise your risk by following the posts in this forum which show you how to configure Linux for use with a flash card, e.g. directing the most frequent logs to a RAM disk which also saves the flash card from wear. Downside, you loose your logs.

JamesH is right, this issue is not unique to the PI and s@#t does happen. When the s@#t does happen though the SD card could well be fried.

Short term the answer is to minimise the number of writes (get logs onto a RAM disk) and drill the kids to shutdown.  Buy a stack of disks and setup a good cloning system.

I would imagine that the battery powered community will come up with something that does what you want, performs an orderly shutdown on lose of power.

It won't makes economic sense to buy/make a small UPS though even with naughty school kids. Flash corruption is only an issue where the end user cannot easily replace the card, MIL systems etc. Thats assuming that the most frequent log writes have been taken out.

PS

Don't store your home work on the disk that is receiving all the log writes, unless you can get away with "the PI ate it" excuse.

EDIT:

I don't have data at hand but I once went looking for this problem, setup an automated jig with embedded computer logging with random power cycling. The fault does happen but its not very likely.

The disks were near empty so wear was not a factor.

[TrevorF]

Solution (mitigation?) could/should be:

SD card on device is for the operating system only. Student stores *their* content on the network.

SO, if any single RasPi is powered off and it does corrupt the OS on the SD card, replace the SD card with another one, format the corrupt one and place the vanilla OS that class is using on it.

In short: DO NOT store the content on the removable medium in the first place.

TrevorF

[rurwin]

What about on a model A? No USB storage, no network.

[jamesh]

Well, I haven't trashed my SD card on my alpha board, and I was pressing the power button all the time on that at fairly random intervals. I don;t think I have ever issued the 'shutdown' command. I think the problem is being massively overstated.

Backup your work just on the off chance would be the correct action I think.

[ukscone]

I was thinking something like a minimal initramfs concatted (compiled into the kernel) to ensure the ability to boot into a shell, pivot_root to the full rootfs for actual usage in a seperate partition and then store the users home dir in a seperate partition. in other words how linux generally boots -- the only data that is at risk then is the users home dir and if we teach them good data practices then it shouldn't be much of a problem. there will be the odd time where they'll have unrecoverable corruption but that's what backups and frequent saving are for

/me going to back up his hd before he forgets and pulls the power by accident

[selvmarcus]

So, this board needs a shutdown button, that's a simple, cheap thing to attach to one of the GPIO pins. Should need a bit of pressure to engage. Will cause interrupt and then "shutdown -h now" or whatever you tell it to do. After you get some feedback (LED?), you can safely pull the plug. (Could even restart the system if you press again, getting you out of some system locks in a safe way).

This should be fixed, looks like a flaw in the design.

Marcus