Minimising SD card writes


9 posts
by smithnerd » Thu Jul 19, 2012 2:19 pm
I'm trying to minimise unnecessary writes to the SD card.

I have nulled .bash_history, which keeps the current shell history in memory, but never writes it anywhere, thus:

Code: Select all
pi$ ln -sf /dev/null ~/.bash_history


I don't really want to use NFS, so I installed 'sshfs' and added 'pi' to the 'fuser' group. This gives me a working subdirectory, mounted from my Linux laptop, from which to write code:

Code: Select all
pi$ mkdir ~/laptop_mnt

pi$ sshfs piuser@laptop: ./laptop_mnt


Does anyone have any other ideas?
Posts: 8
Joined: Thu Jun 14, 2012 1:18 pm
by lavers » Thu Jul 19, 2012 2:31 pm
As an alternative to linking .bash_history to /dev/null, you can just set the HISTFILE shell variable (in your .bashrc file for example) to blank, which will cause Bash to not write a history file.

I haven't got my Pi yet, so I don't know if distributions like Raspbian already do this, but you might want to mount your SD card with the 'noatime' option to prevent it from writing access times on files every time you access them. Additionally, maybe turn down or off any log files you can find, or redirect them somewhere else.
Posts: 17
Joined: Mon Jul 09, 2012 3:45 pm
Location: Canada
by RRvW » Thu Jul 19, 2012 3:49 pm
have you also looked at the /tmp and /var/lock directorys? Most linux distros mount them as tmpfs.
Posts: 1
Joined: Thu Jul 19, 2012 3:38 pm
by kadamski » Thu Jul 19, 2012 6:45 pm
Isn't bash default behavior to only write history to HISTFILE when exiting from shell? And since it's usually not really done often, does this "optimization" make much sense?
Posts: 187
Joined: Fri Jun 08, 2012 10:56 pm
by Atilla » Thu Jul 19, 2012 9:49 pm
Indeed, bash does save only on exit, unless you force it to, which you can easily confirm by murdering a terminal and checking your history afterwards.
Posts: 4
Joined: Thu Jul 19, 2012 9:19 pm
by AndrewS » Fri Jul 20, 2012 2:17 pm
So obvious it hopefully doesn't need suggesting, but make sure you've disabled swap.

If you want to really minimise the number of writes to the SD card, you could transfer your entire rootfs over to a USB (flash- or hard-) drive. Only the /boot partition needs to stay on SD.

And have a look at this thread www.raspberrypi.org/phpBB3/viewtopic.php?f=29&t=10775
User avatar
Posts: 3608
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
by gdt » Sat Jul 21, 2012 1:01 pm
I'd rather think that /var/log/* is the main source of writes, especially since the as-shipped /etc/rsyslog.conf has some files written a line at a line.

You could simply configure rsyslog to discard all messages.

What would be better requires some programming. Configure rsyslog to log to a UNIX domain socket (see the omucsock output module), write a daemon to listen on that socket and keep the last N KB of messages, have a client program which will retrieve and print the recent messages cached by the daemon. And yes, there a strong parallel here with the kernel's "dmesg" utility.
Posts: 55
Joined: Thu Jul 19, 2012 10:19 am
by gdt » Sat Jul 21, 2012 3:02 pm
(Note to moderators: it would be nice if the configuration below became the default on the RPi's images, as it will make people's SD cards last longer. If you could bring it to the attention of the person responsible for such things, then that would be appreciated).

I've just had a play, and you can make rsyslog send to /dev/kmsg (which is readable by dmesg).

Edit /etc/rsyslog.conf
Remove "$ModLoad imklog" since we don't want to log kernel messages back to the kernel, and thus loop infinitely.
Similarly, alter the line starting "kern.*" to be "kern.* ~" (ie, discard kernel messages, don't fret as they are still in the kernel's message ring buffer).
Now for every line that writes to a file, replace the filename with "|/dev/kmsg". For example, the line
daemon.* -/var/log/daemon.log
becomes
daemon.* |/dev/kmsg

Restart rsyslog with
/etc/init.d/rsyslog restart

Now there should be no syslog messages written to /var/log.

To see the most recent syslogged messages enter the command
dmesg

When cleaning up, do not delete files in /var/syslog. Rather say "echo -n > /var/log/syslog". This removes the contents of the file, but keeps the file's ownership and permissions, which can be important if it is something other than rsyslog which was writing to the file.

Note that this isn't the best way to write to the kernel's log buffer (the kernel isn't told the message's attributes, such as "facility" or "priority" and it incorrectly classifies the message as kernel space rather than user space). To do it properly a rsyslog output module should be written. Also this isn't the most efficient approach -- having the kernel maintain the log is necessarily less efficient than doing it from user space. But it's not like the RPi writes enough messages that we overly care about the message's metadata or its efficiency. We rather take the reduced writes to the SD card, both so that the RPi runs more quickly and so the SD card lasts longer.
Posts: 55
Joined: Thu Jul 19, 2012 10:19 am
by obarthelemy » Sat Jul 21, 2012 9:33 pm
I had started a whole topic about that a while back: viewtopic.php?f=41&t=850&hilit=optimizing+for+flash

Haven't worked on it since then though, sorry.
Posts: 1399
Joined: Tue Aug 09, 2011 10:53 pm