unclejed613
Posts: 55
Joined: Sun Jan 15, 2012 10:49 pm

sd card life running a pi 24/7

Fri Mar 15, 2013 4:29 am

so back in december i bought one of the pi kits from Maker shed. which came with a Kingston SD4/4GB SDHC card. i set up the Pi as a web server for a tor onion site. except for downtime for backups of the SD card, the web server has been running 24/7 for almost 3 months. this last weekend, the Pi crashed, and after i restored the card from a backup, got it up and running again, but noticed there were some errors in some of the HTML files. i just bought a replacement card, and the pi is running again. when i did a fsck.ext4 of the original SD card (the linux partition, not the boot partition), there were many random errors, i reformatted the card, and it is about 100Megs smaller now. i think i'm going to buy several spare SD cards, since running 24/7 is probably kind of hard on the card. i'm now pretty much satisfied with the setup i have, and won't be going through the cycle of "let's try installing this.... i don't like it so uninstall it.... this looks like it might be better, so install it... yeah that works...". so the new SD card might last a bit longer than 3 months... yes it seems to be running much smoother now.... the periodic slamming of the CPU seems to have gone away (it had started doing that about the middle of last week... slamming the CPU for 15-30 seconds once every 5 minutes)

Aydan
Posts: 667
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 11:21 am

In such a scenario you'll have to be careful with proper handling of wear leeveling.

#1: run trim periodically (e.g. once a day)
this erases unused blocks and puts them back into the wear leveling pool. The FS driver doesn't do this automatically, except if you mount with the discard option, but this will lower write performance.

#2: leave enough free space on your card as wear leveling pool.

Regards
Aydan

User avatar
RaTTuS
Posts: 10323
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 11:34 am

if you are doing writes to the SD card [logs storing stuff] then minimise that so it does not ...
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

poing
Posts: 1131
Joined: Thu Mar 08, 2012 3:32 pm

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 11:46 am

In this thread: http://www.raspberrypi.org/phpBB3/viewt ... 66&t=37213 someone else is complaining on the same issue with a 4GB Kingston card. I use 8GB Sandisk cards (blue with red, old type) and never had a card corruption, even while often 'pulling the plug' without proper shutdown. So maybe the Kingston cards are not that suitable for use in the Pi.

User avatar
Mortimer
Posts: 923
Joined: Sun Jun 10, 2012 3:57 pm

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 12:04 pm

Poing, isn't more likely that your 8GB card has more room for wear levelling than unclejed613's 4GB card.

I stopped using my 4GB cards in the RPi, because I thought the amount of spare space was too low, and started using 16GB cards instead. I am beginning to think this might have been a good move.
--------------
The purpose of a little toe is to ensure you keep your furniture in the right place.

Aydan
Posts: 667
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 12:28 pm

The easiest way to not run into wear leveling problems is to not partition the whole card.
So even if all blocks on the partition are in use (meaning non-zero contents, not disk full) you'll have blocks available "outside" for the wear leveling pool.
You'll just have to make sure that the blocks outside of the partition really are zeroed out.

Regards
Aydan

poing
Posts: 1131
Joined: Thu Mar 08, 2012 3:32 pm

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 12:47 pm

Mortimer wrote:Poing, isn't more likely that your 8GB card has more room for wear levelling than unclejed613's 4GB card.
I'm far from an expert on wear leveling, but the OP states that he found errors in his HTML files. These are static so I doubt very much the corruption was due to wear :D

Assuming he had 2GB left on his card and each block can be written to 4,000 times the theoretical amount of data that can be written is 8TB. I've no clue about his setup, but in 'almost three month' that would come down to writing roughly 100GB per day to the card.

Assuming only writing lines to a log file and blocks of 512KB the amount of write operations would need to be almost 200,000 per day in the same period for the card to wear out.

User avatar
Mortimer
Posts: 923
Joined: Sun Jun 10, 2012 3:57 pm

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 1:23 pm

If the HTML files do not occupy a whole block, they could get rewritten to a new block, when they share the block with a file that is getting rewritten, modified, deleted or whatever. If the new block is worn out, you might get a corrupted HTML file. Just because a file is never changed on flash storage, doesn't mean it won't get moved around. And if the trim function isn't being used, then the number of block receiving wear could be reduced somewhat.
--------------
The purpose of a little toe is to ensure you keep your furniture in the right place.

poing
Posts: 1131
Joined: Thu Mar 08, 2012 3:32 pm

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 2:22 pm

I didn't know a block could contain parts of more than one file? I thought the reason between 'real occupied disk space' and 'real file size' was because a block can only be used by on file and if the file is smaller than the block it uses the size of the block instead of it's own size.

User avatar
Mortimer
Posts: 923
Joined: Sun Jun 10, 2012 3:57 pm

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 4:07 pm

You could be right, but the answer could lie in something called "Static Wear Levelling". The first wear levelling algorithms tried to even wear by changing the flash memory block allocated to a logical block address (LBA) when data was being written. Thus a file that never changed protected the blocks it used from ever being worn out. Newer algorithms incorporate "Static Wear Levelling", which means that files that don't change are periodically copied to different blocks so that the blocks they occupied can be cycled through the wear levelling pool. Thus a non-changing file is not necessarily protected from corruption as a card nears the end of its useful life.
--------------
The purpose of a little toe is to ensure you keep your furniture in the right place.

Kronalias
Posts: 13
Joined: Sat Dec 22, 2012 4:39 pm

Re: sd card life running a pi 24/7

Fri Mar 15, 2013 8:08 pm

Interesting discussion.

FWIW I've been running two Pi's 24/7 - one since just before Xmas 2012 (you can prolly guess why), and one from about a month ago. All(?) my transient files are held in ram - either in a 32MB ramdisk I've made, or another chunk of ram that's used for all the /var/log stuff. I'm not using fstrim, primarily because I'd not thought of it...

Both Pi's are headless, and both use 4GB SD cards.

The first one crashes (or more precisely reboots on a kernel panic or when the wifi drops and cannot reconnect) a few times a day. It also runs a webserver ...
http://kronalias.no-ip.org/
.... so the restarts could be caused by traffic(?). Its card is a Transcend 4GB Class 6 SDHC.

The second Pi is not accessible to the outside world (it's on the dark web), and its uptime is faultless.
That one uses a Hama 4GB Class 10 SDHC card.

Both cards have been tested with fsck, as in I've put the cards in a different PC, made sure they're not mounted, and executed these commands on both cards:
sudo fsck /dev/mmcblk0p1 -v
sudo fsck /dev/mmcblk0p2 -v
They're fine.

HTH

BTW, I'm assuming that the Class is the number inside the Big C on the physical card label, but I could well be wrong on this one.

unclejed613
Posts: 55
Joined: Sun Jan 15, 2012 10:49 pm

Re: sd card life running a pi 24/7

Sat Mar 16, 2013 2:53 am

actually, when the pi crashed last saturday, there were libraries used by Apache that had got corrupted, as well as an ogv file i was using as part of an html5 experiment. when i re-flashed the card, i had removed another large file (an mp4 file that i had used for testing omxplayer) before copying my web data directories across (i back up just the web directories once or twice a week using gftp, and do a whole-card backup once a month). so the oder of where certain files were on the card had been reshuffled a bit. after the crash, the pi would boot ok, but complain that one of the apache libraries was corrupted, so apache wouldn't start. so i re-flashed the card and reinstalled the web data. as i browsed through the wep page, to make sure all the links and files were ok, i saw one that had corrupted data in it. tnx for the info about trim. can i run that without unmounting the partitions?


edit:
trim
bash: trim: command not found

unclejed613
Posts: 55
Joined: Sun Jan 15, 2012 10:49 pm

Re: sd card life running a pi 24/7

Sat Mar 16, 2013 3:11 am

nvm... kronalias mentioned fstrim.... found it, reading the manpage now

User avatar
abishur
Posts: 4477
Joined: Thu Jul 28, 2011 4:10 am
Location: USA
Contact: Website

Re: sd card life running a pi 24/7

Sat Mar 16, 2013 3:23 am

out of curiosity, what sort of of turbo settings (if any) do you have enabled?
Dear forum: Play nice ;-)

inerlogic
Posts: 8
Joined: Sat Dec 22, 2012 4:23 am

Re: sd card life running a pi 24/7

Sat Mar 16, 2013 4:33 am

there's a tutorial for switching the system over to a USB HDD....
i've got RASPBMC and a development Pi running from old portable USB drives, powering the Pis and drives from powered Hubs.....

sorry, too lazy to google the link for you, but... yeh.... solid state memory has a finite lifespan (more so than HDDs) so if you're going to be running a server or moving large files around frequently.... use your head...

unclejed613
Posts: 55
Joined: Sun Jan 15, 2012 10:49 pm

Re: sd card life running a pi 24/7

Mon Mar 18, 2013 3:13 am

no turbo settings, running "plain vanilla".

i did notice that the main partition is ext4. this uses journaling. this means a fair amount of writes during normal operation. wouldn't ext2 that doesn't have journaling have been a better choice for a flash drive?

forumisto
Posts: 386
Joined: Fri Mar 16, 2012 8:41 am

Re: sd card life running a pi 24/7

Mon Mar 18, 2013 11:39 am

how we can make full backup of SD card without powering off the RPi?
It's interesting if we need to replace the SD card and don't reinstall and reconfigure the programs

Aydan
Posts: 667
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: sd card life running a pi 24/7

Mon Mar 18, 2013 3:25 pm

unclejed613 wrote:trim
bash: trim: command not found
The command is called fstrim and you run it while mounted.
The first time after boot it will "trim" all blocks unused by the fs, no matter if they're zeroed/trimmed already or not. Depending on the speed of the SD card this can take a few minutes.
subsequent runs only trim blocks freed after the last run.

Regards
Aydan

unclejed613
Posts: 55
Joined: Sun Jan 15, 2012 10:49 pm

Re: sd card life running a pi 24/7

Wed Mar 20, 2013 3:46 am

i'm not sure if you could dd the partitions to an NFS share, that might be one way to make a backup while the pi is running of it's possible....

you can copy all of the contents while the pi is running, but dd'ing it while it's running might not be such a good idea.... you are still writing log files, writing to swap etc (plus, if i understand correctly the SD card's memory manager is operating at the same time, moving stuff around for wear leveling), and you might get a corrupted image just because of that.

Aydan
Posts: 667
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: sd card life running a pi 24/7

Wed Mar 20, 2013 6:36 am

unclejed613 wrote:writing to swap
this shouldn't matter since swap is reinitialized after boot.
unclejed613 wrote:plus, if i understand correctly the SD card's memory manager is operating at the same time, moving stuff around for wear leveling, and you might get a corrupted image just because of that.
even if an sd card did static wear leveling (moving long unchanged blocks around in it's spare time), which is highly doubtful, it would be transparent (invisible outside the card). So no corruption there.

Even so, having a slightly corrupted image is still better than having no backup at all, and who cares about a corrupted logfile as long as the "production" data is OK.

Regards
Aydan

Return to “General discussion”