Hello guys!
After I have got my first rpi I installed the services I needed. Before making it online I wanted to test it with some automatic tools to see if it can go public. My SD card crashed then.
The problem comes when you write the same part of an SD card many times - if you have a public server, apache for example - users or maybe attackers can send a lot of requests which are logged.
Logging means IO usage - write. That's what flash cards don't like.
After sending thousand of requests to my rpi it' not a big deal to find out what happened. An attacker can use this method to DOS you server.
Of course there is protection for this: use ramfs and/or tmps.
Using ramfs won't hurt your SD card at all and make the system load decrease, but if you shut down the server you loose the logs. Using tmps with "noatime" mount option will decrease the IO usage - it won't save when was the file last accessed but after shut down you will have your logs.
Turning off swap is also a good idea, so not even tmpfs will use it. However that needs some more space management to not to run out of space for logging.
Be careful with big logs - they can destroy your SD card fast.
Important notice for rpi public servers
Last edited by Near on Tue May 07, 2013 5:30 pm, edited 2 times in total.
Re: Important notice for rpi public servers
The problem with tmpfs is that if there is not enough free memory to hold the files they will be written to the swap file or partition instead. That swap will be on your SD card and so you will only have reduced the writes somewhat.
One of the best ways to prevent the SD card from wearing out too soon is to have the root partition (and a swapfile/partition if you need one) on a USB HD. A small drive should be sufficient as long as it has its own power supply.
I don't understand this statement. As the tmpfs partition is just a ram disk that allows swap to be used when needed, how is it going to save your logs across shutdowns?Using tmps will decrease the IO usage, but after shut down you will have your logs.
One of the best ways to prevent the SD card from wearing out too soon is to have the root partition (and a swapfile/partition if you need one) on a USB HD. A small drive should be sufficient as long as it has its own power supply.
Re: Important notice for rpi public servers
I just forgot to mention swap and "noatime" mount option, but I have edited and corrected it. Hope it's ok now.rpdom wrote:The problem with tmpfs is that if there is not enough free memory to hold the files they will be written to the swap file or partition instead. That swap will be on your SD card and so you will only have reduced the writes somewhat.
I don't understand this statement. As the tmpfs partition is just a ram disk that allows swap to be used when needed, how is it going to save your logs across shutdowns?Using tmps will decrease the IO usage, but after shut down you will have your logs.
One of the best ways to prevent the SD card from wearing out too soon is to have the root partition (and a swapfile/partition if you need one) on a USB HD. A small drive should be sufficient as long as it has its own power supply.
Re: Important notice for rpi public servers
As someone who runs a public Pi server, and regularly demonstrates its power by running ApacheBench against it (and throwing 5000 connections at it at once.. repeatedly), I would dispute the view that a Pi is DoS'ed any more easily than any traditional server.
I've had multiple experiences with "poweful" cPanel servers effectively going offline due to slightly increased load.
Issues around SD cards crashing come up here often, and it's often attributed to cheap cards moreso than running public web servers.
I've had multiple experiences with "poweful" cPanel servers effectively going offline due to slightly increased load.
Issues around SD cards crashing come up here often, and it's often attributed to cheap cards moreso than running public web servers.
Re: Important notice for rpi public servers
DoS is caused by making the SD card wear out. Normally, even good quality cards can take ~100.000 writes. On a stress and/or pentest ten thousands of requests are sent and logged. 100.000 writes are not so much if we speak about logs, mainly if the logs rewrite itself and not expanding.technion wrote:As someone who runs a public Pi server, and regularly demonstrates its power by running ApacheBench against it (and throwing 5000 connections at it at once.. repeatedly), I would dispute the view that a Pi is DoS'ed any more easily than any traditional server.
I've had multiple experiences with "poweful" cPanel servers effectively going offline due to slightly increased load.
Issues around SD cards crashing come up here often, and it's often attributed to cheap cards moreso than running public web servers.
Re: Important notice for rpi public servers
Hi,
If it's really a big worry, use a USB hard drive for the logs (or a large capacity [wear levelling?] pen drive/SSD)?
Cheers,
Adam
EDIT: Sorry - I missed this:
If it's really a big worry, use a USB hard drive for the logs (or a large capacity [wear levelling?] pen drive/SSD)?
Cheers,
Adam
EDIT: Sorry - I missed this:
rpdom wrote:One of the best ways to prevent the SD card from wearing out too soon is to have the root partition (and a swapfile/partition if you need one) on a USB HD. A small drive should be sufficient as long as it has its own power supply.
Re: Important notice for rpi public servers
I am afraid that it is 3 000-10 000 writes for MLC cards, not 100 000. And for newer cards using TLC it can be as low as 1 000.Near wrote: DoS is caused by making the SD card wear out. Normally, even good quality cards can take ~100.000 writes. On a stress and/or pentest ten thousands of requests are sent and logged. 100.000 writes are not so much if we speak about logs, mainly if the logs rewrite itself and not expanding.
BTW you can also use separate /tmp partition with very long commit interval and may be without journal (ext2) to reduce writes.
I am running web server on Pi for few mounth now, and still have no problems with card. But site which is hosted there is not too popular, something like 100 hosts/day.
Re: Important notice for rpi public servers
It might be good to have a table somewhere for webservers
card brand, size, low/med/hi loaded site, time to failure
on some wiki to track this issue. Some people think it is a big fuss about nothing.
Do the current sd/microsd cards have wear levelling? In which case a larger sd card
might solve the problem for smaller sites.
I have disabled swap, and put my logs in tmpfs. Also changed logrotate to keep fewer
logs so they dont fill up the space. I also awstat the web logs frequently enough to not care
much when I lose them on a reboot.
Some interesting commands to check for io writes are
1. iotop -oa (its a top for io)
2. iostat -d <time> <count> to see how many writes to sd happen per <time> seconds.
The goal is to get it 0 writes when the pi is not busy.
--
card brand, size, low/med/hi loaded site, time to failure
on some wiki to track this issue. Some people think it is a big fuss about nothing.
Do the current sd/microsd cards have wear levelling? In which case a larger sd card
might solve the problem for smaller sites.
I have disabled swap, and put my logs in tmpfs. Also changed logrotate to keep fewer
logs so they dont fill up the space. I also awstat the web logs frequently enough to not care
much when I lose them on a reboot.
Some interesting commands to check for io writes are
1. iotop -oa (its a top for io)
2. iostat -d <time> <count> to see how many writes to sd happen per <time> seconds.
The goal is to get it 0 writes when the pi is not busy.
--
Re: Important notice for rpi public servers
I'm getting a 512MB SD Card with the boot partition on it. 
Then I will add USB HDD of some form with the OS.
Finally I will change config.txt to reflect the USB HDD.
Then I will add USB HDD of some form with the OS.
Finally I will change config.txt to reflect the USB HDD.
Code: Select all
int main() {
try
throw "off table";
} catch (void e) {
printf("Phew...");
}
}