mrreality13
Posts: 56
Joined: Tue Feb 26, 2019 6:45 am
Location: texas usa

GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 3:44 pm

got this in my rss feed thought id pass it on for discussion:
The fragility of SD cards is the weak link in the Raspberry Pi ecosystem. Most of us seem to have at least one Pi tucked away somewhere, running a Magic Mirror, driving security cameras, or even taking care of a media library. But chances are, that Pi is writing lots and lots of log files. Logging is good — it helps when tracking down issues — but uncontrolled logging can lead to problems down the road with the Pi’s SD card.

[Erich Styger] has a neat way to avoid SD card logging issues on Raspberry Pi, he calls it a solution to reduce “thrashing” of the SD card. The problem is that flash memory segments wear out after a fairly low number of erase cycles, and the SD card’s wear-leveling algorithm will eventually cordon off enough of the card to cause file system issues. His “Log2Ram” is a simple Unix shell script that sets up a mount point for logging in RAM rather than on the SD card.

The idea is that any application or service sending log entries to /var/log will actually be writing them to virtual log files, which won’t rack up any activity on the SD card. Every hour, a cron job sweeps the virtual logs out to the SD card, greatly reducing its wear. There’s still a chance to lose logging data before it’s swept to disk, but if you have relatively stable system it’s a small price to pay for the long-term health of a Pi that’s out of sight and out of mind.

One thing we really like about [Erich]’s project is that it’s a great example of shell scripting and Linux admin concepts. If you need more information on such things, check out [Al Williams’] Linux-Fu series. It goes back quite a way, so settle in for some good binge reading.
https://hackaday.com/2019/04/08/give-yo ... og-to-ram/
remember ,never approach a computer thinking this will only take a minute
~~pi zero w and a 3b+

echmain
Posts: 230
Joined: Fri Mar 04, 2016 8:26 pm

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 4:44 pm

Hmmm...help me understand something...

If the log files in ram are eventually written to the SD card anyway how does this method reduce writes?

Heater
Posts: 13116
Joined: Tue Jul 17, 2012 3:02 pm

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 4:56 pm

I guess the idea is that writing large amounts of data occasionally is better for the SD than writing lots of little updates all the time.

Recall that when you write to SD it does not just write the few bytes you give it but it has to erase a whole FLASH block and rewrite it.

I have no idea how effective this is. When I want a reliable Pi running from SD card I mount the root file system as read only.

swahren
Posts: 102
Joined: Mon Sep 19, 2016 5:24 pm
Location: Germany

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 5:42 pm

The ext4 file system already provide such a mechanism. As per default the data will be cached for 5 sec before writing. The delay is adjustable via "commit" parameter.

https://www.kernel.org/doc/Documentatio ... s/ext4.txt

Heater
Posts: 13116
Joined: Tue Jul 17, 2012 3:02 pm

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 6:01 pm

Well, that is the thing. If the file system is syncing to disk every 5 seconds and you are logging something every 5 or six seconds then the fs mechanism is not saving you any SD block writes.

But if you increase the time between syncing cache to disk you increase the risk of losing updates to other data you might value.

One has to tune the parameters accordingly.

User avatar
AdamStanislav
Posts: 146
Joined: Sun Mar 10, 2019 2:44 am
Location: Wisconsin
Contact: YouTube

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 7:10 pm

This may be a dumb question, but is it possible to simply turn the logging off? I never read any logs, so why waste my SD card for something I do not need or even want?

W. H. Heydt
Posts: 10772
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 8:34 pm

And on a minor side note for the OP...please don't use all caps for your thread title. It makes it look like you're shouting at people.

Andyroo
Posts: 4219
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 08, 2019 8:56 pm

One thing to note - some programs expect a log file to be present at boot time (Lighttpd for example) and if the old logs are not moved to RAM before they start you have a host of issues.

If you are seeing that many entries in your logs that they are impacting the life of the card then you need to fix the issues being logged :lol:
Need Pi spray - these things are breeding in my house...

MarkDH102
Posts: 340
Joined: Fri Feb 13, 2015 3:18 pm

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Tue Apr 09, 2019 6:25 am

In all of my data logging pi's, I log the data to RAM for 24 hours and then commit it to SD card (and email a copy) at midnight.
Other than that my applications do not write to the SD card. These Pi's are also UPS protected...
But I have had a couple of SD cards go write only on me (and only after a couple of years use).

stuartiannaylor

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Tue Apr 09, 2019 2:46 pm

I suggest you have a look at https://github.com/StuartIanNaylor/zram-config

Does zswap, zdir and zlog in one utility.

Log2Ram does some strange things and works with an extremely tight ram allocation.

zram-config uses zram in a OverlayFS upper with the original read only in the lower. Due to the nature of copy-up of COW only writes are in zram and this massively reduces memory footprint.
It also uses the logrotate directive of olddir so logrotate ships oldlogs to persistant rather than choke precious memory.
It also doesn't copy every complete file that has change on every hour as this is pointless really as if you have a system crash and the write of stop isn't accomplished the critical info up to the last hour is lost anyway.

But any directory can be configured with zram-config and because of the OverlayFS no copy of all files is needed on boot.
Large directories can be used in extremely small zram memory footprint because zram is the upper in a OverlayFS mount of the original readonly in lower.

The utility also does zswap and also includes important sys-admin config of mem-limit and tuning parameters of swapiness and page-cache as zram is not HDD like media its a mem technology with near memcpy speed.
You can create any number of zswaps and zdir and a zlog via /etc/ztab, multiple zswap always bemuses me as zram has been multi-steam since kernel 3.15 but hey if you want more than one you can.

Looking to build a community around https://github.com/StuartIanNaylor/zram-config so any ideas or issues please post away and get involved as claim no ownership just annoyed me so many utils for this application are flawed.
Hack, clone and copy as you wish.

Paul Hutch
Posts: 380
Joined: Fri Aug 25, 2017 2:58 pm
Location: Blackstone River Valley, MA, USA
Contact: Website

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Wed Apr 10, 2019 2:45 pm

stuartiannaylor wrote:
Tue Apr 09, 2019 2:46 pm
I suggest you have a look at https://github.com/StuartIanNaylor/zram-config

<snip>

Looking to build a community around https://github.com/StuartIanNaylor/zram-config so any ideas or issues please post away and get involved as claim no ownership just annoyed me so many utils for this application are flawed.
Hack, clone and copy as you wish.
That looks like it could be really useful, except there is no license so it is locked down via full copyright protection.

It's pretty easy to add a license in gitHub and most one of the license templates are for OSI approved licenses.
https://help.github.com/en/articles/add ... repository

If you want everybody including companies to be able use the code choose an OSI approved license because that helps ensure the license can be understood and accepted by company lawyers.

stuartiannaylor

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Thu Apr 11, 2019 4:49 am

Will sort a license after this post also now has a ephemeral mode with complete root ro via overlayfs with zram rw upper.
I need to do a clean install just to check if my hunch is right that the rpi-update and mkinitramfs -o /boot/initrd maybe out of sync with an automated installer, effects only the ephemeral mode on install.
[edit] Nope just me being pessimistic and all is fine as BRANCH=next rpi-update isn't needed and was perplexed why it was included in the https://blockdev.io/read-only-rpi/ setup.

sudo zram-config enable-ephemeral

Code: Select all

[email protected]:~ $ zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lz4          1000M 117.7M 291.2K  680K       4 /rw
/dev/zram1 lz4           750M     4K    76B    4K       4 [SWAP]

[email protected]:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs          466636       0    466636   0% /dev
tmpfs              94948      44     94904   1% /mnt/run
/dev/mmcblk0p2  14803620 1282512  12886860  10% /ro
/dev/zram0        991512  105908    818020  12% /rw
overlayfs-root    991512  105908    818020  12% /
tmpfs             474724       0    474724   0% /dev/shm
tmpfs             474724   18196    456528   4% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             474724       0    474724   0% /sys/fs/cgroup
/dev/mmcblk0p1     44220   28995     15226  66% /boot
tmpfs              94944       0     94944   0% /run/user/1000

If not OK just say on github as for license its do as you wish http://www.wtfpl.net/txt/copying/
If you have any ideas I will create a branch or links to a fork for you so that people have choice.

Code: Select all

MIT License

Copyright (c) 2019 StuartIanNaylor

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Paul Hutch
Posts: 380
Joined: Fri Aug 25, 2017 2:58 pm
Location: Blackstone River Valley, MA, USA
Contact: Website

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Fri Apr 12, 2019 6:08 pm

Excellent license choice, well known, OSI approved, and legally tested, thanks for adding it. I'll be trying your scripts to use in a work project early next week.

XxionxX
Posts: 7
Joined: Sat Apr 13, 2019 5:34 am

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Sat Apr 13, 2019 7:07 am

I was wondering why this wasn't a standard feature on the raspberry pi OS when I read the article. I'm glad to find out that there are some alternatives. I'm going to have to get this implemented when I finally get my pi working correctly.

stuartiannaylor

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Sun Apr 14, 2019 7:40 am

I know the Raspberry engineers had a look and decided not to go with zram probably as it was the time of the original Pi.
If you use a Pi Zero you can really start to notice zram when swappiness is above the default of 60.
Pi2 there is far more IO wait than process wait and Zram on a Pi3/+ works really well.

I think they looked at effect on boot of original models and went against the idea and that is now sort of set in stone in as a Raspberry foundation.
I think they might reevaluate as even on a zero it works well as with most applications don't have anywhere near the constant load that boot does, its just the intense boot of the pi zero where zram load increases overall load.
When I was playing purely with swaps I did a rough and ready with https://github.com/StuartIanNaylor/zram ... d-balancer
As it increases swapiness after load starts to stabilise and overall loadavg does drop.
https://github.com/StuartIanNaylor/zram ... d-balancer was just a rough and ready with using a pi-zero and you can change swapinness and so effect how much swap is used and the load incurred by zram.

Also zram for some reason just got labelled with swaps as it can make extremely fast zram disks compared to sd-media and with the compression that from usage is approx 300%+ with LZ4 even with meagre resources there are a lot uses.
Logs are not the only thing that can write heavily and sd block wear isn't the only use as near memcpy disk storage can be used and then shipped to persistent on stop as say with Samba maybe /var/cache would benefit of residing on zram.

A lot of what is said about Micro SD is pure hocus and it was designed for storage and reading rather than constant writes and still is.
Most applications we don't need to bother and SD cards are so cheap who really cares.
In some applications the cost of access and maintenance to a failed SD card is far more than the couple of $/£ of a replacement card.

The Pi when it comes to swap in default offering with the 100m dphys-swap is bemusing to say the least and the Raspibian engineers also decided to dodge f2fs Flash-Friendly File System which probably wise at the time but maybe especially as the PI has become more powerful that zram and f2fs, but maybe they might be reevaluated.
f2fs got a lot of commits in kernel 4.18 so with Raspbian Buster (4.19) prob months away I have also been looking at f2fs as like Pi's getting faster the price and size of SD cards has also vastly changed since 2012.
Buster also includes the Zstd compression alg that has amazing text compression ratio's, but generally beats Zlib on compression ratio whilst providing near LZO speed. Zstd for Zdir is likely to make an extremely good candidate.
I did try with the testing version of Buster with f2fs as f2fs tries to serialise writes and minimise and also supports a native over-provision that is block-wear and large write quantity over a long period where a concern you could just go crazy with a 100% over-provision and essentially double life expectancy as the volume will remain 100% available by just allocating what is often spare to the over provision.
It doesn't just have to be zram but yeah maybe Raspberry might think of a new raspi-config entry that allows any device to bind more easily with the current dir structure.
If f2fs is of interest there is a ready img here any can play with as a precursor to buster. https://1drv.ms/u/s!AocmAh35i26QiB7pE5qZnfFyi4Iq
I know the image is ArchV7 but its 4.19 currently so its been handy as I am trying to decide whether to offer f2fs bind mounts or make a utility to resize a card and provide native f2fs for /var and /opt and thinking probably the latter as a zswap f2fs dphy-swap file is also an interesting option.

cpc464
Posts: 209
Joined: Tue Jul 08, 2014 5:10 pm
Contact: Website

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 15, 2019 10:01 am

Must admit I can't help being a bit skeptical about the Pi card issue. Some report that their Pis are killing card after card. But for years I've been running a Wordpress blog on a couple of Pi 2's (unixetc.co.uk), with never a problem. The blog gets about 10,000 requests a day, and every one of them appends a line to the Apache log file, among other things. There is similar traffic to other log files, eg the kernel log (mostly firewall messages), the messages file, syslog and probably half a dozen others. eg. 1800 lines have been written to kern.log already today, and it is only 10:54 am.

The system runs a MySQL database, which generates more disk traffic including 2 large database files being updated every couple of minutes. The above servers also have cron jobs that read and write their own large files several times an hour, and they run four other websites including an instance of Nextcloud with an SQLite database. More disk traffic. And all of this is on the local SD card, there is no other disk.

Probably, most processes on your Pi are spitting out disk traffic of some sort. Even a pipeline (|) in a shell can acquire a temporary file from the root device. Many individual shell commands create their own temporary files, eg sort.

Only one card has ever properly died on me, and that was a SD card (the big sort) in a Pi 1, while it was operating a webcam. I try to never power off a pi without a clean shutdown and if I do, I put the card in a Linux laptop and fsck it (which rarely finds an error, incidentally). Maybe I am just lucky? I dunno.

Jim
Unix engineer since 1989

User avatar
Gavinmc42
Posts: 3633
Joined: Wed Aug 28, 2013 3:31 am

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Mon Apr 15, 2019 10:48 am

Long term stuff I use PiCore, runs from ram.
I collect data every minute, log it to a ram file and use cron to write that file to a 3rd partition once a day.
Still using old SD cards as they have been installed for years.
I might loose data on power out but I get emails when power comes back.
The data is not that vital, just a bonus for predictive maintenance usage.

Latest A1 cards might last longer, but SLC cards will last the longest?
SPI based non volatile memory? Parallel non volatile hat?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

stuartiannaylor

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Tue Apr 16, 2019 9:01 am

cpc464 wrote:
Mon Apr 15, 2019 10:01 am
Must admit I can't help being a bit skeptical about the Pi card issue. Some report that their Pis are killing card after card. But for years I've been running a Wordpress blog on a couple of Pi 2's (unixetc.co.uk), with never a problem. The blog gets about 10,000 requests a day, and every one of them appends a line to the Apache log file, among other things. There is similar traffic to other log files, eg the kernel log (mostly firewall messages), the messages file, syslog and probably half a dozen others. eg. 1800 lines have been written to kern.log already today, and it is only 10:54 am.

The system runs a MySQL database, which generates more disk traffic including 2 large database files being updated every couple of minutes. The above servers also have cron jobs that read and write their own large files several times an hour, and they run four other websites including an instance of Nextcloud with an SQLite database. More disk traffic. And all of this is on the local SD card, there is no other disk.

Probably, most processes on your Pi are spitting out disk traffic of some sort. Even a pipeline (|) in a shell can acquire a temporary file from the root device. Many individual shell commands create their own temporary files, eg sort.

Only one card has ever properly died on me, and that was a SD card (the big sort) in a Pi 1, while it was operating a webcam. I try to never power off a pi without a clean shutdown and if I do, I put the card in a Linux laptop and fsck it (which rarely finds an error, incidentally). Maybe I am just lucky? I dunno.

Jim
Nah as its MTB and yeah some get lucky and some don't, but to be honest much is caused by loss of power, maybe more than block wear.
MySql is likely running in a ram pool that is part the MySQL conf.
Why your kern.log has 1800 lines today is maybe something you need to check out.
Some do report card after card being killed and some don't. Some are not worried and some like to mitigate chance via an easy install.
Its just a matter of choice on what your priority is and the cost of access and maintenance as for many its not worthwhile and for some extremely important.
PS if anyone with a lot of logging could they give this a try and see how it goes.
It should use a tiny amount of memory as uses an overlayFS with zram upper but would be grateful of any realworld tests.
https://github.com/StuartIanNaylor/log2zram

As well as mitigating block wear zram drives can make blistering fast directories as you will see the zram of /var/log vs sd of the pi home dir..

Code: Select all

[email protected]:~ $ sudo dd bs=1M count=256 if=/dev/zero of=/var/log/test conv=fdatasync
256+0 records in
256+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 2.90329 s, 92.5 MB/s

[email protected]:~ $ sudo dd bs=1M count=256 if=/dev/zero of=test conv=fdatasync  256+0 records in
256+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 45.7736 s, 5.9 MB/s
Because zram is the upper in the OverlayFS large directories can still be covered by relatively small areas of zram.
The above writing then below reading the file, with the hybrid OverlayFS/Zram being the top bench with standard SD the lower.

Code: Select all

[email protected]:~ $ dd if=/var/log/test of=/dev/null bs=1M count=512 status=progress
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.747901 s, 718 MB/s
[email protected]:~ $ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"               
[email protected]:~ $ dd if=test of=/dev/null bs=1M count=512 status=progress
530579456 bytes (531 MB, 506 MiB) copied, 23.011 s, 23.1 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 23.2814 s, 23.1 MB/s

glum
Posts: 26
Joined: Wed May 16, 2018 11:27 am

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Sat Apr 27, 2019 11:53 am

stuartiannaylor wrote:
Tue Apr 09, 2019 2:46 pm
I suggest you have a look at https://github.com/StuartIanNaylor/zram-config

Does zswap, zdir and zlog in one utility.

Log2Ram does some strange things and works with an extremely tight ram allocation.

<snip>

This is interesting! I've been doing some research in an effort to understand the rationale and history behind the RAM disks that are "standard" in RPi today (i.e. Disk /dev/ram0 ... /dev/ram15).

Is `zram-config` an alternative to this? And if so, can you explain briefly how it is better/preferable?

stuartiannaylor

Re: GIVE YOUR RASPBERRY PI SD CARD A BREAK: LOG TO RAM

Sat Apr 27, 2019 9:13 pm

Ram disk + compression with any alg from proc crypto.

Default is lzo which means you get approx 300% more or use 300% less ram than a ram disk for minimal cpu overhead.
lz4 is supposed to be faster haven't tested the new lzo-rle released as default in kernel 5.

But depending on contents and how much cpu overhead you could use deflate or zstandard

Also it should have a writeback cache but not that important but strangely not enabled in the kernel even though after that it does nothing until you define and enable it through /proc/... but hey.

I have only been playing really as my knowledge of overlayfs isn't great but somehow become quite expert with zram.
For swap if I was honest if it was enabled I would say use zswap as the idle page order isn't just better it works, but no zswap in this kernel compile.
Even with zram the compile switch for writeback cache is disabled but if you have run out of ram and also run out of zram your system is going to run like pants via sd swap.
Steal chrome OS methods and its create a swap mem_size of approx half ram with a disk size of 1.5x ram set vm page-cache to 0 to cut latency and up your swapiness to 80 or above

zram as a disk I am still getting my head around overlayfs and that it has an absolute lack of user tools. I have one tool and really its depreciated due to the addition of redirect_dir but basically with overlayfs being a CoW system it means the upper zram use is only copy-up writes and those can be compressed by 300-400% so you can create extremely fast zero sd write support for large directories with small ram usage.
Just wish there where more official utils for offline merge down as you would think there might be but no.

Almost forgot about log2ram and its just opinion and from feedback elsewhere I have seen.
Log2Ram actually isn't that bad as tmpfs should push out to swap so having swap avail is important and you should have some avail in fact zram as a backing for log2ram would work well.
It just does some strange things like the hourly write of all files that have changed in case of crash and improper shutdown, but when it comes to logs its likely if you lost up to the last hour the rest is of diminished importance.
If you have a service to stop sd writes then why have hourly write outs? Especially for rare occurances like complete crashes and just write out on shutdown or service stop.
Also it keeps all logs in memory and a few have commented that logrotate will shipout old logs to persistent but for some reason not adopted.
zram-conf does swap, logs and dir in one also does emphemeral complete system and uses an overlayfs in conjunction with zram so that memory usage is much less and it ships out old logs to persistent.
Also due to the overlayfs its doesn't have to copy all files on start so boots are faster and because old logs are shipped out on stop only live logs are merged down to persistent so again faster.
It also doesn't need to start so early in the boot cycle and is less problematic with other services. I am not exactly sure as never traced it exactly as there is no log evidence of log2ram really unless you have an email server installed but sometimes it seems to shutdown and lose log info.
I can be be exactly sure but because it starts so early it also ends late and maybe when it copies out those services needed are not around.
There are few things really and both are not perfect as zram-conf needs testing and polish but some of the methods are likely better by design.

Return to “General discussion”