Holy One
Posts: 29
Joined: Mon Jul 02, 2012 5:44 pm
Contact: Website

using logrotate and ramlog

Sat Jul 28, 2012 2:42 pm

I was just wondering if anyone had tried this?

basically 99% of all logs are unimportant and reducing the writes to the sdcard would be nice. so using a small ramdisk for logging, that is only written out when the system is shut down or after a set period of time seems like a good idea.

using logrotate to cap the size of those logs even better?

more info: http://archive09.linux.com/feature/141231 && http://www.debian-administration.org/articles/117

Elect
Posts: 11
Joined: Tue Aug 07, 2012 7:48 am
Location: Netherlands

Re: using logrotate and ramlog

Mon Aug 13, 2012 9:07 am

Is this the project you are referring to? http://www.tremende.com/ramlog/
I've tried to install ramlog, but dpkg ended with an error. But I'm really interested in working with ramlog as i'm not very happy with writing often to the SD card.

Momentarily i'm using a 1 MB RAM filesystem for my MRTG images. This works well, although i'm losing everything when it restarts. But it's better then buying a new SD card.

Elect
Posts: 11
Joined: Tue Aug 07, 2012 7:48 am
Location: Netherlands

Re: using logrotate and ramlog

Mon Aug 13, 2012 5:20 pm

Got Ramlog to work and wrote a little tutorial. It seemed that I missed a package in order to Ramlog.

http://piserv.nl/blog/index.php/using-t ... pberry-pi/

AlterSchwede
Posts: 23
Joined: Sat Jul 28, 2012 9:00 am

Re: using logrotate and ramlog

Sun Oct 14, 2012 9:45 am

Your ramlog tutorial is not available. Can you post it here?
I have some problems with rsyslog and wonder how you solved it...

Code: Select all

Oct 14 11:28:03 Starting ramlog-tmpfs 2.0.0: Error: /var/log is in use... [fail]
Oct 14 11:28:05 The list of open files: (You need to close below daemons if you want to start/stop ramlog manually)

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
rsyslogd 1805 root    1w   REG  179,2    45580 66966 /var/log/syslog
...

AlterSchwede
Posts: 23
Joined: Sat Jul 28, 2012 9:00 am

Re: using logrotate and ramlog

Sun Oct 14, 2012 1:17 pm

Update:
This is what is needed:

Code: Select all

sudo apt-get install rsync
sudo apt-get install lsof
wget http://www.tremende.com/ramlog/download/ramlog_2.0.0_all.deb
sudo dpkg -i ramlog_2.0.0_all.deb
# you need to add the following in /etc/init.d/ramlog:
# X-Start-Before: rsyslog
# X-Stop-After: rsyslog
# to the ###BEGIN INIT INFO block in /etc/init.d/ramlog
# and perhaps add more services to the list  (lighthttp, samba etc. if you are using these), e.g.
 X-Start-Before: rsyslog lighthttp
# and do
sudo insserv
...
# reboot twice
...
# check:
cat /var/log/ramlog
# that every think is OK
# and
sudo /etc/init.d/ramlog status
# that ramlog is running

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Thu Oct 25, 2012 8:08 pm

Hi,

I try to use Ramlog on a fresh install, but i'm stuck :(

ramlog always start after rsyslog, so /var/log is locked....

I do this (with insserv command and reboot after all modifications)
- add in /etc/init.d/ramlog

Code: Select all

X-Start-Before: rsyslog

==> fail

- add in /etc/init.d/rsyslog

Code: Select all

Required-Start: ramlog
===> fail again

Can anyone help me ?

here my ramlog logs:
Oct 25 21:42:55 Starting ramlog-tmpfs 2.0.0: [fail]
Oct 25 21:44:21 Starting ramlog-tmpfs 2.0.0: Error: /var/log is in use... [fail]
Oct 25 21:44:22 The list of open files: (You need to close below daemons if you want to start/stop ramlog manually)

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1801 root 1w REG 179,2 347486 4426 /var/log/syslog
rsyslogd 1801 root 2w REG 179,2 311681 1594 /var/log/kern.log
rsyslogd 1801 root 5w REG 179,2 18904 1592 /var/log/debug
rsyslogd 1801 root 6w REG 179,2 286188 1600 /var/log/messages
rsyslogd 1801 root 7w REG 179,2 27470 4902 /var/log/daemon.log
rsyslogd 1801 root 8w REG 179,2 21113 4903 /var/log/auth.log
console-k 2019 root 8w REG 179,2 8679 4899 /var/log/ConsoleKit/history
Oct 25 21:44:22 Test result: ramlog cannot be started or stopped at the moment.

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Fri Oct 26, 2012 9:03 am

Solved

I miss a parameter in /etc/default/ramlog....

The boot/stop order works fine
Oct 26 10:39:31 Starting ramlog-tmpfs 2.0.0: [ OK ]
Oct 26 10:45:21 Saving logs to hdd: [ OK ]
Oct 26 10:45:22 Stopping ramlog: [ OK ]
Oct 26 10:45:33 Starting ramlog-tmpfs 2.0.0: [ OK ]
Oct 26 10:47:29 Saving logs to hdd: [ OK ]
Oct 26 10:47:32 Stopping ramlog: [ OK ]
Oct 26 10:47:42 Starting ramlog-tmpfs 2.0.0: [ OK ]

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Tue Oct 30, 2012 6:51 am

Hi,
after having successfully installed ramlog and after fiddling with the order of daemons during boot (some entries here were quite helpful, thank you), I am now stuck with a message that ramlog could not find a shutdown flag file. Googling for shutdown flag file did not give me further clues.

Is there a friendly soul who can give me further hints ?

Thank you all in advance
Joshi

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Tue Oct 30, 2012 9:42 am

Hi,

I had the same issue....
Ramlog start correctly, but the "shutdown flag" error occurs when I try to stop it....

When ramlog process try to stop, if rsyslog is still running, /var/log is lock and ramlog fail to stop.
I solved it by adding ramlog in the "stop order" of daemon "rssylog".

In /etc/init.d/rsyslog, I add :

Code: Select all

# Required-Start:    ramlog
# Required-Stop:     ramlog
Required start : ramlog start before rsyslog
Required stop : rsyslog stop before ramlog

Try this ;)

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Tue Oct 30, 2012 10:11 am

Hi freemanbubu,
thank you for your kind help, but this trick did not solve my problem :-(
Still the same error mesage :-(

But anyway: thanks a lot !

Cheers
Joshi

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Tue Oct 30, 2012 10:25 am

You add this in the LSBInit script, a the top of /etc/init.d/rsyslog ?

like this example (not actual rsyslog script):

Code: Select all

### BEGIN INIT INFO
# Provides:          scriptname
# Required-Start:    $remote_fs $syslog ramlog
# Required-Stop:     $remote_fs $syslog ramlog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO
without uncomment, just add "ramlog" at end of lines Required-start & required-stop

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Tue Oct 30, 2012 12:37 pm

Yes, I added it exactly there and did not uncomment it. The only difference is that I did not put ramlog at the end but rather in the beginning. Don't thing that this will make a difference, but I'll give it a try and put it at the end.
From my experiments with the other daemons: do you think it is worthwhile to add ramlog in the respective files of the other daemons (cups, lighttpd, php-fpm,samba) ? I had to include those in the X-Before section of ramlog.

Thank you and cheers
Joshi

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Tue Oct 30, 2012 1:42 pm

I think process order don't make any difference.

About edit all init.d script, I allready try this, but my ramlog config file was wrong....so I can't tell now if it works or not....
I can't access my Pi now, but I will try this later.

Can you post outpout of command:

Code: Select all

sudo service ramlog stop

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Tue Oct 30, 2012 2:28 pm

You are right, process order did not change anything.
I have put 'ramlog' in all Required Start and Required Stop sections of the daemons (samba, lighttpd, syslog, php5-fmp, cups), but without any effect.
So I have doubts that (at least for my type of problem) the Required Start/Stop thing changes anything ...

Here is the output of 'sudo service ramlog stop':

ramlog: Cannot find shutdown flag file. ... (warning).
ramlog: Cannot find shutdown flag file.Stopping ramlog: Error: ramlog is not running failed!

Cheers
Joshi

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Tue Oct 30, 2012 5:05 pm

... I think I misunderstand something:
Did your ramlog process start ? You have an error when you try to stop it, but does it start ?

can you post output of:

Code: Select all

sudo service ramlog start
and

Code: Select all

df -h
thanks


Here my actual config :

/etc/init.d/rsyslog

Code: Select all

#! /bin/sh
### BEGIN INIT INFO
# Provides:          rsyslog
# Required-Start:    $remote_fs $time ramlog
# Required-Stop:     umountnfs $time ramlog
# X-Stop-After:      sendsigs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: enhanced syslogd
# Description:       Rsyslog is an enhanced multi-threaded syslogd.
#                    It is quite compatible to stock sysklogd and can be
#                    used as a drop-in replacement.
### END INIT INFO
/etc/init.d/ramlog

Code: Select all

### BEGIN INIT INFO
# Provides: ramlog
# X-Start-Before:
# Required-Start:
# Should-Start: $local_fs
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: moves /var/log into ramdisk
# Description: ramlog daemon moves /var/log to ramdisk on startup and copies it back to harddrive on shutdown or restart
### END INIT INFO
df -h command (with ramlog running, see ramlog-tmpfs):

Code: Select all

[email protected] ~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.4G  1.6G  5.5G  23% /
/dev/root       7.4G  1.6G  5.5G  23% /
devtmpfs        117M     0  117M   0% /dev
tmpfs            24M  212K   24M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            47M     0   47M   0% /run/shm
/dev/mmcblk0p1   56M   34M   23M  60% /boot
/dev/root       7.4G  1.6G  5.5G  23% /var/log.hdd
ramlog-tmpfs     10M  3.0M  7.1M  30% /var/log
And ramlog config /etc/default/ramlog

Code: Select all

#Ramlog service configuration

RAMDISKTYPE=0           #0 = tmpfs (can be swapped)
                        #1 = ramfs (no max size in older kernels, cannot be swapped, not SELinux friendly)
                        #2 = old kernel ramdisk
                        # (kernel ramdisk needs kernel parameter "ramdisk_size=" in kilobytes as default kernel ramdisk size
                        #  is usually lower than the size of the /var/log)

TMPFS_RAMFS_SIZE=10m    #Maximum size of memory to be used by tmpfs or ramfs.
                        # The value can be percentage of total RAM or size in megabytes - for example:
                        # TMPFS_RAMFS_SIZE=40%
                        # TMPFS_RAMFS_SIZE=100m
                        # Empty value means default tmpfs/ramfs size: TMPFS_RAMFS_SIZE=
                        # For more options please check 'man mount', section 'Mount options for tmpfs'
                        # (by the way - ramfs supports size limit in newer kernels as well despite man says there are no mount options)
                        # It has only effect if RAMDISKTYPE=0 or 1

KERNEL_RAMDISK_SIZE=MAX #Kernel ramdisk size in kilobytes or MAX to use entire ramdisk. It has only effect if RAMDISKTYPE=2

LOGGING=1               #0=off, 1=on
LOGNAME=ramlog          #name of the ramlog log file (makes sense if LOGGING=1)

VERBOSE=1               #0=off, 1=on (if 1, teststartstop puts detials to the logs and it is called after start or stop fails)

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Tue Oct 30, 2012 7:38 pm

ramlog does not start. In /var/log/ramlog it says: ramlog start [fail]
df -h shows that there is no ramlog
Thank you for posting your /etc/init.d/ramlog. You don't have entries under X-Start-Before whereas I have all my daemons listed there. I will now remove these entries - we'll see what will happen.

Will be back in some minutes.

Joshi

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Wed Oct 31, 2012 5:32 am

Good morning
Still no success. I will now make a fresh start with an image I saved before installing the ramlog.
I have a habit to make an image everytime before I start to install new major things.

Perhaps all the fiddling has damaged something.

Unfortunately today I will be ocupied with other things, so I very likely cannot spend too much time on this. Wille be back tomorrow with more results - hopefully.

Have a nice day
Joshi

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Wed Oct 31, 2012 8:53 am

ok ok,

You can save your actual image, and rebuild a new system from website iso (to start from a fresh install)

Do the config during first boot (raspi-config), then:

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo wget http://www.tremende.com/ramlog/download/ramlog_2.0.0_all.deb
sudo dpkg -i ramlog_2.0.0_all.deb
sudo reboot
sudo reboot
need to reboot twice....

Then, look with df -h if ramlog is OK
If ramlog don't start, do

Code: Select all

sudo servcie ramlog start
In the output, you will see what process use /var/log

Then, insert in these process (/etc/init.d/) "ramlog" in Required-Start and Required-Stop
Reboot twice, and check if all is OK.

If it's working, save your config files, restore your image, and compare....
I don't understand why ramlog never works, you do all configs and modifications possible.... :shock:

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Wed Oct 31, 2012 8:31 pm

Good evening,
it is really a good evening, because it works now :D
Obviously the system was in a sort of limbo state because of my many unsuccessful attempts.
Now I have put in my daemons in X-Start-Before as well as in Required Start. df -h shows the ram disk.
Thank you very much for your helpful hints.
Now I will drink a glass of wine to your health ;)

Have a nice evening
Joshi

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Thu Nov 01, 2012 2:18 pm

Hi

great news !
FYI, I allow 10m to ramlog, it's enough for "basic" usage : web, sql, php, mumble

Have a nice day :)

(i'm french, I hope you drink french wine 8-) )

joshmosh
Posts: 62
Joined: Fri Aug 03, 2012 12:04 pm

Re: using logrotate and ramlog

Thu Nov 01, 2012 7:54 pm

Good evening to you,
I currently have a ramlog size of 16 M - probably a bit of a waste. I will keep an eye on it over the next few days ...

I am German and my preferred wines come from Italy, South Africa and France - in that order, sorry. France has a number of outstanding wines, but they are rather expensive, at least here in Germany.
Right now I am sipping a splendid Vino Nobile de Montepulciano - I hope you can forgive me ;-)

My next project is to use the Raspi as a replacement for my Apple TV. Unfortunately I have an ATV3, which at the moment cannot be jailbroken at therefore is somewhat crippled. If this works reasonably well, then I will order my second Raspi :-)

Have a nice evening - with a glas of good French wine :-)

Santé !
Joshi

User avatar
Thijxx
Posts: 86
Joined: Mon Oct 22, 2012 1:25 pm
Location: The Netherlands

Re: using logrotate and ramlog

Fri Nov 16, 2012 4:06 pm

I also like the Vino Nobile de Montepulciano! But thats offtopic.

Ramlog won't start. At one point I noticed the RAM disc using the "df -h" command but that's also not there anymore. Guess there is something wrong in the 'ramlog' file...

My /etc/init.d/ramlog file:

Code: Select all

### BEGIN INIT INFO
# Provides: ramlog
# Required-Start: 
# X-Start-Before: rsyslog transmission-daemon openvpn pptpd shellinabox
# X-Stop-After: 
# Should-Start: $local_fs
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: moves /var/log into ramdisk
# Description: ramlog daemon moves /var/log to ramdisk on startup and copies it back to harddrive on shutdown or$
### END INIT INFO
                                                                                                                 
############################################################
# Ramlog default variables - please change variables in 
# /etc/sysconfig/ramlog on Fedora/CentOS systems and in
# /etc/default/ramlog on Ubuntu/Debian systems

RAMDISKTYPE=0
TMPFS_RAMFS_SIZE=30m
KERNEL_RAMDISK_SIZE=MAX
LOGGING=1
LOGNAME=ramlog
VERBOSE=1
CONFIGFILE=/etc/init.d/ramlog
 

My /etc/init.d/rsyslog file:

Code: Select all

### BEGIN INIT INFO
# Provides:          rsyslog
# Required-Start:    $remote_fs $time ramlog
# Required-Stop:     umountnfs $time ramlog
# X-Stop-After:      sendsigs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: enhanced syslogd
# Description:       Rsyslog is an enhanced multi-threaded syslogd.
#                    It is quite compatible to stock sysklogd and can be 
#                    used as a drop-in replacement.
### END INIT INFO
My /var/log/ramlog file:

Code: Select all

Nov 16 16:58:11 ramlog: Cannot find shutdown flag file.                  [warning]                               
Nov 16 16:58:11 Stopping ramlog: Error: ramlog is not running            [fail]                                  
Nov 16 16:58:20 Starting ramlog-tmpfs 2.0.0:                             [fail]                                  
The whole X-Start-Before, Requiered-Start etc. is not clear to me. I'll start Googling/reading about it ;) and maybe you guys know how to configure it the right way.

Thanks
Mimi: Where'd you come from?
Doyle: My mom and the authorities are still trying to figure that out.

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Mon Nov 19, 2012 10:14 am

can you try this config for /init.d.ramlog:

Code: Select all

### BEGIN INIT INFO
# Provides: ramlog
# X-Start-Before:
# Required-Start:
# Should-Start: $local_fs
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: moves /var/log into ramdisk
# Description: ramlog daemon moves /var/log to ramdisk on startup and copies it back to harddrive on shutdown or restart
### END INIT INFO
And keep your actual /init.d/rsyslog config ?

User avatar
Thijxx
Posts: 86
Joined: Mon Oct 22, 2012 1:25 pm
Location: The Netherlands

Re: using logrotate and ramlog

Mon Nov 19, 2012 12:13 pm

Thanks for your reply.

I've changed the config, after a couple of reboots, nothing has changed.

What I did find is that when I start Ramlog manually, it can't find the /log mount it should have created (at boot-time?) so maybe it's an issue with credentials.
Mimi: Where'd you come from?
Doyle: My mom and the authorities are still trying to figure that out.

freemanbubu
Posts: 111
Joined: Sun Jun 17, 2012 11:12 am

Re: using logrotate and ramlog

Mon Nov 19, 2012 1:07 pm

/var/log is used by many different processes

If ramlog start after one of them, he can't create a "virtual drive" with this folder (currently in use when try to create the symbolic link)
So df -h command don't show the virtual drive.

When you start ramlog manually, the output show what processes use /var/log.

Add "ramlog" in these processes /etc/init.d/** (# Required-Start & # Required-Stop fields)


Can you post the output when you start ramlog manually ?

Return to “Networking and servers”