How to delete the oldest lines in a log file?

[Pilsener]

I have set up this project for temperature-monitoring:
https://chrisbaume.wordpress.com/2013/0 ... onitoring/

The problem is that the log file is keeping all recorded data, and is showing everything in the graph.
Old data from more than 2 days ago is not interesting for my use of this, so I would like to add an automatic deletion of old data, keeping only the last (insert number) of readings.

The log file is adding one line for each reading, which occurs every minute:
20131118153501,37.0,20.0
20131118153601,38.0,20.0
20131118153702,37.0,20.0
20131118153801,57.0,20.0
20131118153901,58.0,20.0

How can I do this?

[DougieLawson]

The normal way logging works is that your task has to be sensitive to a SIGHUP. When you get a SIGHUP you close the current log and open a new one.

Then with the logrotate package you can use that to archive and delete old logs based on your logrotate policy.

http://www.debianhelp.co.uk/logwatch.htm has more details.

[croston]

You can use a mechanism called logrotate:
http://www.google.co.uk/search?q=logrotate

[Joe Schmoe]

"logrotate" is certainly the "system-y" way to do this sort of thing. For people who like to give "canned" answers to forum questions, that's certainly the kid-tested, mother-approved "canned" answer.

However, these things are relevant:

1. I personally dislike the way logrotate works - the way it names the old log files. Makes traversing them a royal PIA. I hit (and solved) this problem when I wrote a program to summarize the apt history files (so I could see at a glance everything that had been "apt-get install"'d on the system). It's done, but it is still ugly.
2. What the OP really wants is not the mess that logrotate creates, but rather one single log file, with the early stuff removed. See below for suggestions on how to do it.

1. The easiest way to do it is not to do it, but rather to create a browser that gives you the view that you want. So, you write an AWK script (or similar) to ignore lines you don't want to see. Something like:
   

   Code: Select all

   gawk '/no more than 2 days old/' | less

2. You go ahead and do it - but then you have to deal with the issues of making sure that your utility is leaving the file alone so that you can safely edit/rename the file. All easily solvable problems, but beyond the scope of this forum posting.

[pluggy]

I'd use a bash script running every hour or three from crontab.

The crux of it would be

cat path&nameoflogfile|head -n 2880 > tempfilename
rm path&nameoflogfile
mv tempfilename path&nameoflogfile

2880 is the number of minutes in two days.
Nothing magical, it lops off all but the last 2880 lines and puts it in a temporary file, deletes the main file and renames the temporary file to the main file name. Feel free to dress it up.

But as always, there's at least a dozen ways of skinning a cat.....

[Joe Schmoe]

The problem is that the utility (the OP's temperature watching script) has the file open, so renaming/deleting it while it is open is likely to lead to strange results.

As Dougie hinted, you need a way for the "rotator" script to tell the "main" script to close the file and wait a bit while operations are underway. SIGHUP is kinda the "standard" for this.

Of course, when all is said and done, the best way to do this would be to put this functionality into the main script itself. Let it handle all of these issues.

[hampi]

I would store the data to a database like SQLite and then do

Code: Select all

delete from mytable where timestamp<datetime('now','-48 hour');

Or something similar.

[Pilsener]

I found that a script for deleting the log files will also work, as long as it also recreates the symbolic link so a new logfile is created in /var/www/.
The cronjob will read the temperature and, if needed, recreate /home/pi/temp.log.
----------
rm /home/pi/temp.log
rm /var/www/temp.log
ln -s /home/pi/temp.log /var/www/temp.log
----------

Not a perfect solution, but it will suit my needs.
I will also try Pluggy's suggestion, as it may be just what I wanted this thing to do.

I wonder why the solution with a symbolic link was used in the first place, as temp.log can be stored directly into /var/www/, making scripting easier.