Serial data logging in pi into separate files

[ggnamita]

Hello,

I have tried to create a serial data logger using the pi and a small tracer that can be used in a car. It logs in various information about the car like the GPS information, the temperature,the battery information and various other things and could be used for trouble shooting in case of some problems in the car. I have successfully been able to connect to the logger and access the trace data via the raspberry pi. I store all the information into a file called 'testing.py' .
But I dont want all the data to be stored in the same file by just appending it. I want to try and save the log data in different files with different time stamps. I have not much idea on how to create separate files with different time stamps.
Please help.I have synced the raspberry pi's time with the GPS in the serial device.

Any suggestions are welcome
Namita.

[topguy]

Why you would log into a file with ending ".py" is beyond me, but it doesnt matter.

But this led be to believe that you programmed in Python, but that might not be correct, you should tell us what language you have used to make the logger.

Does the Pi run all the time or only when the car is "on" ?
Is it good enough to make a new file for each day ?

[ggnamita]

Hello again,

I use python programming . And i actually store the data in a .dat file called 'gps-log.dat'. This .dat file is appended every time i want to store the traces. The file I created to do this was a .py file called 'testing.py'. I just forgot that for a moment. Sorry for the confusion!

The pi runs only when the car is on. And I wanted the file to be updated every minute because in case of power failure the data should not be completely disrupted..I thought of creating separate files for each minute with the date and time(but I dont know how to go about it)

Thanks for replying!
Namita.

[lunarSunset]

Hi,
Assuming the creation of the your logger file is still in python you can easily change your code to save to multiple files instead of appending your data. To do this you would have to modifiy your code to have a variable filename based on the current time and date on the raspberry pi.

The answer to this user's question should give you an idea on how to save a file using a current date and time:
http://stackoverflow.com/questions/1060 ... -in-python

To test if your solution is working simply run your python script multiple times and see if you get different files created.

After successfully modifying your code, the next step is to call your python code every minute.
It might be worthwhile to take a look at cron jobs to accomplish.

Here is a brief summary from raspberry pi:
http://www.raspberrypi.org/documentatio ... ge/cron.md

Hope this helps

[ggnamita]

Thank you so much for that tip i tried it and my logging program seems to work.

But i dont understand how do i check for the different log files with different times?

I know this might be a silly question but I am a newbie in Python, so please bear with me!

I am also attaching a part of the code i used to create the log file and also to get the time information .

Code: Select all

mport serial, io
import time

s = open('log.dat', 'w')
log = time.strftime("%Y%m%d-%H%M%S")
s = open(log + '.dat', 'w')

addr  = '/dev/ttyACM0'  # serial port to read data from
baud  = 9600            # baud rate for serial port
fname = 'log.dat'   # log file to save data in
fmode =  'a'            # log file mode = append~

Namita.

[Richard-TX]

One of the mechanisms that should take care of the many issues regarding logging and logfiles is the "logger" command. Syslog files are rotated automatically so cleanup suddenly becomes a non-issue.

In most cases, having multiple log files based on data type is architecturally wrong. Having a reasonable descriptor with the data makes it easy to grep for. Writing reports or processing with awk or python or perl is also made easier..

I usually prefer something like this for a log file.

date-time|descriptor|data

by setting FS or IFS to "|" in awk or /bin/sh, the separate fields are easily extracted

[ggnamita]

Thank you for that advice Richard-Tx,

By using the logger command will I be able to get the time stamp for the logs too? And I also wanted the log file to update automatically every minute, and the previously stored data to move into another file with its time stamp so that it could be identified during trouble shooting. Any idea on how I could do this?

I checked on the internet but could not find many examples on how this could be done.
Please help!

Namita.

[mariochn]

@namita you are using code from this thread right?

http://www.raspberrypi.org/forums/viewt ... 5&p=644212

[ggnamita]

@namita you are using code from this thread right?

http://www.raspberrypi.org/forums/viewt ... 5&p=644212

Yes. That is the one.