Baakinira
Posts: 1
Joined: Sun Aug 06, 2017 12:50 pm

Data is not instantly written in SD card

Sun Aug 13, 2017 5:42 am

My code in raspberry pi using python generates data every 10 seconds of size 0.4kB from the sensor because of the delay I've put in the code. But the data is not written in the SD CARD instantly. It waits until the data of 4kB is collected and then writes in the SD CARD which makes around 4 minutes of wait time. I actually want to collect data every 1 minute but if I insert delay of 1 minute, it takes 24 minutes to collect 4kb data which is not good. It's this the problem of raspberry pi or the sd card?

User avatar
rpdom
Posts: 11191
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Data is not instantly written in SD card

Sun Aug 13, 2017 9:42 am

The file writes are being buffered as they should be.

Does your code do something like this?

Open file for writing. Loop( Write some data. Wait 10 seconds )

If you want the data to be instantly written to the card every 10 seconds, you either need to do

Loop( Open file for append. Write some data. Close file. Wait 10 seconds )

or

Open file for writing. Loop( Write some data. Flush data to "disk". Wait 10 seconds )

(None of this is real code, just the idea of what the code does).

Of course, writing the data to the card every 10 seconds will wear out the card a lot quicker. There are a few methods that might be better. It depends what you want to do with the data. You could have two files, one buffered as it is now, and another one in somewhere like /dev/shm/ which is a virtual (RAM) disk and you can flush data to that as much as you like and it won't hurt the card. The reason for using two files is that the /dev/shm/ file will lost on a reboot, while the buffered one on the card will be preserved.

jahboater
Posts: 1538
Joined: Wed Feb 04, 2015 6:38 pm

Re: Data is not instantly written in SD card

Sun Aug 13, 2017 9:58 am

Baakinira wrote:
Sun Aug 13, 2017 5:42 am
It's this the problem of raspberry pi or the sd card?
Neither! It is normal and expected behavior for any modern operating system.

As rpdom says there are various solutions.

In C you can flush the data for your specific file with fsync() or open(...,O_SYNC) - there must be equivalents in Python.

A quick improvement for the entire "/" filesystem is to add "commit=1" to the mount options in /etc/fstab which will flush any cached data once per second.

User avatar
davidcoton
Posts: 2063
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Data is not instantly written in SD card

Sun Aug 13, 2017 11:20 am

You need to step back a bit and look at the "big picture" of your data collection scheme.
  1. Are you just logging the data to the SDCard, or are you transferring it over a comms link as well? If not, how and when do you transfer data?
  2. How long do you need to collect data for? (You already know you want readings ideally every minute.)
  3. How critical is the data? Do lost readings matter?
  4. What happens if mains power fails? Is the new data still valid while power is off? Can you afford to loose existing uncommitted (but valid) data?
Part of the response to the answers to these questions might be a UPS. This will improve retention of data held in a volatile cache, and reduce the chances of a corrupt SDCard caused by power failure during writing (which could result in all saved data being lost). It also allows an orderly shutdown when power fails, or a large enough battery to survive expected power cuts.

Another part might be to save the data to a USB memory stick, hard disk, or SSD. These are less vulnerable to repeated writes than SDCards, but use more power.

If the data is really important, it should be copied ASAP to a second system (independent storage, preferably of a different type).
"If it ain't broke, fix it until it is."
“Raspberry Pi is a trademark of the Raspberry Pi Foundation”
The Pink Unicorn is a symbol of commitment to sensible discussion of future versions of the Raspberry Pi ™

User avatar
r3d4
Posts: 882
Joined: Sat Jul 30, 2011 8:21 am
Location: ./

Re: Data is not instantly written in SD card

Sun Aug 13, 2017 11:51 am

Baakinira wrote:
Sun Aug 13, 2017 5:42 am
data is not written in the SD CARD instantly.
:| https://en.wikipedia.org/wiki/Sync_(Unix)

asandford
Posts: 1548
Joined: Mon Dec 31, 2012 12:54 pm
Location: Ealing

Re: Data is not instantly written in SD card

Mon Aug 14, 2017 12:31 am

Insert and commit your results to a database and let it handle things.
You don't need an RDMS like mysql, sqlite3 is fine for this sort of thing (you can move up to an RDMS in the future if needs be).
More details here.

If you wanted the data to be available to other devices, a consideration would be to use an MQ solution where you publish the data to a topic and any interested parties can subscribe to it (MQTT is quite capable and python bindings are available).

Once you think about adding more sensors, you'll need to expand your programming to multi-threading / callbacks or multiple instances, and it can become a bit of a handful.

Another alternative is to use inbuilt Node Red which can do non-blocking, async sensor reading and processing without too much effort from you.

User avatar
davidcoton
Posts: 2063
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Data is not instantly written in SD card

Mon Aug 14, 2017 8:38 am

asandford wrote:
Mon Aug 14, 2017 12:31 am
Insert and commit your results to a database and let it handle things.
...
Another alternative is to use inbuilt Node Red which can do non-blocking, async sensor reading and processing without too much effort from you.
I don't think either of these will work round the caching of disk writes. There will still be uncommitted writes in RAM.
These techniques may be appropriate but it depends on how the data is used.
"If it ain't broke, fix it until it is."
“Raspberry Pi is a trademark of the Raspberry Pi Foundation”
The Pink Unicorn is a symbol of commitment to sensible discussion of future versions of the Raspberry Pi ™

PiGraham
Posts: 2563
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Data is not instantly written in SD card

Mon Aug 14, 2017 9:19 am

Baakinira wrote:
Sun Aug 13, 2017 5:42 am
I actually want to collect data every 1 minute but if I insert delay of 1 minute, it takes 24 minutes to collect 4kb data which is not good. It's this the problem of raspberry pi or the sd card?
What is not good about that? Are you concerned that data may be lost of power fails? Do you need to access the data during those 24 minutes? Something else?

If you are concerned about data loss you could use a UPS circuit that will keep the Pi on for long enough when power failure occurs to write all data to the card and shutdown safely. You could write data to cloud storage as frequently as you like and trust that they have systems to handle power loss.

If you need to do something with the data you could write it to a file in ramdisk (e.g. in /tmp directory) as often as you like and read it from there for most recent values.

SD cards, and any FLASH based storage, wears out and erases data in blocks so it's best not to do lots of small writes to it. The operating system helps out by buffering data and writing blocks at a time. If you add a few bytes to a file and force a write to FLASH the existing block containing the old version of the file will be marked as deleted and a new file will be created of the same name containing the old content plus the new bytes. At some later time the block will actually be erased and become available to be used again. If you can write whole blocks at a time you have much more efficient use of the storage and fewer erase operations means it lasts longer.

Martin Frezman
Posts: 858
Joined: Mon Oct 31, 2016 10:05 am

Re: Data is not instantly written in SD card

Mon Aug 14, 2017 9:55 am

sqline3 *is* an RDMS.
If this post appears in the wrong forums category, my apologies.

User avatar
davidcoton
Posts: 2063
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Data is not instantly written in SD card

Mon Aug 14, 2017 9:59 am

PiGraham wrote:
Mon Aug 14, 2017 9:19 am
If you need to do something with the data you could write it to a file in ramdisk (e.g. in /tmp directory) as often as you like and read it from there for most recent values.
That shouldn't be an issue. Any read operation will know about the cache, readings there will be treated the same as ones actually written to the SDCard.
But without answers to my "big picture" questions, we're tinkering on the edges of an undefined (possibly non-existent) problem.
"If it ain't broke, fix it until it is."
“Raspberry Pi is a trademark of the Raspberry Pi Foundation”
The Pink Unicorn is a symbol of commitment to sensible discussion of future versions of the Raspberry Pi ™

PiGraham
Posts: 2563
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Data is not instantly written in SD card

Mon Aug 14, 2017 10:19 am

davidcoton wrote:
Mon Aug 14, 2017 9:59 am
PiGraham wrote:
Mon Aug 14, 2017 9:19 am
If you need to do something with the data you could write it to a file in ramdisk (e.g. in /tmp directory) as often as you like and read it from there for most recent values.
That shouldn't be an issue. Any read operation will know about the cache, readings there will be treated the same as ones actually written to the SDCard.
But without answers to my "big picture" questions, we're tinkering on the edges of an undefined (possibly non-existent) problem.
Good points.

asandford
Posts: 1548
Joined: Mon Dec 31, 2012 12:54 pm
Location: Ealing

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 1:53 am

davidcoton wrote:
Mon Aug 14, 2017 8:38 am
I don't think either of these will work round the caching of disk writes. There will still be uncommitted writes in RAM.
These techniques may be appropriate but it depends on how the data is used.
I would hope that a database commit command would flush disk caches and actually commit changes to disk.

asandford
Posts: 1548
Joined: Mon Dec 31, 2012 12:54 pm
Location: Ealing

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 2:12 am

davidcoton wrote:
Mon Aug 14, 2017 9:59 am
PiGraham wrote:
Mon Aug 14, 2017 9:19 am
If you need to do something with the data you could write it to a file in ramdisk (e.g. in /tmp directory) as often as you like and read it from there for most recent values.
That shouldn't be an issue. Any read operation will know about the cache, readings there will be treated the same as ones actually written to the SDCard.
But without answers to my "big picture" questions, we're tinkering on the edges of an undefined (possibly non-existent) problem.
Using /tmp is really not a good idea *unless* you clean up after yourself and accept the fact that it is volatile. I've seen too many production machines fall over as developers use /tmp as a quick and dirty solution to whatever problem they have. The squeaky bum moment when they realise they've brought down a production system (due to filling up /tmp) and lost all the data due to a reboot usually stops them doing it again (the arrogant ones ask for a restore of /tmp which always raises a chuckle as it's never backed up).

User avatar
rpdom
Posts: 11191
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 5:40 am

asandford wrote:
Tue Aug 15, 2017 2:12 am
davidcoton wrote:
Mon Aug 14, 2017 9:59 am
PiGraham wrote:
Mon Aug 14, 2017 9:19 am
If you need to do something with the data you could write it to a file in ramdisk (e.g. in /tmp directory) as often as you like and read it from there for most recent values.
That shouldn't be an issue. Any read operation will know about the cache, readings there will be treated the same as ones actually written to the SDCard.
But without answers to my "big picture" questions, we're tinkering on the edges of an undefined (possibly non-existent) problem.
Using /tmp is really not a good idea *unless* you clean up after yourself and accept the fact that it is volatile. I've seen too many production machines fall over as developers use /tmp as a quick and dirty solution to whatever problem they have. The squeaky bum moment when they realise they've brought down a production system (due to filling up /tmp) and lost all the data due to a reboot usually stops them doing it again (the arrogant ones ask for a restore of /tmp which always raises a chuckle as it's never backed up).
Also, I don't recall seeing any Raspbian (or Debian based) system recently that has had /tmp as a ramdisk, unless specifically set that way. I seem to remember one of the Debian versions (about 10 years ago) introduced that change, but removed it again when people found the default /tmp size wasn't big enough for their use.

Ernst
Posts: 381
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 7:45 am

Baakinira wrote:
Sun Aug 13, 2017 5:42 am
My code in raspberry pi using python generates data every 10 seconds of size 0.4kB from the sensor because of the delay I've put in the code. But the data is not written in the SD CARD instantly. It waits until the data of 4kB is collected and then writes in the SD CARD which makes around 4 minutes of wait time. I actually want to collect data every 1 minute but if I insert delay of 1 minute, it takes 24 minutes to collect 4kb data which is not good. It's this the problem of raspberry pi or the sd card?
Why not increase the data size to 4Kb to give a 4Kb write ?
My first computer was an ICT1500

PiGraham
Posts: 2563
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 7:51 am

asandford wrote:
Tue Aug 15, 2017 2:12 am
Using /tmp is really not a good idea *unless* you clean up after yourself and accept the fact that it is volatile.
As davidcoton pointed out the OS disk caching handles write-read anyway and does the clean-up and flushing automatically so use of ramfs is not useful here anyway.

Hopefully Baakinira will explain what problem results from caching in this case.

jahboater
Posts: 1538
Joined: Wed Feb 04, 2015 6:38 pm

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 8:25 am

asandford wrote:
Tue Aug 15, 2017 1:53 am
I would hope that a database commit command would flush disk caches and actually commit changes to disk.
It would I am sure, but you don't need a database for something simple like that.

Code: Select all

os.fsync(fd)

    Force write of file with filedescriptor fd to disk. On Unix, this calls the native fsync() function; on Windows, the MS _commit() function.

    If you’re starting with a buffered Python file object f, first do f.flush(), and then do os.fsync(f.fileno()), to ensure that all internal buffers associated with f are written to disk.

    Availability: Unix, Windows.
https://docs.python.org/3/library/os.html

PiGraham
Posts: 2563
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 10:35 am

Until we know what the problem is it's hard to suggest a good solution, but fsync for each tiny data packet is probably a bad idea. There's a reason for the 4k size. Sync'ing to FLASH each time you have a new sensor reading will not be efficient use of storage.

fsync before shutdown to preserve data (shutdown does that anyway?).
What reasons could there be to fsync more frequently than every 24mins / 4k of data?

jahboater
Posts: 1538
Joined: Wed Feb 04, 2015 6:38 pm

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 10:44 am

PiGraham wrote:
Tue Aug 15, 2017 10:35 am
Until we know what the problem is it's hard to suggest a good solution, but fsync for each tiny data packet is probably a bad idea. There's a reason for the 4k size.
Obviously.
The point about os.fsync() is that you can choose when to force a write to the disk, and importantly, it only affects that one file.
O_SYNC passed to open is probably a bad idea for the reasons you mention, unless you buffer up the data to write 4k, or multiples of 4k, at a time - which is pretty simple.
PiGraham wrote:
Tue Aug 15, 2017 10:35 am
fsync before shutdown to preserve data (shutdown does that anyway?).
Yes it does.

PiGraham
Posts: 2563
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 10:54 am

jahboater wrote:
Tue Aug 15, 2017 10:44 am
PiGraham wrote:
Tue Aug 15, 2017 10:35 am
Until we know what the problem is it's hard to suggest a good solution, but fsync for each tiny data packet is probably a bad idea. There's a reason for the 4k size.
Obviously.
Probably not obvious to Baakinira who only has 170 bytes/min (4k/24 mins) of data coming in and, for reasons we don't yet know, wants to write to physical FLASH more frequently than is sensible.

jahboater
Posts: 1538
Joined: Wed Feb 04, 2015 6:38 pm

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 11:05 am

PiGraham wrote:
Tue Aug 15, 2017 10:54 am
Probably not obvious to Baakinira who only has 170 bytes/min (4k/24 mins) of data coming in and, for reasons we don't yet know, wants to write to physical FLASH more frequently than is sensible.
Thats 4080 bytes every 24 minutes.

how about something like this (in silly C style pseudo code!)

Code: Select all

char buf[8192];
char *ptr = buf;
char * const last = buf + 4096;

while( .... )
{
     write 170 bytes to ptr
     ptr += 170
    if( ptr > last )
    {
       os.write( fd, buf, 4096 )
       os.fsync( fd )
       memcpy( buf, last, ptr - last )
       ptr = buf + (ptr - last)
    }
}
os.write( fd, buf, ptr - buf )
In fact you could then use O_SYNC and forget the calls to fsync().

User avatar
davidcoton
Posts: 2063
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 11:11 am

jahboater wrote:
Tue Aug 15, 2017 11:05 am
PiGraham wrote:
Tue Aug 15, 2017 10:54 am
Probably not obvious to Baakinira who only has 170 bytes/min (4k/24 mins) of data coming in and, for reasons we don't yet know, wants to write to physical FLASH more frequently than is sensible.
Thats 4080 bytes every 24 minutes.

how about something like this (in silly C style pseudo code!)
WHY? What problem are you trying to solve?
Why don't we all shut up until Baakinira replies and gives enough information for our suggestions to be meaningful?
"If it ain't broke, fix it until it is."
“Raspberry Pi is a trademark of the Raspberry Pi Foundation”
The Pink Unicorn is a symbol of commitment to sensible discussion of future versions of the Raspberry Pi ™

PiGraham
Posts: 2563
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 11:13 am

jahboater wrote:
Tue Aug 15, 2017 11:05 am
PiGraham wrote:
Tue Aug 15, 2017 10:54 am
Probably not obvious to Baakinira who only has 170 bytes/min (4k/24 mins) of data coming in and, for reasons we don't yet know, wants to write to physical FLASH more frequently than is sensible.
Thats 4080 bytes every 24 minutes.

how about something like this (in silly C style pseudo code!)

Code: Select all

char buf[8192];
char *ptr = buf;
char * const last = buf + 4096;

while( .... )
{
     write 170 bytes to ptr
     ptr += 170
    if( ptr > last )
    {
       os.write( fd, buf, 4096 )
       os.fsync( fd )
       memcpy( buf, last, ptr - last )
       ptr = buf + (ptr - last)
    }
}
os.write( fd, buf, ptr - buf )
In fact you could then use O_SYNC and forget the calls to fsync().
But the OS takes care of all that anyway. Just write 170 bytes once a minute, or one byte every 3 seconds or whatever variation you like and it gets cached and the OS burns it into FLASH every 4k/24 mins.

jahboater
Posts: 1538
Joined: Wed Feb 04, 2015 6:38 pm

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 1:28 pm

PiGraham wrote:
Tue Aug 15, 2017 11:13 am
But the OS takes care of all that anyway. Just write 170 bytes once a minute, or one byte every 3 seconds or whatever variation you like and it gets cached and the OS burns it into FLASH every 4k/24 mins.
From the original question:
But the data is not written in the SD CARD instantly. It waits until the data of 4kB is collected and then writes in the SD CARD which makes around 4 minutes of wait time.
I think the Python file is buffered, so there are two stages. The Python buffer is likely being automatically flushed every time its full with 4k of data. Then the OS will physically write those blocks to the SD card at some indeterminate time in the future. Having said that, the volume of data is very low and the default commit interval is 5 seconds, so yes, you would think it should be written out regularly.

PiGraham
Posts: 2563
Joined: Fri Jun 07, 2013 12:37 pm
Location: Waterlooville

Re: Data is not instantly written in SD card

Tue Aug 15, 2017 1:49 pm

jahboater wrote:
Tue Aug 15, 2017 1:28 pm
PiGraham wrote:
Tue Aug 15, 2017 11:13 am
But the OS takes care of all that anyway. Just write 170 bytes once a minute, or one byte every 3 seconds or whatever variation you like and it gets cached and the OS burns it into FLASH every 4k/24 4mins.
From the original question:
But the data is not written in the SD CARD instantly. It waits until the data of 4kB is collected and then writes in the SD CARD which makes around 4 minutes of wait time.
I think the Python file is buffered. So that buffer is likely being automatically flushed every time its full with 4k of data. Then the OS will physically write those blocks to the SD card at some indeterminate time in the future. Having said that, the volume of data is very low and the default commit interval is 5 seconds, so yes, you would think it should be written out regularly.
The original question was effectively "how can I make the data write direct to the SD card more often the OS does it for my data?" and you did answer that, but is it a sensible thing to do? If it takes 4 minutes to get one block of data does it make any sense to write four quarter blocks at one per minute? I don't think so. It's not clear that would serve any purpose.

Does default commit interval apply to tiny packets of data? It makes more sense for larger files.

Return to “Advanced users”

Who is online

Users browsing this forum: jojopi and 9 guests