Heater
Posts: 14250
Joined: Tue Jul 17, 2012 3:02 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 3:00 am

OK, now we are talking. Here is the plan:

1) Calculate the md5sum of data you want to write to a file.

2) Write the file as normal. But use the md5sum as the file name.

3) Just so you know where your data is perhaps put that in a directory with your original file name like: /home/pi/myFile/3f2aaae888c6028c2e2683d6547963d4

4) Drop caches as stated above to ensure everything is committed to the file. Nothing left in RAM

5) When it's time to read the file, recalculate the md5sum from the data read and check it against the file name.

6) You could of course do step 5) after every update to the file just to be sure it is intact on the file system.

There we go. A robust way to maintain the file with error checking.

But wait, we already have a system that maintains checksums on files like that. It's git. The version control system. So we can use git to do all the above for us, except perhaps the cache dropping part.

Guess what? Someone has already create gitfs, the git file system, to do just that: https://github.com/PressLabs/gitfs

Or use git directly from your program to save your files: https://github.com/nodegit/nodegit
Memory in C++ is a leaky abstraction .

Francois117
Posts: 11
Joined: Thu Apr 07, 2016 11:26 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 7:46 am

jojopi wrote:
Conjur wrote:in theory; if the SD card is not retaining data, it will NOT retain data through umounting and mounting.
If the data are only being retained in the kernel's buffer cache and not on the card, just dropping the caches should be sufficient to make recent changes disappear.

Code: Select all

touch TESTFILE
stat TESTFILE
sync
echo 3 |sudo tee /proc/sys/vm/drop_caches
stat TESTFILE
--tested - the two stats show exactly the same. after reboot TESTFILE is gone from the bad card.

conjor - I also tried your mount/unmount solution but get this error on a good SD card and on the bad one:
mount: special device /dev/sda4 does not exist

User avatar
RaTTuS
Posts: 10514
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 7:56 am

whip out the Sdcard ,
slap it into a windows machine
sdformat it to fat [or use a mobile phone / camera]

use
ftp://ftp.heise.de/pub/ct/ctsi/h2testw_1.4.zip

and test the sdcard
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

Heater
Posts: 14250
Joined: Tue Jul 17, 2012 3:02 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 8:15 am

Francois117,

Whip out the bad SD card. And throw it in the bin. Preferably do some physical damage to it first so that it is sure not to bother anyone again.

At least that is my approach today. After having so many hassles with SD cards in the past if they show the slightest sign of failure they are for the chop.
Memory in C++ is a leaky abstraction .

Francois117
Posts: 11
Joined: Thu Apr 07, 2016 11:26 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 3:00 pm

I agree... I really want to trash it but I am developing a project on this hardware that will run in the field for many years. I need to know when that SD card goes bad - without knowing, the device will operate in a semi-working condition for a long time (until the ram fills up) risking back-end database corruption.

I have developed a php-based system of writing the date to a file, rebooting at 1am, reading the file and comparing it to the real date... It' did not test properly this morning so it is still work in progress.

In the end I hope to have a system of detecting this kind of error and being able to report it to our back-end servers.

Thanks for all your input.

Heater
Posts: 14250
Joined: Tue Jul 17, 2012 3:02 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 3:07 pm

Francois117,
...I am developing a project on this hardware that will run in the field for many years...
In that case be sure to have your root file system, the SD card, mounted as read only. If the card can not be written to it cannot be corrupted and cause your system to not reboot after a power failure.

There are hints and tips on arranging a read-only root file system here: https://wiki.debian.org/ReadonlyRoot

Write your data to a USB stick. Or better still send it off the Pi over the net to some server somewhere if possible.
Memory in C++ is a leaky abstraction .

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 3:35 pm

Heater wrote:Write your data to a USB stick.
It does seem wise to separate system files from data and everything from any swap when using flash devices. Of course swap shouldn't be needed in a data collecting device that has a half to one gig of RAM.

Heater
Posts: 14250
Joined: Tue Jul 17, 2012 3:02 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 13, 2016 4:20 pm

Oh yeah, never use swap. For a dedicated, embedded system swap is probably not required. I have only needed it when doing monster jobs like compiling the Qt libraries or trying to run far to many things at the same time.
Memory in C++ is a leaky abstraction .

tpylkko
Posts: 384
Joined: Tue Oct 14, 2014 5:21 pm

Re: filesystem write files - after reboot the old data is ba

Wed Apr 20, 2016 8:41 am

I wonder if those solutions would work, as from what I've read the cards only start dumping and using cache once they are out of real space. So if you bought a 32 GB card that was in reality a 4 GB factory reject (but telling the kernel that it is 32 GB), then you would need to fill the 4 GB at least at first, wouldn't you?

Hiradur
Posts: 96
Joined: Fri Mar 01, 2013 10:59 am

Re: filesystem write files - after reboot the old data is ba

Thu Apr 21, 2016 8:36 am

Francois117 wrote:Sooo... Three days later.
I managed to find an older version of my development SD card and cloned it onto the SD card of the system that was showing this weird non-write error. And guess what - after the dd, the SD card still had all the old data! Like the dd did not happen!

Somehow it fools the ubuntu 14 OS the think that the writes are succesful, and comitted to SD card disk. But after a reboot, the SD card is back to the state it was in before any writes!

BEWARE: SD card fail in weird ways!
I had the exact same issue with a Transcend 8GB card from Amazon just like yours after running it 24/7 for ~10 months with Debian 7 on an Odroid U2. All writes looked like they were processed but in fact went into oblivion. I realized it when more and more software segfaulted for no apparent reason. Since then I've switched to a SanDisk 16GB card which is still working fine with an uptime of 466 days.

By the way: unexpected power loss WILL cause file system corruption sooner or later unless perhaps you are using a journaling filesystem (not 100% safe either though).

Return to “General discussion”