These systems presently use a Pi 2 with a micro SDHC (consumer grade Samsung 16G), so there is no lock switch on the card.
One sure fire way to know when this happens:
- touch a new file, confirm it exists
- the new file doesn't exist
Another symptom is that a rotating log file will have a start date of a long time ago, the newest entries are current, and there's a large date skip somewhere in the middle. The newest entries are from memory and a power cycle loses them.
I'm trying to find a way to tell when it happens without rebooting, and without having to wait days for the log timestamp side effect to be detectable. Something scriptable.
The file system hasn't exactly gone read-only. I can create files. /proc/mount shows everything as rw, so looking there isn't informative.
I don't think the SD card has bad sectors, because if I run "badblocks -v /dev/mmcblk0" it returns 0 errors. Also I can write a new image to an afflicted SD card and it works fine after that. (is rw)
I tried using dd with the understanding that it writes directly to disk, skipping caching, but it all seems normal:
.. though after a reboot, abc.bin and xyz.bin are gone.
Code: Select all
$ sudo dd bs=4096 count=1 skip=200000 if=/dev/mmcblk0 of=xyz.bin 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00329162 s, 1.2 MB/s $ sudo dd bs=4096 count=1 seek=200000 if=/dev/urandom of=/dev/mmcblk0 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00170464 s, 2.4 MB/s $ sudo dd bs=4096 count=1 skip=200000 if=/dev/mmcblk0 of=abc.bin 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00352715 s, 1.2 MB/s $ sudo dd bs=4096 count=1 seek=200000 if=xyz.bin of=/dev/mmcblk0 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.003651 s, 1.1 MB/s $ cmp xyz.bin abc.bin xyz.bin abc.bin differ: byte 1, line 1
One one afflicted card, df says /root is 27% used, and /boot is 33% used. Everything else is either 0, 1, or 2% used. This is typical.
Intentionally making the SD card read-only isn't an option. The application uses a sqlite DB and the application log files are also needed. I'm considering making the SD card read-only and using a mounted USB drive for the writable stuff, but without knowing the root cause I can't be sure that would really help.
Can anyone shed some light on what might be going on, or suggest other things I could try?