steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Root filesystem not persisting across reboots

Sun Jun 05, 2016 2:19 pm

Platform: raspberry pi 2 model B, 16GB Transcend microSDHC
raspbian "jessie" (from the Lite image)

The raspberry pi is set up as a router/firewall between the home LAN and the internet (see Using a Raspberry Pi 2 Model B as a router/firewall for the home LAN for more detail).

It worked much as I expected it for most of May, but for the last week or so, it has stopped persisting file system changes to the SD card. I.e. the system behaves normally when booted and there are no obvious error messages, but after a reboot everything created since the previous reboot is lost.

Is this behaviour known? Does it mean the SD card is hosed? Is recovery possible? What should I do? fsck? fsck followed by fstrim? Just fstrim? Something else?

I've done a bit of googling and found contradictory information, from a blogpost that advices me to put /var/log on tmpfs, to the first comment to this stackexchange which indicates that in normal use, a 16GB SD card should last approximate 27 years.

What happened was this:
  • On June 1 my mosh session to the raspberry pi startet misbehaving: it said file not found on commands I tried to run, and even failed to log out
  • I tried to log in using the console (ie. the USB keyboard and the HDMI monitor physically connected to the raspberry p), but that login failed to complete. The monitor was full of kernel error messages, mentioning ext4 (I didn't take a picture unfortunately)
  • I pulled the power and reconnected the power on the raspberry pi, and it booted normally, and seemed to behave normally
  • Then I noticed that "apt-get dist-upgrade" tried to install the same packages twice, and the third time this happened, I found that it was the reboot that caused dist-upgrade to want to install the same packages again (since one of the packages was the kernel, a reboot seemed like a natural thing)
  • I then tried to just create a file, and then rebooted, and the file was gone
So in the current state it sort of works when I don't reboot, but I can't do dist-upgrade or modify it in any way.
(which isn't optimal...)

Thanks!


- Steinar

epoch1970
Posts: 1113
Joined: Thu May 05, 2016 9:33 am

Re: Root filesystem not persisting across reboots

Sun Jun 05, 2016 3:14 pm

Your FS is hosed, so the system mounts root in read-only mode to avoid further damage.
The bad news is that a quick check at boot doesn't repair the system. You'd need to run fsck.ext4 on the partition from another machine.
Before doing that, copy essential files and try to take an image of the broken SD as second backup.
Is the FS hosed because of a defect in the SD? Likely no, but if you can repair the FS on the backup image, but not on the SD itself, then that would be a yes.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

JimmyN
Posts: 1109
Joined: Wed Mar 18, 2015 7:05 pm
Location: Virginia, USA

Re: Root filesystem not persisting across reboots

Sun Jun 05, 2016 3:18 pm

What do you get from the following two commands?

Code: Select all

mount | grep /dev/mmc

df -h

steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Re: Root filesystem not persisting across reboots

Wed Jun 08, 2016 6:18 pm

JimmyN wrote:What do you get from the following two commands?

Code: Select all

mount | grep /dev/mmc
I get this:

Code: Select all

/dev/mmcblk0p2 on / type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
JimmyN wrote:

Code: Select all

df -h
I get this:

Code: Select all

root@ocon:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  2.3G   12G  17% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           463M     0  463M   0% /dev/shm
tmpfs           463M  6.3M  457M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           463M     0  463M   0% /sys/fs/cgroup
/dev/mmcblk0p1   60M   21M   40M  35% /boot
tmpfs            93M     0   93M   0% /run/user/1001
(and /dev/root (which presumably should have been a symlink to /dev/mmcblk0p2..?) doesn't exist)

Thanks!


- Steinar

steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Re: Root filesystem not persisting across reboots

Wed Jun 08, 2016 7:23 pm

epoch1970 wrote:Your FS is hosed, so the system mounts root in read-only mode to avoid further damage.
Thanks! I thought it might be something like this (I have seen the described behaviour on "regular" GNU/linux systems with spinning disks in the process of going bad).

(What threw me was that the fs isn't read-only, just non-persistent. I can create files without problems, but they are gone after a reboot)
epoch1970 wrote:The bad news is that a quick check at boot doesn't repair the system. You'd need to run fsck.ext4 on the partition from another machine.
Ok, that's a way of doing it? I was trying to find out how to get the rPi itself into single user mode. Thanks for the heads up.
epoch1970 wrote:Before doing that, copy essential files and try to take an image of the broken SD as second backup.
The essential files are already copied out (I have RCS-versioned the config files under /etc and have tar.gz containing the RCS files under /etc, so basically: untar the backup, install the extra packages, revert the config and I'm back to where I was... the only nuisance is that I will be without a Home LAN router (and the rest of the family without a network connection) while setting up the new sd card, so I'm delaying it as long as I can).

I will take also make an image of the broken SD as suggested, and a lesson for next time: the minute I have the system up and running as desired: dd the image and then use GParted (or something) to shrink it.
epoch1970 wrote:Is the FS hosed because of a defect in the SD? Likely no, but if you can repair the FS on the backup image, but not on the SD itself, then that would be a yes.
Ah, so recovery is possible? Good to know! I will try the fsck.ext4 approach with the sd card plugged into a card reader on a debian computer.

Another data-point, here's a gist containing dmesg after a reboot: https://gist.github.com/steinarb/2df343 ... 96ade5ed5b

Note the messages at the end:

Code: Select all

[ 1391.870183] mmc0: tried to reset card
[ 1391.870860] error 0 requesting status 0x4000900
[ 1391.870878] blk_update_request: I/O error, dev mmcblk0, sector 675016
[ 1391.879726] error 0 requesting status 0x4000900
[ 1391.879747] blk_update_request: I/O error, dev mmcblk0, sector 675352
[ 1391.881578] error 0 requesting status 0x4000900
[ 1391.881594] blk_update_request: I/O error, dev mmcblk0, sector 675376
[ 1423.872165] error 0 requesting status 0x4000900
[ 1423.872192] blk_update_request: I/O error, dev mmcblk0, sector 675048
[ 7366.192097] error 0 requesting status 0x4000900
[ 7366.192125] blk_update_request: I/O error, dev mmcblk0, sector 3827536
[ 7366.201463] error 0 requesting status 0x4000900
[ 7366.201484] blk_update_request: I/O error, dev mmcblk0, sector 3827888
[ 7396.638362] error 0 requesting status 0x4000900
[ 7396.638388] blk_update_request: I/O error, dev mmcblk0, sector 3827592
[ 8885.776518] error 0 requesting status 0x4000900
[ 8885.776545] blk_update_request: I/O error, dev mmcblk0, sector 3833608
[ 8885.781189] error 0 requesting status 0x4000900
[ 8885.781210] blk_update_request: I/O error, dev mmcblk0, sector 3833760
[ 8920.679522] error 0 requesting status 0x4000900
[ 8920.679558] blk_update_request: I/O error, dev mmcblk0, sector 3833408
[ 8920.681733] error 0 requesting status 0x4000900
[ 8920.681761] blk_update_request: I/O error, dev mmcblk0, sector 3833752
[23266.265249] error 0 requesting status 0x4000900
[23266.265279] blk_update_request: I/O error, dev mmcblk0, sector 6160384
[23266.270294] error 0 requesting status 0x4000900
[23266.270315] blk_update_request: I/O error, dev mmcblk0, sector 6160552
[23266.454746] error 0 requesting status 0x4000900
[23266.454775] blk_update_request: I/O error, dev mmcblk0, sector 6167904
[23296.795326] error 0 requesting status 0x4000900
[23296.795353] blk_update_request: I/O error, dev mmcblk0, sector 6160432
[23296.862773] error 0 requesting status 0x4000900
[23296.862799] blk_update_request: I/O error, dev mmcblk0, sector 6167552
[23296.866613] error 0 requesting status 0x4000900
[23296.866628] blk_update_request: I/O error, dev mmcblk0, sector 6167680
Googling the error message "mmc0: tried to reset card" found me this: http://askubuntu.com/a/277867 which told me the reason was that the card had been pulled without an unmount, and I yanked the power, which may be similar.

I tried this approach to get fsck, that google found me:

Code: Select all

touch /forcefsck
sync; reboot
and after the reboot the error messages at the end of dmesg disappeared but the fs was still non-persistent, and df still listed / as being mounted on /dev/root which still didn't exist.

(so it could be that fsck.ext4 on the debian machine won't help, but I can't see it doing any harm either)

epoch1970
Posts: 1113
Joined: Thu May 05, 2016 9:33 am

Re: Root filesystem not persisting across reboots

Wed Jun 08, 2016 8:38 pm

Did I guarantee recovery of something? I think you were wise to take a backup.

Looking at your dmesg log, I saw these:
1) The SD doesn't seem totally dead:
[ 2.900957] mmc0: new high speed SDHC card at address 59b4
[ 2.901920] mmcblk0: mmc0:59b4 USDU1 15.0 GiB
[ 2.903343] mmcblk0: p1 p2

2) Root gets cleaned up from "orphaned files" (could be your new files...)
[ 2.974347] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
[ 2.984877] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
...
[ 3.476885] EXT4-fs (mmcblk0p2): orphan cleanup on readonly fs
[ 3.493591] EXT4-fs (mmcblk0p2): 1 orphan inode deleted
[ 3.500597] EXT4-fs (mmcblk0p2): recovery complete
[ 3.523647] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.

3) Root remounts, but is it rw at that time? Partition /boot is damaged too. It's a FAT partition, you can repair if from any PC or Mac.
[ 6.774150] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 7.047066] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

I did not suggest writing /forcefsck to the root, since you said files would not persist (which I understand as: root is mounted ro).
I hope repairing *both partitions* from another linux comp will yield good results.

(and I have not killed an SD yet, but I have killed and USB key simply by removing power. AFAIK if you do that during a write cycle, the thing dies, completely. Perhaps µSDs are more resilient.)
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Re: Root filesystem not persisting across reboots

Wed Jun 08, 2016 10:24 pm

epoch1970 wrote:Did I guarantee recovery of something? I think you were wise to take a backup.
Sorry for the misunderstanding, by "Recovery is possible?", I meant to say "Recovery may be possible".
epoch1970 wrote:Looking at your dmesg log, I saw these:
1) The SD doesn't seem totally dead:
[ 2.900957] mmc0: new high speed SDHC card at address 59b4
[ 2.901920] mmcblk0: mmc0:59b4 USDU1 15.0 GiB
[ 2.903343] mmcblk0: p1 p2
That's good.
epoch1970 wrote:2) Root gets cleaned up from "orphaned files" (could be your new files...)
Right.
(I should have searched the entire dmesg for "mmc", not just seen the messages at the end and assumed that was all sd releated. My bad!)
epoch1970 wrote:3) Root remounts, but is it rw at that time? Partition /boot is damaged too. It's a FAT partition, you can repair if from any PC or Mac.
[ 6.774150] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 7.047066] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Right!
(should have searched the dmesg output for "fsck" as well)
epoch1970 wrote:I did not suggest writing /forcefsck to the root, since you said files would not persist (which I understand as: root is mounted ro).
Right, it shouldn't have been persisted. I didn't think of that.
So maybe it was just coincidence that the error messages were gone from the dmesg on the next reboot?
epoch1970 wrote:I hope repairing *both partitions* from another linux comp will yield good results.
Ok, the plan for now, is:
  1. Plug the sd card into the debian computer
  2. dd the sd to an image
  3. try repairing both partitions with fsck
  4. Plug the sd card back in and see if it boots
  5. If it boots, make a file and reboot, and check if it is still present
  6. Make a new image from an sd I know works
  7. Shrink the image using GParted (or something)
  8. Put the new image on a different sd and see if it boots and everything works
  9. Put the original sd back in and let it run
If it fails somewhere in there, I will most probably do a new installation starting with a Raspbian Lite image.
And this time: make an image as soon as I know I have it up and working.

Thanks again for your help!


- Steinar

steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Re: Root filesystem not persisting across reboots

Sat Jun 11, 2016 3:54 pm

I didn't have much success with fsck, that is: at the end of the day the rPi is up and running with the same unchangable file system as before.

Here's what I did:
  1. Power down the rPi with "sync; halt" and then pull the power
  2. Eject the sd card and put it into a card reader on a debian computer
  3. Type "mount" to find the devices for the sd card partitions

    Code: Select all

    /dev/sdd1 on /media/sb/boot type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=utf8,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)
    /dev/sdd2 on /media/sb/a045dc3a-fdf5-408d-aa6b-7c9c23cfef4d type ext4 (rw,nosuid,nodev,relatime,data=ordered,uhelper=udisks2)
  4. Unmount the partitions and make a copy of the SD-card to a file:

    Code: Select all

    root@lorenzo:~# umount /media/sb/boot/
    root@lorenzo:~# umount /media/sb/a045dc3a-fdf5-408d-aa6b-7c9c23cfef4d/
    root@lorenzo:~# dd bs=4M if=/dev/sdd of=/home/sb/Downloads/ocon-image-backup-foer-fsck.img
    3835+1 records in
    3835+1 records out
    16088301568 bytes (16 GB) copied, 887.084 s, 18.1 MB/s
    root@lorenzo:~# ls -al /home/sb/Downloads/ocon-image-backup-foer-fsck.img 
    -rw-r--r-- 1 root root 16088301568 Jun 11 15:27 /home/sb/Downloads/ocon-image-backup-foer-fsck.img
    root@lorenzo:~# 
  5. Did an fsck on the VFAT partition (which was /dev/sdd1 in the card reader), but didn't find much interesting. Full output here: https://gist.github.com/steinarb/875da3 ... fa312ee0d0
  6. Tried fsck on the ext4 partition (/dev/sdd2 in the card reader), but that wasn't successful:

    Code: Select all

    root@lorenzo:~# fsck /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Run journal anyway<y>? yes
    fsck.ext4: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
    root@lorenzo:~# fsck /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Run journal anyway<y>? yes
    fsck.ext4: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
  7. Tried clearing the journal flag (and making the fs ext2) to avoid the problem with restoring the journal, but that wasn't successful either:

    Code: Select all

    root@lorenzo:~# tune2fs -O ^has_journal /dev/sdd2 
    tune2fs 1.42.12 (29-Aug-2014)
    The needs_recovery flag is set.  Please run e2fsck before clearing
    the has_journal flag.
    root@lorenzo:~# e2fsck -fy /dev/sdd2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Run journal anyway? yes
    
    e2fsck: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
  8. Tried Tried deleting the needs_recovery flag and then run fsck, but failed:

    Code: Select all

    root@lorenzo:~# debugfs -w /dev/sdd2
    debugfs 1.42.12 (29-Aug-2014)
    debugfs:  features ^needs_recovery
    Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file uninit_bg dir_nlink extra_isize
    debugfs:  quit
    root@lorenzo:~# fsck -n /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    Warning: skipping journal recovery because doing a read-only filesystem check.
    /dev/sdd2: clean, 105496/927360 files, 583127/3911424 blocks
    root@lorenzo:~# fsck -y /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Run journal anyway? yes
    
    fsck.ext4: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
    
    root@lorenzo:~# 
    (I see here that I perhaps should have dropped the -y flag and said not use the journal...?)
  9. Tried using a backup superblock:

    Code: Select all

    mke2fs 1.42.12 (29-Aug-2014)
    /dev/sdd2 contains a ext4 file system
            last mounted on / on Sat May 21 08:43:51 2016
    Proceed anyway? (y,n) y
    Creating filesystem with 3911424 4k blocks and 979200 inodes
    Filesystem UUID: a004071d-7f65-4c1d-8776-d666d2e7c998
    Superblock backups stored on blocks: 
            32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208
    
    root@lorenzo:~# fsck -b 163840 -B 4096 /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Recovery flag not set in backup superblock, so running journal anyway.
    fsck.ext4: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ***** FILE SYSTEM WAS MODIFIED *****
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
    
    root@lorenzo:~# fsck -y -b 163840 -B 4096 /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Recovery flag not set in backup superblock, so running journal anyway.
    fsck.ext4: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ***** FILE SYSTEM WAS MODIFIED *****
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
  10. Tried creating a new journal, failed because a journal already exists:

    Code: Select all

    root@lorenzo:~# tune2fs -j /dev/sdd2
    tune2fs 1.42.12 (29-Aug-2014)
    The filesystem already has a journal.
  11. Put the sd card rPi and it rebooted fine, but still had the same issues with an unchangable file system, from dmesg:

    Code: Select all

    ...
    [    2.934917] mmc0: new high speed SDHC card at address 59b4
    [    2.942631] mmcblk0: mmc0:59b4 USDU1 15.0 GiB 
    [    2.954075]  mmcblk0: p1 p2
    [    3.061962] usb 1-1: new high-speed USB device number 2 using dwc_otg
    [    3.065001] EXT4-fs (mmcblk0p2): INFO: recovery required on readonly filesystem
    [    3.065008] EXT4-fs (mmcblk0p2): write access will be enabled during recovery
    [    3.091204] Indeed it is in host mode hprt0 = 00001101
    [    3.292333] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
    [    3.302582] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
    [    3.314227] hub 1-1:1.0: USB hub found
    [    3.319887] hub 1-1:1.0: 5 ports detected
    [    3.552124] EXT4-fs (mmcblk0p2): orphan cleanup on readonly fs
    [    3.568865] EXT4-fs (mmcblk0p2): 1 orphan inode deleted
    [    3.576003] EXT4-fs (mmcblk0p2): recovery complete
    [    3.587324] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)
    [    3.599126] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
    ...
The way forward:
  1. Try running fsck on the image I took to see if it can be recovered (if so, then the problem is with the sd card)
  2. Start work on a new installation on a new sd card (regardless of what I find above, I won't really trust either the card or the fs on it), and as soon as I've got it working, make an image from it

epoch1970
Posts: 1113
Joined: Thu May 05, 2016 9:33 am

Re: Root filesystem not persisting across reboots

Sat Jun 11, 2016 4:28 pm

This is not exactly up my alley, but is it possible you should have removed the need_recovery flag *before* getting rid of the journal?
Source: untested blog post about an EXT3 issue that looks like yours.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Re: Root filesystem not persisting across reboots

Tue Jun 14, 2016 6:37 pm

epoch1970 wrote:This is not exactly up my alley, but is it possible you should have removed the need_recovery flag *before* getting rid of the journal?
Source: untested blog post about an EXT3 issue that looks like yours.
Perhaps...?

But I just got around to running fsck on the ext4 partition of the image and that was so trouble-free that I think it is the sd card that is hosed (and I'm wondering if it was ntop that wrote the SD card to death the month the rPi was running until it started having problems...?).

Here's what I did:
  1. Mounted the image as a loopback device on a debian jessie computer

    Code: Select all

    root@lorenzo:~# modprobe loop
    root@lorenzo:~# losetup -f
    /dev/loop0
    root@lorenzo:~# losetup /dev/loop0 /home/sb/Downloads/ocon-image-copy.img 
    root@lorenzo:~# partprobe /dev/loop0
    root@lorenzo:~# ls -al /dev/loop0*
    brw-rw---- 1 root disk   7, 0 Jun 14 20:13 /dev/loop0
    brw-rw---- 1 root disk 259, 0 Jun 14 20:13 /dev/loop0p1
    brw-rw---- 1 root disk 259, 1 Jun 14 20:13 /dev/loop0p2
  2. fsck'd the VFAT partition, which was as eventful and with approximately the same output as the VFAT partition on the sd card, (ie. https://gist.github.com/steinarb/875da3 ... fa312ee0d0 (which is the output from the SD card, I didn't bother to make a new gist of this))
  3. fsck'd the ext4 partition, and it was quick and problem free, and used the journal

    Code: Select all

    root@lorenzo:~# fsck /dev/loop0p2
    fsck from util-linux 2.25.2
    e2fsck 1.43 (17-May-2016)
    /dev/loop0p2: recovering journal
    Clearing orphaned inode 17325 (uid=0, gid=0, mode=0100644, size=178908)
    Setting free inodes count to 781411 (was 821865)
    Setting free blocks count to 3269746 (was 3328341)
    /dev/loop0p2: clean, 145949/927360 files, 641678/3911424 blocks
Going forward I think I will try creating a new sd card from the fsck'd image and see if it will boot and run normally.

But I also think I need to to something about ntop's writing, and that would be either uninstalling ntop or moving to a persistent RAM disk.

epoch1970
Posts: 1113
Joined: Thu May 05, 2016 9:33 am

Re: Root filesystem not persisting across reboots

Tue Jun 14, 2016 9:04 pm

Interesting, and hopefully cool.

TBH, most of what I've seen using the error message as search keywords ended with "forget it. drive was dead."
And running a packet capture application on an SD does seem bold :)
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

RoundDuckMan
Posts: 18
Joined: Thu Nov 06, 2014 4:02 am

Re: Root filesystem not persisting across reboots

Wed Jun 15, 2016 3:44 am

I might be having a similar issue, on some reboots Raspbian corrupts for no apparent reason. I'm right now testing another SD card, but It probably not go too well either. Maybe systemd is bugging out again, considering it's a buggy piece of crap on PCs... :roll:

beta-tester
Posts: 1129
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: Root filesystem not persisting across reboots

Wed Jun 15, 2016 6:48 am

steinarbang wrote:I didn't have much success with fsck, that is: at the end of the day the rPi is up and running with the same unchangable file system as before.

Here's what I did:
  • 6. Tried fsck on the ext4 partition (/dev/sdd2 in the card reader), but that wasn't successful:

    Code: Select all

    root@lorenzo:~# fsck /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Run journal anyway<y>? yes
    fsck.ext4: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
    root@lorenzo:~# fsck /dev/sdd2
    fsck from util-linux 2.25.2
    e2fsck 1.42.12 (29-Aug-2014)
    /dev/sdd2: recovering journal
    Superblock needs_recovery flag is clear, but journal has data.
    Run journal anyway<y>? yes
    fsck.ext4: unable to set superblock flags on /dev/sdd2
    
    
    /dev/sdd2: ********** WARNING: Filesystem still has errors **********
does fsck repair the fs, when there is no parameter given?
i thought without a parameter, fsck will only report issues, without touching the fs.
man fsck wrote:NAME
  • fsck - check and repair a Linux filesystem
SYNOPSIS
  • fsck [-lsAVRTMNP] [-r [fd]] [-C [fd]] [-t fstype] [filesystem...] [--]
    [fs-specific-options]
DESCRIPTION
  • fsck is used to check and optionally repair one or more Linux filesys‐
    tems.
...
i guess you have to tell fsck explicitly to repair the fs.
man fsck wrote: Options to different filesystem-specific fsck's are not standardized.
If in doubt, please consult the man pages of the filesystem-specific
checker. Although not guaranteed, the following options are supported
by most filesystem checkers:
...
  • [-r] Interactively repair the filesystem (ask for confirmations).
...
for example, for interactive repair, try:

Code: Select all

fsck -r /dev/sdd2
{ I only give negative feedback }
RPi Model B (rev1, 256MB) & B (rev2, 512MB) & B+, RPi2B (1GB), 64GB microSDXC1 class 10, HDMI 1920x1080, keyboard-mouse-combo (wireless), PiCamera, ethernet-cable, 5V/1.2A power supply, Wifi dongle (rt5370)

steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Re: Root filesystem not persisting across reboots

Sun Jul 03, 2016 6:37 am

epoch1970 wrote:Interesting, and hopefully cool.

TBH, most of what I've seen using the error message as search keywords ended with "forget it. drive was dead."
And running a packet capture application on an SD does seem bold :)
In my defense: I had no idea at that time how often it would actually write to disk. As it turns out: quite often, and to quite a lot of files (the top talker files), and with /var/tmp/ntopng using half a gig at the end of the month when the sd card packed it in.

Logging to /var/log probably took its toll as well, because some DHCP clients kept requesting confirmation of their address, no matter how long a lease the DHCP server gave them. The /var/log/daemon.log and /var/log/syslog files had a high write rate.

Anyway, now the rPi is up and running again, with the fsck'd image burnt to a new card, ntopng removed, and /var/log put on "persistent tmpfs". Hopefully this time it will run for more than a month before the SD card starting to fail. :-)

And if the new SD card fails, this time I have saved an image of the working sd card (and have a spare sd card ready to go in a drawer next to the rPi), so recovery should be quick.

Lesson learned!

davidtuti
Posts: 83
Joined: Tue Oct 22, 2013 6:21 am

Re: Root filesystem not persisting across reboots

Sat Aug 06, 2016 11:50 pm

Hi
I have the same problem. It gives me error when I do fsck and when I create a file in sd card, when I reboot the file is lost.
Is possible to create a image with dd and restore in same dd without errors?
Thanks and sorry for my English!

steinarbang
Posts: 8
Joined: Sun Jun 05, 2016 8:56 am

Re: Root filesystem not persisting across reboots

Sun Aug 07, 2016 9:28 am

In my case the card itself was ruined, but an image created from the card using dd could be fixed with fsck and used with dd to set up a brand new card. The rPi is currently running happily with the new card.

You can try the same thing and it may work for you as well. But there are now guarantees.

Return to “Troubleshooting”

Who is online

Users browsing this forum: No registered users and 73 guests