Tools (programs) for dealing with superblocks?

[ab1jx]

They aren't fun, but I just got this
[quote]
e2fsck: unable to set superblock flags on /dev/sda2
[quote]
that's after about half an hour of figuring out the block size and the block numbers where my superblocks are. Then in dmesg I see:

Code: Select all

[30665.306564] EXT4-fs error (device sda2): __ext4_get_inode_loc:4072: inode #6843215: block 27264724: comm mc: unable to read itable block
[30728.245726] EXT4-fs (sda2): error count since last fsck: 8
[30728.245756] EXT4-fs (sda2): initial error at time 1497004049: ext4_mb_generate_buddy:758
[30728.245768] EXT4-fs (sda2): last error at time 1497049827: __ext4_get_inode_loc:4072: inode 6843215: block 27264724

And sure enough, those times are today.

But what exactly is wrong with inode 6843215? And why can't the superblock flags be set, a bit that won't toggle or is this still read-only for some reason? It's on a Sandisk SD card about a year old, I got up this morning and suddenly the whole file system is read-only.

And for all the times I've run fsck using alternate superblocks I have to wonder why it's such a pain finding them. And why I'm guessing which one might be good? I mean, I've got:

Code: Select all

  Primary superblock at 0, Group descriptors at 1-8
  Backup superblock at 32768, Group descriptors at 32769-32776
  Backup superblock at 98304, Group descriptors at 98305-98312
  Backup superblock at 163840, Group descriptors at 163841-163848
  Backup superblock at 229376, Group descriptors at 229377-229384
  Backup superblock at 294912, Group descriptors at 294913-294920
  Backup superblock at 819200, Group descriptors at 819201-819208
  Backup superblock at 884736, Group descriptors at 884737-884744
  Backup superblock at 1605632, Group descriptors at 1605633-1605640
  Backup superblock at 2654208, Group descriptors at 2654209-2654216
  Backup superblock at 4096000, Group descriptors at 4096001-4096008
  Backup superblock at 7962624, Group descriptors at 7962625-7962632
  Backup superblock at 11239424, Group descriptors at 11239425-11239432
  Backup superblock at 20480000, Group descriptors at 20480001-20480008
  Backup superblock at 23887872, Group descriptors at 23887873-23887880

Most of them are probably OK, but why am I guessing? How come there isn't a program to find them all and compare them, tell me which ones are bad? It seems like it could be done if you understood the format. Some numbers probably change depending on where in the partition the superblock is, but most probably don't. And if you can find half a dozen that agree, they're probably right. Maybe it's like fixing a roof when it isn't raining, it just gets forgotten about.

[jojopi]

EXT4-fs error (device sda2): __ext4_get_inode_loc:4072: inode #6843215: block 27264724: comm mc: unable to read itable block

The inode table block that contains the inode could not be read. This is not really a filesystem error. I would expect to see some nearby errors from the underlying block device driver, but you have not included any.

No amount of messing with the filesystem will fix a problem with the SD card, card reader, USB connection, etc.

[ab1jx]

EXT4-fs error (device sda2): __ext4_get_inode_loc:4072: inode #6843215: block 27264724: comm mc: unable to read itable block

The inode table block that contains the inode could not be read. This is not really a filesystem error. I would expect to see some nearby errors from the underlying block device driver, but you have not included any.

Maybe:

Code: Select all

[30600.215744] [<805b7644>] (__schedule) from [<805b7bbc>] (schedule+0x50/0xa8)
[30600.215756] [<805b7bbc>] (schedule) from [<805baab8>] (schedule_timeout+0x1e8/0x270)
[30600.215769] [<805baab8>] (schedule_timeout) from [<805b73e0>] (io_schedule_timeout+0xc0/0x138)
[30600.215780] [<805b73e0>] (io_schedule_timeout) from [<805b8494>] (bit_wait_io+0x20/0x6c)
[30600.215792] [<805b8494>] (bit_wait_io) from [<805b80d4>] (__wait_on_bit+0x94/0xd0)
[30600.215802] [<805b80d4>] (__wait_on_bit) from [<805b818c>] (out_of_line_wait_on_bit+0x7c/0x84)
[30600.215815] [<805b818c>] (out_of_line_wait_on_bit) from [<8018d404>] (__wait_on_buffer+0x40/0x44)
[30600.215829] [<8018d404>] (__wait_on_buffer) from [<801e3a64>] (__ext4_get_inode_loc+0x4dc/0x4f8)
[30600.215842] [<801e3a64>] (__ext4_get_inode_loc) from [<801e5fd0>] (ext4_iget+0x6c/0x9ac)
[30600.215852] [<801e5fd0>] (ext4_iget) from [<801e6944>] (ext4_iget_normal+0x34/0x38)
[30600.215865] [<801e6944>] (ext4_iget_normal) from [<801f0660>] (ext4_lookup+0xb0/0x16c)
[30600.215877] [<801f0660>] (ext4_lookup) from [<80161494>] (lookup_real+0x30/0x5c)
[30600.215888] [<80161494>] (lookup_real) from [<80162740>] (__lookup_hash+0x48/0x50)
[30600.215902] [<80162740>] (__lookup_hash) from [<801647c4>] (walk_component+0x230/0x314)
[30600.215913] [<801647c4>] (walk_component) from [<80164e48>] (path_lookupat+0x70/0x114)
[30600.215925] [<80164e48>] (path_lookupat) from [<80166988>] (filename_lookup+0x9c/0xf8)
[30600.215940] [<80166988>] (filename_lookup) from [<80166acc>] (user_path_at_empty+0x50/0x58)
[30600.215951] [<80166acc>] (user_path_at_empty) from [<8015c6e0>] (vfs_fstatat+0x5c/0xb0)
[30600.215961] [<8015c6e0>] (vfs_fstatat) from [<8015c788>] (vfs_lstat+0x28/0x2c)
[30600.215970] [<8015c788>] (vfs_lstat) from [<8015cf08>] (SyS_lstat64+0x24/0x40)
[30600.215983] [<8015cf08>] (SyS_lstat64) from [<8000fb40>] (ret_fast_syscall+0x0/0x1c)
[30600.216004] INFO: task jbd2/sda2-8:7553 blocked for more than 120 seconds.
[30600.216010]       Not tainted 4.4.34-v7+ #930
[30600.216014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[30600.216019] jbd2/sda2-8     D 805b7644     0  7553      2 0x00000000
[30600.216036] [<805b7644>] (__schedule) from [<805b7bbc>] (schedule+0x50/0xa8)
[30600.216046] [<805b7bbc>] (schedule) from [<805baab8>] (schedule_timeout+0x1e8/0x270)
[30600.216058] [<805baab8>] (schedule_timeout) from [<805b73e0>] (io_schedule_timeout+0xc0/0x138)
[30600.216069] [<805b73e0>] (io_schedule_timeout) from [<805b8494>] (bit_wait_io+0x20/0x6c)
[30600.216079] [<805b8494>] (bit_wait_io) from [<805b80d4>] (__wait_on_bit+0x94/0xd0)
[30600.216089] [<805b80d4>] (__wait_on_bit) from [<805b818c>] (out_of_line_wait_on_bit+0x7c/0x84)
[30600.216099] [<805b818c>] (out_of_line_wait_on_bit) from [<8018d404>] (__wait_on_buffer+0x40/0x44)
[30600.216112] [<8018d404>] (__wait_on_buffer) from [<8023c38c>] (jbd2_write_superblock+0x18c/0x23c)
[30600.216125] [<8023c38c>] (jbd2_write_superblock) from [<8023cf4c>] (jbd2_journal_u[30600.216139] [<8023cf4c>] (jbd2_journal_update_sb_log_tail) from [<802346e4>] (jbd2_journal_commit_transaction+0x258/0x17f8)
[30600.216152] [<802346e4>] (jbd2_journal_commit_transaction) from [<8023b5d8>] (kjournald2+0xf4/0x2a4)
[30600.216164] [<8023b5d8>] (kjournald2) from [<80042860>] (kthread+0xec/0x104)
[30600.216176] [<80042860>] (kthread) from [<8000fbc8>] (ret_from_fork+0x14/0x2c)
[30603.205487] usb 1-1.4.2: reset high-speed USB device number 12 using dwc_otg
[30634.175576] usb 1-1.4.2: reset high-speed USB device number 12 using dwc_otg
[30665.205638] usb 1-1.4.2: reset high-speed USB device number 12 using dwc_otg
[30665.306434] sd 2:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x05 driverbyte=0x00
[30665.306453] sd 2:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 0d 02 36 20 00 00 e0 00
[30665.306463] blk_update_request: I/O error, dev sda, sector 218248736
[30665.306564] EXT4-fs error (device sda2): __ext4_get_inode_loc:4072: inode #6843215: block 27264724: comm mc: unable to read itable block
[30728.245726] EXT4-fs (sda2): error count since last fsck: 8
[30728.245726] EXT4-fs (sda2): error count since last fsck: 8
[30728.245756] EXT4-fs (sda2): initial error at time 1497004049: ext4_mb_generate_buddy:758
[30728.245768] EXT4-fs (sda2): last error at time 1497049827: __ext4_get_inode_loc:4072: inode 6843215: block 27264724

I didn't know they were relevant, maybe they aren't. When the problem first happened obviously the card was plugged directly into the Pi because I was booted from it. In troubleshooting since I had it plugged into an Iogear reader, plugged into a 6 foot extension cable, plugged into a powered hub. For convenience and because it seemed to work. I've since plugged the reader directly into the Pi, these messages are from the backscroll buffer of an rxvt that was open. Also see viewtopic.php?f=28&t=34370&p=1173302#p1173302 for a screenshot of the crash.

Now I see:

Code: Select all

[71395.573253] usb 1-1.2: new high-speed USB device number 13 using dwc_otg
[71396.073253] usb 1-1.2: device not accepting address 13, error -71
[71396.333270] usb 1-1.2: new high-speed USB device number 15 using dwc_otg
[71396.440649] usb 1-1.2: New USB device found, idVendor=0bda, idProduct=0119
[71396.440665] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[71396.440671] usb 1-1.2: Product: USB2.0-CRW
[71396.440678] usb 1-1.2: Manufacturer: Generic
[71396.440685] usb 1-1.2: SerialNumber: 20090815198100000
[71396.443279] usb-storage 1-1.2:1.0: USB Mass Storage device detected
[71396.445347] scsi host3: usb-storage 1-1.2:1.0
[71397.383125] usb 1-1.2: USB disconnect, device number 15
[71397.623242] usb 1-1.2: new high-speed USB device number 16 using dwc_otg
[71397.730764] usb 1-1.2: New USB device found, idVendor=0bda, idProduct=0119
[71397.730781] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[71397.730788] usb 1-1.2: Product: USB2.0-CRW
[71397.730794] usb 1-1.2: Manufacturer: Generic
[71397.730800] usb 1-1.2: SerialNumber: 20090815198100000
[71397.732704] usb-storage 1-1.2:1.0: USB Mass Storage device detected
[71397.732918] scsi host4: usb-storage 1-1.2:1.0
[71397.895212] usb 1-1.2: USB disconnect, device number 16
[71398.133257] usb 1-1.2: new high-speed USB device number 17 using dwc_otg
[71398.240657] usb 1-1.2: New USB device found, idVendor=0bda, idProduct=0119
[71398.240672] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[71398.240678] usb 1-1.2: Product: USB2.0-CRW
[71398.240685] usb 1-1.2: Manufacturer: Generic
[71398.240691] usb 1-1.2: SerialNumber: 20090815198100000
[71398.242586] usb-storage 1-1.2:1.0: USB Mass Storage device detected
[71398.242838] scsi host5: usb-storage 1-1.2:1.0
[71399.235687] scsi 5:0:0:0: Direct-Access     Generic- SD/MMC           1.00 PQ: 0 ANSI: 0 CCS
[71399.238304] sd 5:0:0:0: Attached scsi generic sg0 type 0
[71400.018867] sd 5:0:0:0: [sda] 249737216 512-byte logical blocks: (128 GB/119 GiB)
[71400.019078] sd 5:0:0:0: [sda] Write Protect is off
[71400.019087] sd 5:0:0:0: [sda] Mode Sense: 03 00 00 00
[71400.019287] sd 5:0:0:0: [sda] No Caching mode page found
[71400.019296] sd 5:0:0:0: [sda] Assuming drive cache: write through
[71400.023433]  sda: sda1 sda2
[71400.026614] sd 5:0:0:0: [sda] Attached SCSI removable disk
[72279.006580] EXT4-fs (sda2): warning: mounting fs with errors, running e2fsck is recommended
[72279.025268] EXT4-fs (sda2): Errors on filesystem, clearing orphan list.

[72279.025285] EXT4-fs (sda2): recovery complete
[72279.038114] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[72579.229149] EXT4-fs (sda2): error count since last fsck: 7
[72579.229180] EXT4-fs (sda2): initial error at time 1497004049: ext4_mb_generate_buddy:758
[72579.229190] EXT4-fs (sda2): last error at time 1497006429: ext4_mb_generate_buddy:758
[73327.989177] usb 1-1.4.1.3: USB disconnect, device number 9
[73327.989551] cp210x ttyUSB0: cp210x converter now disconnected from ttyUSB0
[73327.989594] cp210x 1-1.4.1.3:1.0: device disconnected

I did mount the card through the reader, also unplugged a Gekko Science bitcoin plugminer (the cp210x I think) . I still see:

Code: Select all

pi1# e2fsck -f -y /dev/sda2
e2fsck 1.42.12 (29-Aug-2014)
/dev/sda2: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway? yes

e2fsck: unable to set superblock flags on /dev/sda2


/dev/sda2: ********** WARNING: Filesystem still has errors **********

No amount of messing with the filesystem will fix a problem with the SD card, card reader, USB connection, etc.

I do have ample hard drive space in an amd64 Linux laptop to copy the card off to (about 60 GB I think) but dd might not be the best way to do it. Dump/restore? A tarball? I seem to be able to read from it, at least in limited amounts. Trying to do a du is giving me some Input/output error messages (and taking a long time). I wonder if I copy off what I can, format the card, copy everything back if that might work. I had cloned this card a couple weeks before this.

In the few days before this I did notice that the Pi shut itself down twice in the middle of the night sometime, I thought that was due to an RTL2832 dongle that was plugged in and I was running rtl_power on.

Running badblocks now, found 7 so far.

[ab1jx]

OK, that took hours on a 128 GB card. It found 3936096 bad blocks and I have 124803071 so by calc 3.153845% of the space is bad. Do I believe it? I'm not sure but I've about had it with SD cards. 10 years ago I could have afforded a new card every month but now that I'm retired on Social Security I can't. Health monitoring would help: https://superuser.com/questions/830849/ ... -a-sd-card

It seems like I've read that the Pi will boot from USB devices, or if not I'll just keep several copies of a small boot-only SD card. A 1 TB hard drive only costs about $10 more than another 128 GB SD card, and I just happen to have a new spare 270 GB lying around. I think it was about $30. I need to come up with a way to adapt a laptop SATA drive to USB that's all. It's perfectly feasible, but I'm going to need a case and power supply. Something like https://www.startech.com/HDD/Enclosures ... NI251BMU33 but https://www.startech.com/HDD/Enclosures ... ~IDE2510U2 would have done the trick. I don't really want USB-powered, and I don't have room for any more wall warts, so a manageable 2-wire AC cord would be best.

Actually my earlier idea of making a RAID array of SD cards in readers is seeming practical too, they're a lot easier to swap out than a hard drive.

[rpdom]

I have used either old SATA laptop drives (zero cost, I already had them) with an adaptor from ebay (about £3), or 2TB USB hard drives in enclosures for around £65. The 2TB drives are USB3, but seem to work fine on a Pi2 or 3 as long as you boot from an SD card. They don't seem to work with USB boot on the Pi 3, which is a pity as that is what I'd like to do.

[ab1jx]

Well, I guess the crisis is over. I mounted the dead SD in a reader and I was able to cherry pick the most important stuff to salvage and copy it off to a USB stick. Then I plugged the SD into a Linux laptop and used dd to copy everything off into its own partition. Then I ran fsck on that and was able to mount it. There wasn't much left, nothing worth salvaging. So now I've got a dead SD to play with. It's a Sandisk, but one of the lower priced ones I suspect. I don't think it was a counterfeit one, I bought it someplace like Adorama. I have 2 of them so I'm not sure which was which. Paid about $45 each for 128 GB ones.

Avoid these

bad_sd_2_16c.gif (5.8 KiB) Viewed 1662 times

I've had hard drives die but not in 20 years or so. I generally take them out of use when they're 3-5 years old, put them in a safe place in case I want to copy more stuff off later. Eventually it becomes unimportant because it's so old. Hard drives have some really fun neodymium magnets in them, and the platters make decent mirrors. Or frisbees.