Shell corrupted - virus or SD failure?

[andig2]

Just tried installing some node modules on a pi2 that I've recently setup and noticed this:

[email protected]:~ $ npm
-bash: /usr/local/bin/npm: /bin/sh: bad interpreter: No such file or directory
[email protected]:~ $ ls -l /bin/sh*
lrwxrwxrwx 1 root root 4 Jan 21 2014 /bin/sh -> da3h
[email protected]:~ $ sudo cat /bin/sh*
cat: /bin/sh: No such file or directory

Sounds like an SD broken early or something worse?

[JimmyN]

It normally should look something like this, it links "sh" to "dash" as the default shell.

Code: Select all

ls -l /bin/sh*
lrwxrwxrwx 1 root root 4 Jan 21  2014 /bin/sh -> dash
lrwxrwxrwx 1 root root 4 Jan 21  2014 /bin/sh.distrib -> dash

and

Code: Select all

ls -l /bin/dash
-rwxr-xr-x 1 root root 104088 Jan 21  2014 /bin/dash

[DougieLawson]

Most likely is a trashed SDCard. Time to restore it from your backup.

[jojopi]

lrwxrwxrwx 1 root root 4 Jan 21 2014 /bin/sh -> da3h

The correct third character "s" is ASCII 0x73, and the incorrect "3" is 0x33. That is exactly one bit that has flipped, from binary 01110011 to 00110011. It looks very much like a hardware error, rather than anything malicious.

If the system is still running, you should be able to fix the fault with:

Code: Select all

sudo ln -snf dash /bin/sh

The target of a symbolic link is stored inside the inode when it is short enough to fit there, as it is in this case. Inodes fit about sixteen to the filesystem block, so they have to be updated in blocks. The bit flip most likely happened while a nearby inode was being changed and this one had to be read and re-written. The fault could be either with the SD card or the Pi itself.

Have you enabled any over-clocking? It may be worth checking if this finds any problem with the Pi's RAM:

Code: Select all

sudo apt-get install memtester
memtester 768M

[andig2]

Thanks for the help. I didn't get as far as the mem test:


[email protected]:~ $ sudo apt-get install memtester
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
memtester
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 17.5 kB of archives.
After this operation, 27.6 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
memtester
Install these packages without verification? [y/N] y
Err http://mirrordirector.raspbian.org/raspbian/ jessie/main memtester armhf 4.3 .0-3
Could not resolve 'mirrordirector.raspbian.org'
E: Failed to fetch http://mirrordirector.raspbian.org/rasp ... in/m/memte ster/memtester_4.3.0-3_armhf.deb Could not resolve 'mirrordirector.raspbian.org '

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-mis sing?

Running non-overclocked this is the 2nd SD thats broken now. Altough they are some 2 years old, the SDs were rarely used. Trying to setup a third now- if that turns corrupt, too I'd rather retire this specific Raspi I guess.

[tpylkko]

I don't know if you realize, but corrupted data does not need to mean that the hardware is bad. It certainly can be because the sd-card is broken, but can also be that there is nothing wrong with the card and that re-imaging it or fixing the corrupted data could suffice.

[andig2]

For sure, but- this has just happened again- on a 3rd SD. Broke while updating:

[email protected]:~ $ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... The following packages were automatically installed and are no longer required:
libc-ares2 libv8-3.14.5
Use 'apt-get autoremove' to remove them.
Done
The following packages will be upgraded:
dmsetup dpkg dpkg-dev firmware-atheros firmware-brcm80211 firmware-libertas firmware-ralink firmware-realtek initramfs-tools libdevmapper1.02.1 libdpkg-perl libexpat1
libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0 libssl1.0.0 libwbclient0 libxapian22 libxml2 openssl perl perl-base perl-modules raspberrypi-bootloader
raspberrypi-kernel samba-common tzdata
28 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 85.7 MB of archives.
After this operation, 507 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.raspberrypi.org/debian/ jessie/main firmware-atheros all 0.43+rpi5 [873 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main dpkg armhf 1.17.27 [2,930 kB]
Get:3 http://archive.raspberrypi.org/debian/ jessie/main firmware-brcm80211 all 0.43+rpi5 [1,678 kB]
Get:4 http://archive.raspberrypi.org/debian/ jessie/main firmware-libertas all 0.43+rpi5 [1,842 kB]
....
Get:26 http://archive.raspberrypi.org/debian/ jessie/main libraspberrypi-bin armhf 1.20160620-1 [253 kB]
Get:27 http://archive.raspberrypi.org/debian/ jessie/main libraspberrypi0 armhf 1.20160620-1 [726 kB]
Get:28 http://archive.raspberrypi.org/debian/ jessie/main raspberrypi-bootloader armhf 1.20160620-1 [3,149 kB]
Fetched 85.7 MB in 26s (3,276 kB/s)
Reading changelogs... Done
Preconfiguring packages ...
(Reading database ... 32066 files and directories currently installed.)
Preparing to unpack .../dpkg_1.17.27_armhf.deb ...
Unpacking dpkg (1.17.27) over (1.17.26) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up dpkg (1.17.27) ...
(Reading database ... 32066 files and directories currently installed.)
Preparing to unpack .../perl_5.20.2-3+deb8u5_armhf.deb ...
Unpacking perl (5.20.2-3+deb8u5) over (5.20.2-3+deb8u4) ...
Preparing to unpack .../perl-base_5.20.2-3+deb8u5_armhf.deb ...
Unpacking perl-base (5.20.2-3+deb8u5) over (5.20.2-3+deb8u4) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up perl-base (5.20.2-3+deb8u5) ...
(Reading database ... 32067 files and directories currently installed.)
Preparing to unpack .../perl-modules_5.20.2-3+deb8u5_all.deb ...
Unpacking perl-modules (5.20.2-3+deb8u5) over (5.20.2-3+deb8u4) ...
dpkg: error processing archive /var/cache/apt/archives/perl-modules_5.20.2-3+deb8u5_all.deb (--unpack):
unable to create `/usr/share/perl/5.20.2/Unicode/Collate/Locale/tr.pl.dpkg-new' (while processing `./usr/share/perl/5.20.2/Unicode/Collate/Locale/tr.pl'): Read-only file system
dpkg: error while cleaning up:
unable to remove newly-extracted version of `/usr/share/perl/5.20.2/Unicode/Collate/Locale/tr.pl': Read-only file system
dpkg: error while cleaning up:
unable to restore backup version of `/usr/share/perl/5.20.2/Unicode/Collate/Locale/as.pl': Read-only file system
dpkg: error while cleaning up:
unable to restore backup version of `/usr/share/perl/5.20.2/Unicode/Collate/Locale/zh_strk.pl': Read-only file system
dpkg: error while cleaning up:
unable to restore backup version of `/usr/share/perl/5.20.2/Unicode/Collate/Locale/nb.pl': Read-only file system
dpkg: error while cleaning up:
unable to restore backup version of `/usr/share/perl/5.20.2/Unicode/Collate/Locale/yo.pl': Read-only file system

My feeling is that this particular raspi eats SDs or just doesn't like them. I'll try one last time with a brand new SD, if that fails as well then it can only be something about the raspi (and its got a 2A power supply and no peripherals attached).

[drgeoff]

You are shutting down properly and waiting before removing power?

[andig2]

I didn't even switch it off in between. If I do when shutdown first and remove power then. This last SD died right in the middle of apt-get, without mechanical influence or anything. Have not seen such behaviour with the older Bs I was using sofar...

[skspurling]

SD failure is mostly an electronic thing. You don't have to be removing things. I think the newer versions of Rasbian are a bit harder on the SD cards, just anecdotally.

So, the things that affect SD card life are the number of writes made, and the amount of writes that are cached and not immediately synced to disk. The first is a known quantity, and one method of circumvention is operating out of a ram cache and only writing to SD when needed. Has a side effect of making frequent r/w cycles even faster at the expense of a little RAM. Worst part is it makes it even more important to do a graceful shutdown, because if you don't signal the kernel to sync and wait for the write to complete, things can get really messed up very easily. It's almost like this latter tact has become more common in newer releases. It appears to have become more important than ever to do clean shutdowns.

So, this can be an issue. I'm also a bit shaking on trying to figure out when the shutdown is complete by using the LED's. Frankly the HDMI and Ethernet are shutdown fairly early in the process, and it's like the writes sometime drag out a bit. Is there a good rule of thumb that people use? Is there a way to adjust the amount of caching for different applications? Reduce the amount of unneeded disk writes? I am thinking about applications like a data logger that would not be updating system files very often, but would not be impacted too bad if a few reading were lost due to brownout or power loss.

[jojopi]

Worst part is it makes it even more important to do a graceful shutdown, because if you don't signal the kernel to sync and wait for the write to complete, things can get really messed up very easily.

If you switch off without saving, or shortly after saving while the data are still in the kernel buffers, then you lose those data. That is a different issue to other parts of the filesystem being corrupted, which I suggest should only be able to occur if you happen to switch off during a write, while the card itself is moving blocks around.

I am not aware of anything that would make writes more or less frequent between Raspbian images, other than differing workloads. The length of time that non-fsynced data is held by the kernel before being flushed can be tuned using sysctl parameters such as vm.dirty_expire_centisecs, which defaults to 3000. See https://github.com/raspberrypi/linux/bl ... ctl/vm.txt

Anyway, OP's problem of bit flips and metadata corruption being detected while running, can only be some kind of hardware or compatibility issue.

[tpylkko]

The cards could be coerupted over and over again by faulty ram or by bad writes due to bad power or overclocking and such. If the sd drive is the problem, you could try moving everything possible off the card to a usb drive

[stderr]

For sure, but- this has just happened again- on a 3rd SD. Broke while updating:

Swap out everything, the usb cable, the power supply, the sdcard with one from known working system, run with nothing but display and keyboard. Is there still a reoccurring problem?

[davidcoton]

It could be a faulty Pi causing corruption of the data. I'm assuming the SDCards are recoverable? Can you use a card that has been tested in another Pi? If that fails, and if its not a PSU issue (red LED on, no rainbow square in the corner of the screen, swapping PSU doesn't solve it), then it's time to swap Pis.

When shutting down, the activity LED (green on most models) should flash ten times before settling permanently one way. Wait for that before turning off or unplugging anything.

[andig2]

Checking back in for an update.

> Swap out everything, the usb cable, the power supply, the sdcard with one from known working system, run with nothing but display and keyboard. Is there still a reoccurring problem?

There wasn't much so swap- only the PSU with fixed cable is attached plus ethernet. Before swapping the PSU I've done one last test with a brand new Samsung micro SD.

Guess what: running like a pro since my last posting while the others (cards rarely used but 2~3 years old) "died", that is showed varying symptoms, within minutes our hours.

So for time being- and while nothing else has gone wrong- I have to assume that the SDs had aged and were no longer in mint condition...

Thanks for all your help,
Andreas