andrum99
Posts: 1162
Joined: Fri Jul 20, 2012 2:41 pm

Force a full fsck on reboot, ignoring the dirty flag

Tue Aug 28, 2012 2:09 pm

I have experienced several instances of data corruption on the SD card on my Pi. I am still investigating the root cause of this. At the moment to correct filesystem errors I shutdown the pi and put the card into my laptop to fsck the root filesystem. This is because a "forced" fsck on reboot does not always work: the filesystem will only be checked if the dirty flag is set. Usually the dirty flag is not set, since the OS thinks the filesystem is fine, except for reading duff data from certain parts of the FS. I need a way to forcible run an "e2fsck -fy" on the root filesystem on reboot. This would remove the need to take the SD card out of the Pi and do the filesystem check on my laptop. Does anyone have any ideas how to accomplish this?

Thanks

Andrew.

Eddy1A1
Posts: 9
Joined: Tue Aug 28, 2012 4:41 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 7:31 am

I'm new in using a RPi, so forgive me if I didn't understand very well this problem. If you have problems with RPi's OS try to format the SD card and then rewrite the IMG file on it.

andrum99
Posts: 1162
Joined: Fri Jul 20, 2012 2:41 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 9:18 am

I'm trying to avoid reformating the SD card, although I have done that a few times when the corruption seems too bad for me to easily fix the problems it causes.

Thanks

Andrew.

ksangeelee
Posts: 192
Joined: Sun Dec 25, 2011 5:25 pm
Location: Edinburgh, UK
Contact: Website

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 10:37 am

As root (or using sudo), I 'mount / -o remount,ro' to remount the filesystem read-only, then fsck can be run safely.

This might require shutting down daemons (e.g. /etc/init.d/apache2 stop) that hold write-locks on files in your root filesystem. The utility 'lsof' (apt-get install lsof) is useful to identify owning processes of currently open files, should you keep getting the '/ is busy' messages when trying to remount read-only.

andrum99
Posts: 1162
Joined: Fri Jul 20, 2012 2:41 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 10:50 am

As root (or using sudo), I 'mount / -o remount,ro' to remount the filesystem read-only, then fsck can be run safely.

This might require shutting down daemons (e.g. /etc/init.d/apache2 stop) that hold write-locks on files in your root filesystem. The utility 'lsof' (apt-get install lsof) is useful to identify owning processes of currently open files, should you keep getting the '/ is busy' messages when trying to remount read-only.
I tried that and did indeed get the '/ is busy' message. lsof did not prove useful though as I could not see any way to identify which open files had write locks set on them. What I really need is a way to just force a filesystem check on each boot, before the root filesystem gets remounted rw.

Thanks

Andrew.

andrum99
Posts: 1162
Joined: Fri Jul 20, 2012 2:41 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 11:21 am

I have done some more digging. It seems that forcing an fsck in reboot adds '-f' to the command line for fsck, so that should work. I have found the script that does the fsck on boot - it is /etc/rcS.d/S05checkroot.sh . For some reason in this script the 'rootcheck' variable is being set to 'no', which is disabling the root filesystem check. I am still investigating this but it looks like it is actually a bug in raspbian. I will post more once I have idenitifed why rootcheck is being set to 'no'.

Cheers

Andrew.

ksangeelee
Posts: 192
Joined: Sun Dec 25, 2011 5:25 pm
Location: Edinburgh, UK
Contact: Website

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 12:10 pm

andrum99 wrote: I tried that and did indeed get the '/ is busy' message. lsof did not prove useful though as I could not see any way to identify which open files had write locks set on them. What I really need is a way to just force a filesystem check on each boot, before the root filesystem gets remounted rw.
First of all, lsof's output, the FD column entries which are a number followed by 'w' or 'u' will have write or read/write locks (e.g. '2w' or '7u' - the number is the file-descriptor).

Sorry for not reading your original question properly - I'd incorrectly read that you wanted to do a one-time fsck.

I also had problems last night getting a boot-time fsck to work. Using 'touch /forcefsck' or shutdown with a -F flag didn't work.

Looking deeper, I found that my fstab file didn't have the DUMP and PASS fields defined for the root entry, and these default to zero if not specified. Adding '0 1' to the end of my root entry fixed this (a PASS of zero causes the init script to skip the filesystem - see /lib/init/mount-functions.sh)

andrum99
Posts: 1162
Joined: Fri Jul 20, 2012 2:41 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 3:39 pm

Sorry for not reading your original question properly - I'd incorrectly read that you wanted to do a one-time fsck.
I changed my requirements halfway through this thread - sorry. I had thought a one-time check would be OK but then revised this to a check on every boot.

I have set up my fstab as you suggested and my system now responds to the presence of the /forcefsck file.

I will ask the raspbian developer mailing list why they disabled fsck on boot as on the face of it this seems like an odd thing to do.

I have also changed FSCKFIX to yes in /etc/default/rcS to allow all fixes to be made to the filesystem during the boot time check.

Thanks for your help in tracking this one down.

Andrew.

andrum99
Posts: 1162
Joined: Fri Jul 20, 2012 2:41 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 4:01 pm

I added '/usr/bin/touch /forcefsck' to /etc/rc.local so that the Pi always does a forced fsck of the root filesystem on each boot. It also just picked up some filesystem corruption, but luckily none of the daemons seems to be affected.

Cheers

Andrew.

ksangeelee
Posts: 192
Joined: Sun Dec 25, 2011 5:25 pm
Location: Edinburgh, UK
Contact: Website

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 6:25 pm

andrum99 wrote: It also just picked up some filesystem corruption, but luckily none of the daemons seems to be affected.
I assume you've already followed the various threads that discuss power-supplies, poor solder-joints, bent card-slot connectors, etc. as causes of filesystem corruption. If you're on recent firmware and kernel, without over-clocking, and if the filesystems are being unmounted cleanly, then you shouldn't be seeing this kind of problem.

andrum99
Posts: 1162
Joined: Fri Jul 20, 2012 2:41 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Wed Aug 29, 2012 6:43 pm

I've not followed those threads - that's my next port of call.

Cheers

Andrew.

diggool
Posts: 1
Joined: Sat Aug 02, 2014 11:38 am

Re: Force a full fsck on reboot, ignoring the dirty flag

Sat Aug 02, 2014 2:57 pm

Hi,

I'm having problems forcing the "fsck" to run. I'm sure I've followed the suggestions in the the posts but I still no sign that "fsck" is running at start up and getting the boot message to run "fsck".
Could someone list the files that need to be changed / created
then the commands that can be used
Just so I'm clear that I am doing things right, I'm not as clued up on files and structure as the rest of you seem to be.

Many Thanks
Jon

davemunn73
Posts: 1
Joined: Thu Apr 02, 2015 1:59 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Thu Apr 02, 2015 2:02 pm

Hi Andrum99
Can you tell me how you got to pi to do a fsck on every reboot when the filesystem was flagged as dirty?
Id like to do that for some pi's i have running chromium in kiosk mode.
Thanks
Dave

Fahrenheit
Posts: 22
Joined: Fri Feb 14, 2014 3:39 am

Re: Force a full fsck on reboot, ignoring the dirty flag

Thu Jan 14, 2016 7:23 pm

diggool wrote:Hi,

I'm having problems forcing the "fsck" to run. I'm sure I've followed the suggestions in the the posts but I still no sign that "fsck" is running at start up and getting the boot message to run "fsck".
Could someone list the files that need to be changed / created
then the commands that can be used
Just so I'm clear that I am doing things right, I'm not as clued up on files and structure as the rest of you seem to be.

Many Thanks
Jon
Looks like this is still an issue in Raspbian Jessie. To recap what the gentlemen did above (and how I got it to work on my system):

Code: Select all

sudo nano /cat/default/rcS
Change the FSCKFIX option to yes

Code: Select all

FSCKFIX=Yes
ctrl-x and y to save

Next use the "touch" executable to create a "forcefsck" file in the root /

Code: Select all

sudo touch /forcefsck
Reboot the Pi

Code: Select all

sudo shutdown -h -r
You should see fsck lines during boot, checking all attached drives. You can validate whether it ran by checking the syslog:

Code: Select all

sudo cat /var/log/syslog | grep fsck
Funny, it actually says not to use "/forcefsck" but I'm not sure how to pass "fsck.mode=force" to the kernel. Also, I haven't seen if the devs removed the feature for a particular reason.

User avatar
jojopi
Posts: 3193
Joined: Tue Oct 11, 2011 8:38 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Thu Jan 14, 2016 10:23 pm

Fahrenheit wrote:FSCKFIX=Yes
You must spell it "yes", not "Yes".
I'm not sure how to pass "fsck.mode=force" to the kernel
That is really a userspace parameter; the kernel knows nothing about fsck. To pass such an option to the kernel and userspace on every boot, add it to /boot/cmdline.txt, keeping that file all on one line.

But the whole premise of this thread is flawed. fsck can repair routine inconsistencies, not corruption. If a filesystem is marked clean, but is actually corrupt, then some device-level problem has occurred. Perhaps the SD card is faulty, or power has been removed before the card finished reallocating data.

For fsck to run at all, the filesystem must be good enough to mount and boot, and to find and load the fsck binary and all its libraries correctly. Then it can repair any inconsistencies, and gloss over any corruption. If there are device-level problems, sooner or later the system will stop booting, regardless of how often or how hard you run fsck.

ferdyp
Posts: 1
Joined: Tue Aug 23, 2016 6:53 am

Re: Force a full fsck on reboot, ignoring the dirty flag

Tue Aug 23, 2016 6:55 am

Code: Select all

shutdown -r -F now
works for me @wheezy

alexghiti
Posts: 2
Joined: Tue Jan 31, 2017 10:56 am

Re: Force a full fsck on reboot, ignoring the dirty flag

Tue Jan 31, 2017 11:54 am

Hi all,

Old thread but without any clear good solution. The only good solution I found is to create a new kernel image with initramfs embedded that include e2fsck and that handles mounting rootfs afterwards.
If anyone interested, I will push on github the buildroot .config and init script. However, I think that the best way for users to have easily access to the above solution would be a package in raspbian repos.

So if any moderator/contributor/user is interested, do not hesitate to contact me.

Thanks

Alex

havasu
Posts: 1
Joined: Mon Mar 27, 2017 6:12 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Mon Mar 27, 2017 6:30 pm

I have remote wifi routers based on raspberries. It is critical make it boot on power failures. So far /usr/bin/touch /forcefsck' to /etc/rc.local helps but guess could be other filesystem failure situations and it would stop doing fsck on boot. alexghiti: kernel image with initramfs embedded would be best solution. I googled but could not find any samples how to make initramfs with fsck. Please let us know how to make it.

alexghiti
Posts: 2
Joined: Tue Jan 31, 2017 10:56 am

Re: Force a full fsck on reboot, ignoring the dirty flag

Tue Mar 28, 2017 1:20 pm

Hi @havasu,

I uploaded on github all you need : https://github.com/AlexGhiti/rpi_initramfs_e2fsck
I have just tested what I wrote in the README, if anyone needs more infos, please file an issue in github.

Hope it will help some people, it helped me :)

Alex

valent
Posts: 28
Joined: Mon Jan 30, 2012 9:40 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Fri Apr 21, 2017 1:46 pm

There were typos and wrong path names, so to have correct steps in one place here is what you need to do:

Code: Select all

sudo su -c 'echo "FSCKFIX=yes" >> /etc/default/rcS'
sudo touch /forcefsck
sudo reboot & exit

capricorn180
Posts: 1
Joined: Fri Oct 03, 2014 7:47 am

Re: Force a full fsck on reboot, ignoring the dirty flag

Sun Jul 23, 2017 5:32 am

Fahrenheit wrote:
diggool wrote:Hi,

Code: Select all

sudo nano /cat/default/rcS
Change the FSCKFIX option to yes

Code: Select all

FSCKFIX=Yes
ctrl-x and y to save
I just found this page and its really useful, anyone following the steps above the rcS is file path should be

Code: Select all

sudo nano /etc/default/rcS

User avatar
ab1jx
Posts: 885
Joined: Thu Sep 26, 2013 1:54 pm
Location: Heath, MA USA
Contact: Website

Re: Force a full fsck on reboot, ignoring the dirty flag

Sun Jul 08, 2018 12:14 pm

Simpler method, thanks to Redhat: cd /, touch forcefsck, reboot. May need some sudos in there.

I created the file /_forcefsck as a reminder:

Code: Select all

If you do:
cd /
touch forcefsck

At least in Redhat this will force an fsck to run on boot before the partitions
are mounted.  Very useful for when things are acting weird and you don't want to
boot from a different SD so you can fsck the one you booted from.  Or you don't
have a reader to do that with.  Or a spare bootable SD.

From:
https://access.redhat.com/discussions/3218241
Maybe it's not a full fsck, but the /forcefsck file gets removed like it knows what it means. Watching it, it ran fairly quickly, then rebooted (I saw the rainbow splash screen again). Google forcefsck

klesk
Posts: 18
Joined: Tue Aug 08, 2017 7:16 am

Re: Force a full fsck on reboot, ignoring the dirty flag

Fri Feb 14, 2020 7:29 pm

I would contribute my solution to this:

You can add:
PARTUUID=<UUID> / ext4 noatime,errors=continue 0 1

errors=continue to your fstab instead of FSCKFIX=yes. This way it will try to boot no matter how badly the filesystem is corrupted, I would say it matters more to have a headless system on a remote location back up and running then try to repair the SDcard and might fail in the process.

Let's discuss!

User avatar
jojopi
Posts: 3193
Joined: Tue Oct 11, 2011 8:38 pm

Re: Force a full fsck on reboot, ignoring the dirty flag

Sat Feb 15, 2020 12:51 pm

klesk wrote:
Fri Feb 14, 2020 7:29 pm
errors=continue to your fstab instead of FSCKFIX=yes.
"errors=continue" is the default:

Code: Select all

[email protected]:~ $ sudo tune2fs -l /dev/mmcblk0p2 |grep -i error
Errors behavior:          Continue
"fsck.repair=yes" is also the default in current images.

No amount of ignoring or repairing errors can guarantee to keep the system working, if the SD card is losing data.

Return to “Troubleshooting”