sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

CM3 module crash after ejecting sd card

Tue Jun 12, 2018 11:52 am

Hi all,

I have a very annoying issue with the CM3 module which is used in a custom carrier board. After ejecting the external sd-card (SD1 interface) the complete module will crash. I've already removed the non-removable stuff from the devicetree. So I'm able to insert and remove the sd-card after the CM3 has booted. In /var/log/syslog I can see the card detect and remove messages so this works fine. However when I'm writing a (big) file to the sd-card and eject the card (I know you should not do something like this, but it's always possible somebody removes a card) the whole CM3 crashes.
I've tested this with a 4.9 kernel from last year and also the lastest raspbian release. Both images result in the same issue.
Any idea where to look and to solve this issue?

Regards,
Chris

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2577
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 12:00 pm

1. Have you looked at /var/log/syslog for error messages?
2. Do you see any messages on the display or via a serial port on GPIO14&15?

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 12:05 pm

Hi,

1. No error messages in /var/log/syslog
2. Nothing in the serial debug terminal. I'm using GPIO32/GPIO33. I have a LCD connected so the GPIO14/GPIO15 pins are in use.

Regards,

Chris

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2577
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 12:16 pm

So does it just freeze? Or reboot? Any information you give may be useful.

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 12:22 pm

Correct. No reboot. Just freeze. No warning message or whatever in the console or log.
I have to do a manual reboot to get it working again.
Inserting removing the card is no issue. /var/log/syslog confirms. So only when I remove the card during the writing proces of a large file the CM3 freezes. If I don't remove the card during writing there is no issue at all.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2577
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 1:29 pm

It does sound a bit like the old joke:
Patient: Doctor, it hurts when I <something random>.
Doctor: Well, don't <something random>.
Despite that, I took a look at the problem (using an NFS-rooted Pi 3B+), and I also found it didn't respond well to having the card removed during writes. I thought it was possibly more reliable using the sdhost (rather than mmc) interface on the SD card, but I managed to crash an ext4 process, after which it was very unhappy.

Perhaps the Dr. was right.

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 1:57 pm

Luckily another patient has the same problem :D

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2577
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 1:59 pm

I think the same advice applies. Seriously, what is your use case that makes it sensible to allow an SD card to be removed while it is being written to?

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 2:33 pm

I fully agree you should not remove the card during any operation at all. So certainly it's not a use case. Unfortunately the sd-card is free accessible just as in a laptop so people are able to remove the card.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2577
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 2:37 pm

What is the data being written to the card?

Have you considered a different filing system?

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Tue Jun 12, 2018 5:01 pm

At the moment just some log data. Customer will use the card to exchange some files. So in the end there's always a possibility somebody ejects the card too early. The application software should give an error on the screen or something like that. Freezing is not an option for this design. That means somebody has to unplug and take the whole machine offline.
I tried the same thing on another carrierboard and ARM module (imx6) I've made some time ago. So when I write some files the kernel gives some I/O errors in /var/log/syslog and error -110 requesting status.
Any help would be very appreciated.

Regards,
Chris

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1467
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 module crash after ejecting sd card

Wed Jun 13, 2018 10:30 am

I think you've found one of the reasons that mobile phones do not have externally accessible SD card storage, if they have it it is under the battery or inserted with the SIM card...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Wed Jun 13, 2018 5:33 pm

Would be nice if we can find a solution for this issue

User avatar
rpdom
Posts: 16082
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: CM3 module crash after ejecting sd card

Wed Jun 13, 2018 7:08 pm

gsh wrote:
Wed Jun 13, 2018 10:30 am
I think you've found one of the reasons that mobile phones do not have externally accessible SD card storage, if they have it it is under the battery or inserted with the SIM card...
My old phone did have an external microSD slot. It was very handy for getting photos off it an onto my PC. My current one (yes, it's a bit old), has the SD inside the back cover, but can be removed without removing the battery or SIM card. Just select "Unmount SD card" from the menus and it is fine.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2577
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3 module crash after ejecting sd card

Wed Jun 13, 2018 7:16 pm

Did you ever try pulling the card out while, for example, recording video to it?

User avatar
rpdom
Posts: 16082
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: CM3 module crash after ejecting sd card

Wed Jun 13, 2018 7:27 pm

PhilE wrote:
Wed Jun 13, 2018 7:16 pm
Did you ever try pulling the card out while, for example, recording video to it?
No. I was very careful to wait until everything had finished writing to the card. It is a Windows Mobile phone (from around 2006). Still works a bit.

stevegee
Posts: 1
Joined: Tue Jun 19, 2018 7:37 am

Re: CM3 module crash after ejecting sd card

Tue Jun 19, 2018 7:54 am

Hi,

This discussion appears to have ended on a side-track.

I am having the same issue with a SD card on SD1. I fully agree that the kernel should be able to cope with an (illegally) ejected sd card and not freeze up.
Any ideas on how to dig deeper into this problem?

regards,

Steve

rln
Posts: 176
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: CM3 module crash after ejecting sd card

Tue Jun 19, 2018 1:38 pm

You might want to try the same test with my Nard SDK, which is specifically designed to be robust and withstand abuse. Use a regular Pi and eject the SD card during heavy writing to a test-file in /sdcard/ It should survive and recover fine.

http://www.nard.se/
Author of the robust Nard distro http://www.nard.se

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Wed Jun 27, 2018 7:59 am

Hi all,

Even without removing the sd-card I have had a complete freeze a couple of times. If I don't use the sd-card everything seems to be working fine up to now. Somehow I have the feeling the kind of glitch is the same as when you remove the sd-card.
I tried the sd_debug parameter in the device-tree. But this enables debugging for SD0. Is there a simple way to enable the same debug info from SD1?

josco92
Posts: 1
Joined: Wed Jan 08, 2020 8:17 am
Location: fine power x in south korea

Re: CM3 module crash after ejecting sd card

Wed Jan 08, 2020 8:26 am

Hi, All

I have the same problem. Did this problem been solved?

sirhc
Posts: 9
Joined: Tue Jun 12, 2018 11:41 am
Location: Maastricht-Airport Netherlands

Re: CM3 module crash after ejecting sd card

Wed Feb 12, 2020 9:27 am

Unfortunately I was not able to solve this issue. Did you have any progress with this issue?

Regards Chris

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1467
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 module crash after ejecting sd card

Wed Feb 12, 2020 9:54 am

It is impossible to solve this issue while you are using a writable ext4 partition, this just won't ever work because it isn't designed to. Even x86 servers running with SSDs will run into this problem if you just randomly power off the servers, that's why we have UPSs for server systems, they're not there to keep the server running during a power cut, but a method used to make sure the operating system flushes it's caches and unloads the filesystem.

Even journalling filesystems will have this problem, because the kernel drivers cannot handle some types of corruption errors and require that you run fsck.ext4 on the filesystem before it'll boot again (on a different computer)

To solve this kind of problem, the best thing to do is to use the overlayfs on the ext4 partition (you can find it as an option in raspi-config in Raspbian) and then create a new partition for your data (Maybe FAT32 or something so it's easy to access from a Windows machine). Run an fsck on the data partition on every boot, and if it fails reformat it from scratch.

Either that or use a buildroot with a ram based filesystem...
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

Return to “Compute Module”