ryanteck
Posts: 82
Joined: Mon Apr 16, 2012 10:49 am
Location: Great Yarmouth , United Kingdom
Contact: Website Twitter

CM3+ EMMC Corruptions?

Tue Jan 14, 2020 1:04 pm

Hi,

I'm currently developing a board that uses the RPi CM3+ and I've been noticing an unusal issue on it.

Sometimes when it is powered up or rebooted it seems like the data has corrupted on the EMMC, however other times it boots up perfectly fine. Along with this I have re-flashed the EMMC and the issues come back so I don't believe it's a once off bad write.

I'm fairly certain it is along the lines of corruption as either it'll refuse to SSH in with the error "ssh_exchange_identification: read: Connection reset by peer" possibly indicating corrupt SSH Keys. And if I can get logged in I get errors such as "-bash: /bin/ls: Input/output error"

Now my main thing is what could be causing this, I have experienced the issues with an 8GB CM3+ and a 16GB CM3+.

The only thing I can think potentially is that the CM3+ datasheet says the supply sequencing should be in order from highest vltage to lowest voltage however I did not implement this staggering into the design (I misread this section) and could potentially explain why it is intermittent.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2658
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3+ EMMC Corruptions?

Tue Jan 14, 2020 1:11 pm

Sometimes when it is powered up or rebooted it seems like the data has corrupted on the EMMC, however other times it boots up perfectly fine.
To avoid any confusion, do you always have to fix the corruption before it will boot successfully? In other words, is this occasional corruption caused by a read failure or a write failure/write in error?

ryanteck
Posts: 82
Joined: Mon Apr 16, 2012 10:49 am
Location: Great Yarmouth , United Kingdom
Contact: Website Twitter

Re: CM3+ EMMC Corruptions?

Tue Jan 14, 2020 1:52 pm

It doesn't seem it, I had to power cycle it a couple of times and it failed both times.

Then plugged in a UART cable to see if I could get any error logs and it then worked perfectly that time. It does kind of make me think based on that it could be power.

However sometimes corruptions / boot failures can then happen from running ```sudo reboot``` which has no power cycling involved.

I'll try rebooting it some more to see if I can get it to happen again. If not while I have tried both my CM3+ modules I haven't tested to see if it happens with my CM3 module.

Edit: While doing some more work with it had it crash. Here's the log.

Code: Select all

[   17.119135] EXT4-fs error (device mmcblk0p2): __ext4_get_inode_loc:4678: inode #127343: block 524430: comm fsck.vfat: unable to read itable block
[   17.237022] EXT4-fs error (device mmcblk0p2): ext4_find_entry:1455: inode #24524: comm (s_env.sh): reading directory lblock 0
[   17.274395] EXT4-fs error (device mmcblk0p2): ext4_journal_check_start:61: Detected aborted journal
[   17.288747] EXT4-fs (mmcblk0p2): Remounting filesystem read-only
[   17.297552] EXT4-fs (mmcblk0p2): ext4_writepages: jbd2_start: 2147483646 pages, ino 125261; err -30
[   17.337090] EXT4-fs error (device mmcblk0p2): __ext4_get_inode_loc:4678: inode #15: block 186: comm systemd: unable to read itable block
[   17.381182] EXT4-fs error (device mmcblk0p2): __ext4_get_inode_loc:4678: inode #13: block 186: comm (tmpfiles): unable to read itable block
[   17.455141] EXT4-fs error (device mmcblk0p2): ext4_find_entry:1455: inode #28: comm systemd: reading directory lblock 0
[   17.471519] EXT4-fs error (device mmcblk0p2): ext4_find_entry:1455: inode #28: comm systemd: reading directory lblock 0
[   17.488312] EXT4-fs error (device mmcblk0p2): ext4_find_entry:1455: inode #28: comm systemd: reading directory lblock 0
[   17.505235] EXT4-fs error (device mmcblk0p2): ext4_find_entry:1455: inode #28: comm systemd: reading directory lblock 0
[   17.528171] EXT4-fs error (device mmcblk0p2): __ext4_get_inode_loc:4678: inode #27: block 187: comm systemd: unable to read itable block
[   17.573663] EXT4-fs error (device mmcblk0p2): ext4_find_entry:1455: inode #1565: comm (ate-utmp): reading directory lblock 0

ryanteck
Posts: 82
Joined: Mon Apr 16, 2012 10:49 am
Location: Great Yarmouth , United Kingdom
Contact: Website Twitter

Re: CM3+ EMMC Corruptions?

Wed Jan 15, 2020 9:03 pm

It seems the most likely culpret is something along the power lines so will do some more testing to see if I can solve it. I have probed the voltage rails and they did seem within tolerance.

The CM3+ is on it's own daughterboard from our main board as this allowed us to do an extra feature by having the USB switch for re-programming on this smaller daughterboard.

The Daughterboard has 5V go into it which directly connects to the VBAT, into a 500mA 3v3 Regulator and 300mA 1v8 Regulator (No VDAC used) so didn't expect these issues. However will re-spin it with a larger 1v8 regulator and better 5V rail.

The 1V8 rail not being quite powerful enough could potentially explain the issues but not fully sure, The datasheet says that 3v3 powers various PHYs, IO and eMMC Flash and the 1v8 powers SDRAM however it looks like that the actual EMMC controller is 1.8V as well while the flash itself is 3v3.

Woife
Posts: 2
Joined: Thu Jan 16, 2020 1:32 pm

Re: CM3+ EMMC Corruptions?

Thu Jan 16, 2020 1:45 pm

Exactly the same issue here!

I work in a company where we want to develop a serial product with the CM3.

I flash them by the help of the Waveshare IO Board Plus. I wanted to flash the Raspbian Lite on (2019-09-26) it and add the xfce desktop. When I am connected directly on the board (Start it on the waveshare). On the connected screen I can see the exact same errors. (The installation of xcfe never finised during my tests)

I atm tried two differen CM3 boards. When I am flashing our old serial image on it afterwards (created by our software supplier with buildroot) it works again. It works at least for some minutes on our custom made carrier board (I did not try longer because the errors occur 1-2 minutes after boot).

I also flashed the Raspian Full (2019-09-26) (could also be the middle one I forgot :oops: ) on a 16GB sdcard for a raspberry pi 3B without any issues.

ryanteck
Posts: 82
Joined: Mon Apr 16, 2012 10:49 am
Location: Great Yarmouth , United Kingdom
Contact: Website Twitter

Re: CM3+ EMMC Corruptions?

Thu Jan 16, 2020 6:26 pm

It does seem rather odd, I haven't actually tried it with my CM3 but do have it with two different CM3+ Boards.

But indeed, the primary reason for using the CM3 is to not have the risk of SD Corruptions where as this is what's happening :shock:

Woife
Posts: 2
Joined: Thu Jan 16, 2020 1:32 pm

Re: CM3+ EMMC Corruptions?

Mon Jan 20, 2020 7:24 am

ryanteck wrote:
Thu Jan 16, 2020 6:26 pm
It does seem rather odd, I haven't actually tried it with my CM3 but do have it with two different CM3+ Boards.
I have no CM3+ around to verify your problem. But could you quickly test it for the CM3?

How should we continue with this? Providing more info? Is there maybe something wrong with the Raspbian image?

ryanteck
Posts: 82
Joined: Mon Apr 16, 2012 10:49 am
Location: Great Yarmouth , United Kingdom
Contact: Website Twitter

Re: CM3+ EMMC Corruptions?

Mon Jan 20, 2020 8:26 pm

I haven't had a chance to test on my CM3 but will try to.

I don't believe it is the image at fault, we do create our own in our case using the iot-pi-gen tool for a Raspbian Lite image with our software pre-loaded.

Return to “Compute Module”