drmatt
Posts: 37
Joined: Sat Sep 07, 2019 3:47 am

cm4 dead - any recovery options?

Wed Jun 16, 2021 9:51 pm

A cm4 module that we'd been using for several months in a custom carrier board appears to be fried. When we powered up the board today, the power LED lit up but there was no other activity (such as ethernet LEDs) and we weren't able to ssh to it. We then powered it down and back on again, and this time the power LED came on for a few seconds and then went dark permanently. We tried transferring the cm4 into a standard pi I/O board with the same result - no LEDs, just a very faint buzz. Are there any simple diagnostics that can be run using a basic scope / multimeter to confirm that the module (vs. the carrier board) is fully dead and what the issue may have been? Also, might there be any way to recover data from the onboard eMMC? Sadly, a few large files generated yesterday didn't have a chance to be backed up.

User avatar
Gavinmc42
Posts: 5884
Joined: Wed Aug 28, 2013 3:31 am

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 2:01 am

Buzzing?
Maybe one or more of those ferrite inductors in the power supply section has cracked?
See if poking them makes a noise difference.
In which case it might be toasted.
Is there a way to check those voltages?

Provide separate power to the CM4 in the right power up sequence.
Possible, not easy.
Fix the inductor if that is the issue.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

aBUGSworstnightmare
Posts: 3232
Joined: Tue Jun 30, 2015 1:35 pm

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 4:31 am

transferring the cm4 into a standard pi I/O board
so the problem occured on a custom carrier board? And, mounting the CM4 under test on an CM4IO gives this ''faint buzz'?

Follow the hardware checklist from the CM4 users manual (using the CM4IO) and report baxk how far you get.
Then do the same with your custom board.

drmatt
Posts: 37
Joined: Sat Sep 07, 2019 3:47 am

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 6:04 am

so the problem occured on a custom carrier board? And, mounting the CM4 under test on an CM4IO gives this ''faint buzz'?
Yes and yes (though maybe more of a higher pitch whine than a buzz). To clarify, the board was manufactured by a third party and I don't have the knowledge or equipment to do much testing beyond probing pins.

I was able to confirm, with a separate working CM4, that the CM4IO board is still functional. The custom carrier board may be toast though, and probably the original CM4 as well.
Follow the hardware checklist from the CM4 users manual (using the CM4IO) and report baxk how far you get.
Then do the same with your custom board.
This may be obvious, but where can the 3.3v and 1.8v rails be accessed? Are there test points on the IO board or on the compute module itself?

The CM4 can be replaced, so if it's dead at this point, we're more interested in whether any data can be retrieved from its eMMC somehow.

aBUGSworstnightmare
Posts: 3232
Joined: Tue Jun 30, 2015 1:35 pm

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 6:12 am

drmatt wrote:
Thu Jun 17, 2021 6:04 am
so the problem occured on a custom carrier board? And, mounting the CM4 under test on an CM4IO gives this ''faint buzz'?
Yes and yes (though maybe more of a higher pitch whine than a buzz). To clarify, the board was manufactured by a third party and I don't have the knowledge or equipment to do much testing beyond probing pins.

I was able to confirm, with a separate working CM4, that the CM4IO board is still functional. The custom carrier board may be toast though, and probably the original CM4 as well.
Follow the hardware checklist from the CM4 users manual (using the CM4IO) and report baxk how far you get.
Then do the same with your custom board.
This may be obvious, but where can the 3.3v and 1.8v rails be accessed? Are there test points on the IO board or on the compute module itself?

The CM4 can be replaced, so if it's dead at this point, we're more interested in whether any data can be retrieved from its eMMC somehow.
CM4IO schematics (from the users manual) and the KiCAD files will allow you to figure out where to measure the voltages (i.e. 3.3V is on pin1 of the 40pin GPIO header).

How far do you get in the troubleshooting section? This will let you now if there is a chance to access the eMMC.

In case the CM4 git damaged you will want to figure out the root cause. Simply replacing it - and killing the next one - might not be a good idea.

User avatar
Gavinmc42
Posts: 5884
Joined: Wed Aug 28, 2013 3:31 am

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 6:45 am

Might do to invent a CM4 PCB breakout board to test motherboards before plugging in a CM4.

I will be doing my testing on the cheapest 1GB lite if I can get some ;)
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

drmatt
Posts: 37
Joined: Sat Sep 07, 2019 3:47 am

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 3:10 pm

CM4IO schematics (from the users manual) and the KiCAD files will allow you to figure out where to measure the voltages (i.e. 3.3V is on pin1 of the 40pin GPIO header).

How far do you get in the troubleshooting section? This will let you now if there is a chance to access the eMMC.
Thank you. Here's what I managed to find out:
  • with GLOBAL_EN pulled low, +5 supply seems good.
  • GLOBAL_EN rises to 5v when not pulled low.
  • only ~0.1v measured on 3.3v rail (GPIO pin 1) with GLOBAL_EN low or high.
  • don't know where to check 1.8v rail.
  • RUN_PG is low after removing the pulldown on GLOBAL_EN.
  • no LED activity at all on the board.

aBUGSworstnightmare
Posts: 3232
Joined: Tue Jun 30, 2015 1:35 pm

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 4:18 pm

drmatt wrote:
Thu Jun 17, 2021 3:10 pm
CM4IO schematics (from the users manual) and the KiCAD files will allow you to figure out where to measure the voltages (i.e. 3.3V is on pin1 of the 40pin GPIO header).

How far do you get in the troubleshooting section? This will let you now if there is a chance to access the eMMC.
Thank you. Here's what I managed to find out:
  • with GLOBAL_EN pulled low, +5 supply seems good.
  • GLOBAL_EN rises to 5v when not pulled low.
  • only ~0.1v measured on 3.3v rail (GPIO pin 1) with GLOBAL_EN low or high.
  • don't know where to check 1.8v rail.
  • RUN_PG is low after removing the pulldown on GLOBAL_EN.
  • no LED activity at all on the board.
you can measure 1.8V at position of R4. R4 is not mounted; it is used in case GPIO_VREF is to be configured for 1.8V. But if memory serves me right it is below the CM4.

Anyhow, as your 3.3V rail is dead the PMIC on the CM4 (or any required component) has died --> your CM4 seems bricked beyond repairability (as at least the PMIC can't be sourced).

drmatt
Posts: 37
Joined: Sat Sep 07, 2019 3:47 am

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 9:44 pm

Anyhow, as your 3.3V rail is dead the PMIC on the CM4 (or any required component) has died --> your CM4 seems bricked beyond repairability (as at least the PMIC can't be sourced).
Thanks, I was afraid of that but not surprised. I suppose there's no way to somehow externally access the eMMC onboard if this is the case?

trejan
Posts: 3357
Joined: Tue Jul 02, 2019 2:28 pm

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 10:02 pm

drmatt wrote:
Thu Jun 17, 2021 9:44 pm
Thanks, I was afraid of that but not surprised. I suppose there's no way to somehow externally access the eMMC onboard if this is the case?
Not easily. You've got two problems. First is powering the eMMC since the PMIC is dead. The second is getting to the data lines.

There should be a set of unpopulated resistor positions that jumper the eMMC interface to the CM4 connector. They're only fitted on the flashless variants so you should be able to tap into those. Nope. I'm wrong.
Last edited by trejan on Thu Jun 17, 2021 11:16 pm, edited 1 time in total.

drmatt
Posts: 37
Joined: Sat Sep 07, 2019 3:47 am

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 11:10 pm

Not easily. You've got two problems. First is powering the eMMC since the PMIC is dead. The second is getting to the data lines. There should be a set of unpopulated resistor positions that jumper the eMMC interface to the CM4 connector. They're only fitted on the flashless variants so you should be able to tap into those.
Got it, we'll look for those and give it a try (what have we got to lose?). Many thanks!

dp11
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 565
Joined: Thu Dec 29, 2011 5:46 pm

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 11:11 pm

trejan wrote:
Thu Jun 17, 2021 10:02 pm
There should be a set of unpopulated resistor positions that jumper the eMMC interface to the CM4 connector. They're only fitted on the flashless variants so you should be able to tap into those.
No there isn't.

trejan
Posts: 3357
Joined: Tue Jul 02, 2019 2:28 pm

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 11:16 pm

dp11 wrote:
Thu Jun 17, 2021 11:11 pm
trejan wrote:
Thu Jun 17, 2021 10:02 pm
There should be a set of unpopulated resistor positions that jumper the eMMC interface to the CM4 connector. They're only fitted on the flashless variants so you should be able to tap into those.
No there isn't.
Ah. It is different on the CM4? I thought it was the same as the previous CMs. No schematics so hard to tell from squinting at the CM4 I've got.

dp11
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 565
Joined: Thu Dec 29, 2011 5:46 pm

Re: cm4 dead - any recovery options?

Thu Jun 17, 2021 11:51 pm

CM4! =CM3

User avatar
Gavinmc42
Posts: 5884
Joined: Wed Aug 28, 2013 3:31 am

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 1:44 am

only ~0.1v measured on 3.3v rail (GPIO pin 1) with GLOBAL_EN low or high.
A finger on the power chip can tell you if something is shorted.

Power off and use a multimeter and check that the inductor for the 3V3 is not open circuit.
Ferrites are brittle, it could have cracked when pushed onto the motherboard etc.
Might even be a dry solder joint.
Check the others while you are at it.

There is a sequence the power supply chip goes through via state machine logic.
If an early stage does not get to the right voltage it shuts down?

There is still a chance to get it working ;)
If the problem was over voltage that's a "bin it" solution.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

drmatt
Posts: 37
Joined: Sat Sep 07, 2019 3:47 am

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 4:49 am

A finger on the power chip can tell you if something is shorted.
Thanks for the tips. This is the PMIC chip labeled MxL7704? If so, it does feel a bit warm. I measured roughly 45C after about 30 seconds of being powered on, but not sure what the normal temperature should be.
Power off and use a multimeter and check that the inductor for the 3V3 is not open circuit.
I think you're referring to the four inductors surrounding the PMIC chip? I don't know which one corresponds to the 3v3 rail, but I did my best to test across all four of them with the equipment I have. They all read near zero resistance - the two close to the edge of the board with high confidence, and the other two with slightly lower confidence due to the size of the probes I'm using and the possibility they were contacting something else nearby. Not much room in there to get at the connections.

What does this tell us, and what might be next steps?

aBUGSworstnightmare
Posts: 3232
Joined: Tue Jun 30, 2015 1:35 pm

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 5:17 am

Gavinmc42 wrote:
Fri Jun 18, 2021 1:44 am
There is still a chance to get it working ;)
...
why don't you ket us know how?
Sure, one can try to backfeed the PMIC and try to access the CM4 from a PC to retrieve those files from the eMMC, but I'd expect it's the same effort compared to re-implementing the changes.

If you don't have any backup at all this is a lesson learned for you! Clone the uSD/eMMC on.a regular basis when compiling on the RPI or start crosscompiling.

User avatar
Gavinmc42
Posts: 5884
Joined: Wed Aug 28, 2013 3:31 am

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 5:32 am

why don't you let us know how?
Need to identify the problem first.

Does it work on the official I/O board?
Is it a fault on the I/O board?
Is it the power supply?
Is it the CPU?
Is it the eMMC?
Wireless module?
Ethernet chip?

The data in the eMMC can be recovered if that chip not the problem.
Does JTAG on the SoC talk to the eMMC port?

I just confirmed a CM4 will boot from USB A if the eMMC is blank.
I don't need a Lite version yet, if USB A booting is good enough for my testing.
Saves messing with Bootloader/USBboot.

It is all a learning process.
Fault finding new hardware, firmware, software is part of the learning.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 4005
Joined: Sat Aug 18, 2012 2:33 pm

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 5:40 am

Gavinmc42 wrote:
Fri Jun 18, 2021 5:32 am
Does JTAG on the SoC talk to the eMMC port?
there are at least 2 jtag ports i'm aware of

Image

i think the missing connector in the top-left is the VPU jtag, that should (in theory) be able to do everything, but broadcom wont release the documentation on how
the arm jtag is muxed into the main gpio array, and is enabled by the firmware during boot

for the VPU jtag to even function, the soc must be getting a core voltage, which would require the PMIC to be working at some level
for arm jtag to function, the system needs to be booting enough to run the official firmware

once either jtag is working, you could then access the emmc via the debug port
but rpiboot and msd/start4.elf may be simpler/faster, if its capable of booting that far

User avatar
Gavinmc42
Posts: 5884
Joined: Wed Aug 28, 2013 3:31 am

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 5:51 am

Had forgotten USB boot mode, that does need power chip and SoC working.
Once in MSD mode the eMMC can be read even if there is file corruption?

Isn't there a recent new feature in Linux that allows Home/Swap to be on a separate drive?
Might be handy for those big compiles.
I have worn out a few uSD's compiling even on Pi3B+ which really needed the bigger swap.
I really don't want to wear out the eMMC and things like this concern me.

How to save the wear on eMMC when developing?
Use/boot from PCIe NVMe, compile on PCIe to USB 3 HDD
Need to figure this out as I am about to do dual camera coding.
8GB/16GB eMMC are more expensive and 16GB is not really enough for developing.
Those dependencies fill storage quick.

I don't want to end up in the same situation as OP.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 4005
Joined: Sat Aug 18, 2012 2:33 pm

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 5:58 am

Gavinmc42 wrote:
Fri Jun 18, 2021 5:51 am
Once in MSD mode the eMMC can be read even if there is file corruption?
when in MSD mode, it just exploses the eMMC as a usb MSD device
so its up to whatever os is mounting that, to handle the corruption, or to just image it with ddrescue and fix it later

assuming the MSD emulation is implemented sanely, and can properly relay IO errors onwards
Gavinmc42 wrote:
Fri Jun 18, 2021 5:51 am
Isn't there a recent new feature in Linux that allows Home/Swap to be on a separate drive?
that has been an option in linux for decades

User avatar
Gavinmc42
Posts: 5884
Joined: Wed Aug 28, 2013 3:31 am

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 6:30 am

that has been an option in linux for decades
I think this was also for those dependency files as well.
Not sure if I saw it in Kernel or Debian news or exactly what it was called.

This was months ago and it was a coming feature.
Separating the OS from all those library files is what I sort of grasped at the time.
Was it a 5.12 kernel feature?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

aBUGSworstnightmare
Posts: 3232
Joined: Tue Jun 30, 2015 1:35 pm

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 6:54 am

Gavinmc42 wrote:
Fri Jun 18, 2021 5:32 am
why don't you let us know how?
Need to identify the problem first.

Does it work on the official I/O board?
Is it a fault on the I/O board?
Is it the power supply?
Is it the CPU?
Is it the eMMC?
Wireless module?
Ethernet chip?

The data in the eMMC can be recovered if that chip not the problem.
Does JTAG on the SoC talk to the eMMC port?

I just confirmed a CM4 will boot from USB A if the eMMC is blank.
I don't need a Lite version yet, if USB A booting is good enough for my testing.
Saves messing with Bootloader/USBboot.

It is all a learning process.
Fault finding new hardware, firmware, software is part of the learning.
DUT is not working on the CM4Io, PMIC is dead so how to supply power?
This has been diecussed already, so the only possible solution which comes to my mind would be removing the PMIC components, supplying power via JTAG connector (if this is able to supply all required voltages which I would not expect) and then keep fingers crossed that SOC reacts.

The above requires decent knowledge and tools on hands and consumes time.
OP needs to answer the question is this time is well spend or if it's better to start over.
If OP has no backup of the files in question (not even an older version) then he/she needs to spend the time on this task even if the result is zero.

Question to OP: Do you have another CM4 where you can meaure the PMiC voltages? Knowing where to inject them would allow you to remove the PCIM, inject the voltages (supllied from bench PSU) and try to boot the CM4.
Note: use at own risk! And, keep in mind the 'standard' MxL7704 is a 5 output PMIC (https://www.maxlinear.com/product/power ... cs/mxl7704) so you maybe need some more PSUs

cleverca22
Posts: 4005
Joined: Sat Aug 18, 2012 2:33 pm

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 7:17 am

Gavinmc42 wrote:
Fri Jun 18, 2021 6:30 am
that has been an option in linux for decades
I think this was also for those dependency files as well.
Not sure if I saw it in Kernel or Debian news or exactly what it was called.

This was months ago and it was a coming feature.
Separating the OS from all those library files is what I sort of grasped at the time.
Was it a 5.12 kernel feature?
having things like swap and /home on different devices has been a feature of linux for as long as i can remember, at least since the 2.6 era

its purely up to the distro and installer to give the user a choice to configure it easily, or for a skilled user to modify the system after installation

User avatar
Gavinmc42
Posts: 5884
Joined: Wed Aug 28, 2013 3:31 am

Re: cm4 dead - any recovery options?

Fri Jun 18, 2021 7:23 am

skilled user
Not me then.

Time to explore other options.
HDD 3.5" 1TB might do me for now.
Will a Pi boot from that?
Seem to remember PC's used to boot from spinning things :lol:


If the eMMC is still alive, take it off, solder to adapter and read it.
https://emmc-pro.com/

How many pins does an eMMC chip use?
8bits of data, power, CMD, Clk and reset?
Could even program a Pico to read that.

I have used Flashrom on Pi's to program SPI flash, will it do eMMC?
https://www.flashrom.org/Downloads
Does not look like it, yet?

The CM4's are my first eMMC parts.
More to learn.

If they can program, they can read?
https://www.xeltek.com/resources/techni ... ogrammers/

Hey, eMMC, isn't that like NAND flash, SMI GPIO mode?
1.8V I/O?
Use a modified CM4 with 1.8V I/O on a CM4 I/O board to read the eMMC.
Only need to solder a dozen tiny wires to the removed eMMC.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Return to “Compute Module”