max.w
Posts: 3
Joined: Sat Feb 25, 2017 2:40 pm

CM3 always crashes within minutes under load

Sat Feb 25, 2017 3:03 pm

Hi everyone,

I'm currently facing a huge problem - likely thermal - with the new CM3 module:
From the point I put the CPU under load, it only takes minutes until the CM3 crashes. Always. It either freezes completely or first shows a kernel Oops - likely related to corrupted memory.
I tested this with a CM3 module and two different CM3L modules, both in the original CM3-IO-board and my own PCB. I'm using the latest version of Raspbian Lite.

I already tried adding different sized heat sinks on the CPU chip, but to no avail. The only way to keep the CM3 running is by adding a heat sink and a fan that blows air both below and above the CM3 module (just cooling the "top side" heat sink is not enough). Thus, I'm by now pretty certain that there is a thermal issue here.

Any pointers would be greatly appreciated as I had intended to use the CM3 for a major project :(

The easiest way to reproduce the issue is to utilize the CM3 CPU, e.g. run

Code: Select all

cat /dev/zero > /dev/null | cat /dev/zero > /dev/null
or if you want faster "results" utilize all cores:

Code: Select all

cat /dev/zero > /dev/null | cat /dev/zero > /dev/null | cat /dev/zero > /dev/null | cat /dev/zero > /dev/null
Please note that you can use any CPU intense application.

Crashes happen for CPU temperatures, as reported by

Code: Select all

vcgencmd measure_temp
as "low" as 55'C.

Update, amendment: Running the identical applications from the identical SD card on the RPi3 does not result in crashes.

If this effects other as well - I don't see how it couldn't: How could this slip into the final product?!

Best regards,
Max

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20291
Joined: Sat Jul 30, 2011 7:41 pm

Re: CM3 always crashes within minutes under load

Sat Feb 25, 2017 4:45 pm

Hmm, should not be crashing at 55, should be good for at least 80.

My suspicion is that this is a duff CM3, do you have another you can test with?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

max.w
Posts: 3
Joined: Sat Feb 25, 2017 2:40 pm

Re: CM3 always crashes within minutes under load

Sat Feb 25, 2017 5:10 pm

I already tried three different CM3 modules.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20291
Joined: Sat Jul 30, 2011 7:41 pm

Re: CM3 always crashes within minutes under load

Sat Feb 25, 2017 5:39 pm

max.w wrote:I already tried three different CM3 modules.
Hmm. Gordon might have to butt in on this one.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

MichaR
Posts: 16
Joined: Thu Mar 10, 2016 5:49 am
Location: Berlin / Germany

Re: CM3 always crashes within minutes under load

Mon Feb 27, 2017 5:10 am

Morning,

I have the same problem with the two CM3L I own. A small heatsink on top of the CPU solves the problem for me.

best regards
Micha

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1306
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 always crashes within minutes under load

Mon Feb 27, 2017 9:59 am

Is this using a standard Raspbian image with no changes?

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

MichaR
Posts: 16
Joined: Thu Mar 10, 2016 5:49 am
Location: Berlin / Germany

Re: CM3 always crashes within minutes under load

Mon Feb 27, 2017 12:17 pm

Tested with Raspbian, OpenElec and PiLFS. Only modification is that the 24 bit DPI-Interface is enabled and configured.

Btw. I think, the core temperature is higher than the 55°. On my two modules the CPU has an outer temperature of more than 40°C. So the core temp. is at least 65°C or even more.

best regards
Micha

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1306
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 always crashes within minutes under load

Mon Feb 27, 2017 4:59 pm

Can you retest with a standard Raspbian release with nothing else enabled. It should not be a problem up to 85 degrees, which is when the throttling is started.

It may also be useful to do

vcgencmd measure_volts
vcgencmd measure_clock arm

while you are running the test...
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1829
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM3 always crashes within minutes under load

Mon Feb 27, 2017 5:02 pm

"vcgencmd get_throttled" will report if the chip has been throttled due to over-temperature or under-voltage. Non-zero is bad.

max.w
Posts: 3
Joined: Sat Feb 25, 2017 2:40 pm

Re: CM3 always crashes within minutes under load

Tue Feb 28, 2017 7:38 am

Hi Gordon, hi Phil,

thank you very much for looking into this.

I'm confused now. With the very latest Raspbian (2017-02-16) my CM3 is working stable. Identical CM3, identical IO-Board-3, identical environment conditions (i.e. cooling).

The situation with any older image was:
  • Runs stable with a huge heat sink or a big cooling fan -> vcgencmd measure_temp did not report above 50'C in this configuration
  • Does not run stable without major cooling effort (usually Kernel Oops or crash around 65-70'C)
With the new image
  • Runs stable - even without cooling. At ~80'C the CPU throttles, but still continues to run stable.
  • Runs stable with a minimum amount of cooling (small heat sink on CPU chip). No throttling at ~70'C
The release_notes.txt do mention an update of kernel and firmware. Could you please give a hint as to what has been changed? I'm really baffled what kind of change can cause the behavior, i.e. how can a kernel/firmware update fix issues related to overheating?

Cheers,
Max

Will_H
Posts: 2
Joined: Wed Mar 01, 2017 2:29 am

Re: CM3 always crashes within minutes under load

Wed Mar 01, 2017 6:01 am

I'm experiencing some issues with the CM3 16GB (NEC) that sound like they may just be related to this.

I've been testing the latest Libreelec and OSMC releases with updates. For some reason I can't search or find any networked devices (SMB, UPnP etc.) on the CM3 when I try to add to "browse" them for Pictures, Videos etc. I just get the wait animation, locked UI, and after several minutes get a "Connection timed out" error. However during this locked UI period the over-temperature icon is displayed and the CPU gets pretty toasty. Network connectivity to the CM3 seems to be otherwise working fine since I'm able to update, install packages, play Youtube etc.

During this lockup:
"vcgencmd measure_temp" returns 85'C
"vcgencmd get_throttled" returns 0x20002
"vcgencmd measure_volts" returns 1.3188V

When not locked up:
"vcgencmd measure_temp" returns 62'C
"vcgencmd get_throttled" returns 0x60000
"vcgencmd measure_volts" returns 1.2000V

The CM3 isn't heat sinked or actively cooled, but is vented.

If I restart the CM3 soon after encountering this condition I find I often get partial boots and the system will freeze at random points (e.g. freeze at the Libreelec boot logo screen). It seems that if I allow it longer in the power-off state then I have a higher success of a complete boot.

I've verified this on 3 CM3 16GB boards so far.

The same versions (but obviously RPi1 builds) running on a CM1 on exactly the same hardware run fine and I can access networked devices immediately.

I'm happy to test any suggestions.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1306
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 always crashes within minutes under load

Wed Mar 01, 2017 9:31 am

Can you confirm the system that you have bought? Where did you get it?

Just trying to confirm information about the device.

Thanks

Gordon Hollingworth
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5566
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: CM3 always crashes within minutes under load

Wed Mar 01, 2017 10:43 am

Can you also confirm that there isn't a dt-blob.bin in /boot?
There have been a couple floating around which have been missing SMPS_SDA and SMPS_SCL defines that were causing issues for me.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

hen
Posts: 11
Joined: Tue Apr 11, 2017 8:39 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 11:10 am

Hi,
I think I'm facing a similar problem, maybe more subtle, with some CM3 modules (tested on both original I/O board V3 and my custom PCB).

On those modules, when I try to build the libraries downloaded from github.com/raspberrypy/userland.git, usually just after some seconds I start to get various kinds of errors, e.g. "compiler error: segmentation fault". Then, after a while, the CM likely freezes.

If I lower the CPU frequency with "arm_freq=600" in /boot/config.txt, I get no errors anymore.

I've tried using an old raspbian image I was using on RPi 2B, with a custom dt-blob.bin that "complies" with the latest updates made by 6by9 found on "minimal-cm-dt-blob.dts" (so SMPS_SCL and SMPS_SDA are correctly defined); then I've also tried with latest (2018-04-18) Raspbian Stretch lite image without any modification. Behavior was the same.

Then I've tried to switch power supply (for I/O board) between a 2A wall cube and a PC USB port, no change.
My custom PCB uses the same 1.8/3.3V PAM2306AYPKE regulator of the I/O board V3.

On one board that shows issues, I've tried also the tests with "cat /dev/zero > /dev/null" by max.w. After a while, I can see the red thermometer on the upper right side screen, but the board seems to run without problems (I've kept this test running for for several minutes).

The worrying thing is that all of the six modules I have at hand now does not pass my "userland build" test without lowering the CPU speed.

Do you have any hints? May I have stumbled upon a faulty batch of CMs?
Thank you

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1306
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 11:45 am

Can you get the output from:

vcgencmd get_config int
vcgencmd get_config strings

also

vcgencmd otp_dump
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

hen
Posts: 11
Joined: Tue Apr 11, 2017 8:39 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 1:22 pm

Hi, thanks for your reply.

This is the ouputs from two CMs, flashed with latest raspbian lite Scratch image of 2018-04-18.

The first one:

[email protected]:~ $ vcgencmd get_config int
aphy_params_current=819
arm_freq=1200
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=400
desired_osc_freq=0x36ee80
disable_commandline_tags=2
disable_l2cache=1
display_hdmi_rotate=-1
display_lcd_rotate=-1
dphy_params_current=547
enable_uart=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
hdmi_force_cec_address=65535
hdmi_force_hotplug=1
ignore_lcd=1
init_uart_clock=0x2dc6c00
over_voltage_avs=0x13d62
pause_burst_frames=1
program_serial_random=1
sdram_freq=450

[email protected]:~ $ vcgencmd get_config str
device_tree=-

[email protected]:~ $ vcgencmd otp_dump
08:00000000
09:00000000
10:00000000
11:00000000
12:00000000
13:00000000
14:00000000
15:00000000
16:00280000
17:1220000a
18:1220000a
19:ffffffff
20:ffffffff
21:ffffffff
22:ffffffff
23:ffffffff
24:ffffffff
25:ffffffff
26:ffffffff
27:00002727
28:dea32c74
29:215cd38b
30:00a020a0
31:00000000
32:00000000
33:00000000
34:00000000
35:00000000
36:00000000
37:00000000
38:00000000
39:00000000
40:00000000
41:00000000
42:00000000
43:00000000
44:00000001
45:00000000
46:00000000
47:00000000
48:00000000
49:00000000
50:00000000
51:00000000
52:00000000
53:00000000
54:00000000
55:00000000
56:00000000
57:00000000
58:00000000
59:00000000
60:00000000
61:00000000
62:00000000
63:00000000
64:00000000
65:00000000
66:00000000

And the second one:

[email protected]:~ $ vcgencmd get_config int
aphy_params_current=819
arm_freq=1200
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=400
desired_osc_freq=0x36ee80
disable_commandline_tags=2
disable_l2cache=1
display_hdmi_rotate=-1
display_lcd_rotate=-1
dphy_params_current=547
enable_uart=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
hdmi_force_cec_address=65535
ignore_lcd=1
init_uart_clock=0x2dc6c00
over_voltage_avs=0x10c8e
pause_burst_frames=1
program_serial_random=1
sdram_freq=450

[email protected]:~ $ vcgencmd get_config str
device_tree=-

[email protected]:~ $ vcgencmd otp_dump
08:00000000
09:00000000
10:00000000
11:00000000
12:00000000
13:00000000
14:00000000
15:00000000
16:00280000
17:1220000a
18:1220000a
19:ffffffff
20:ffffffff
21:ffffffff
22:ffffffff
23:ffffffff
24:ffffffff
25:ffffffff
26:ffffffff
27:00002727
28:91e34c5f
29:6e1cb3a0
30:00a020a0
31:00000000
32:00000000
33:00000000
34:00000000
35:00000000
36:00000000
37:00000000
38:00000000
39:00000000
40:00000000
41:00000000
42:00000000
43:00000000
44:00000001
45:00000000
46:00000000
47:00000000
48:00000000
49:00000000
50:00000000
51:00000000
52:00000000
53:00000000
54:00000000
55:00000000
56:00000000
57:00000000
58:00000000
59:00000000
60:00000000
61:00000000
62:00000000
63:00000000
64:00000000
65:00000000
66:00000000

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1306
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 1:39 pm

Can you please take a photo of the 2D barcode on the device, it's clear that your devices are slightly slower than normal because you have some overvoltage bits set. But I just need to check what happened on the production line.

Thanks
Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

hen
Posts: 11
Joined: Tue Apr 11, 2017 8:39 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 1:56 pm

I can't see any barcode on the modules... where it should be placed?

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1306
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 2:28 pm

Ah, right yes... No barcode!

I'll just get them to trace it through the serial number...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

hen
Posts: 11
Joined: Tue Apr 11, 2017 8:39 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 2:51 pm

Thank you.
I've attached the photos of one of the modules, and the barcode on the box, if helpful.

Do you think that lowering the speed to 600MHz could be a solution to make these modules perform reliably?
Attachments
IMG_20180525_162253278.jpg
IMG_20180525_162253278.jpg (121.68 KiB) Viewed 1177 times
IMG_20180525_162231513.jpg
IMG_20180525_162231513.jpg (151.8 KiB) Viewed 1177 times
IMG_20180525_162104959.jpg
IMG_20180525_162104959.jpg (170.85 KiB) Viewed 1177 times

hen
Posts: 11
Joined: Tue Apr 11, 2017 8:39 am

Re: CM3 always crashes within minutes under load

Fri May 25, 2018 3:16 pm

Don't know if it could be useful.. this is the vcgencmd ouputs of an older CM with the other (older) raspbian image I was using to test. This CM works OK also at full speed:

[email protected]:~ $ vcgencmd get_config int
arm_freq=1200
audio_pwm_mode=1
config_hdmi_boost=5
core_freq=400
desired_osc_freq=0x36ee80
disable_commandline_tags=2
disable_l2cache=1
enable_uart=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
hdmi_force_cec_address=65535
init_uart_clock=0x2dc6c00
lcd_framerate=60
over_voltage_avs=0x155cc
over_voltage_avs_boost=0x155cc
pause_burst_frames=1
program_serial_random=1
sdram_freq=450
temp_limit=85

[email protected]:~ $ vcgencmd get_config str
device_tree=-

[email protected]:~ $ vcgencmd otp_dump
08:00000000
09:00000000
10:00000000
11:00000000
12:00000000
13:00000000
14:00000000
15:00000000
16:00280000
17:1220000a
18:1220000a
19:ffffffff
20:ffffffff
21:ffffffff
22:ffffffff
23:ffffffff
24:ffffffff
25:ffffffff
26:ffffffff
27:00002727
28:85e2af01
29:7a1d50fe
30:00a020a0
31:00000000
32:00000000
33:00000000
34:00000000
35:00000000
36:00000000
37:00000000
38:00000000
39:00000000
40:00000000
41:00000000
42:00000000
43:00000000
44:00000000
45:00000000
46:00000000
47:00000000
48:00000000
49:00000000
50:00000000
51:00000000
52:00000000
53:00000000
54:00000000
55:00000000
56:00000000
57:00000000
58:00000000
59:00000000
60:00000000
61:00000000
62:00000000
63:00000000
64:00000000
65:00000000
66:00000000

John Westlake
Posts: 72
Joined: Thu Nov 09, 2017 4:34 am

Re: CM3 always crashes within minutes under load

Sun May 27, 2018 7:45 pm

We are facing the exact same issue and started a post here under the thread "Dynamic profile VCore difference between RPi3 & CM3"

viewtopic.php?f=98&t=212119

Basically we have no thermal issues running our software on the Rpi3 - but instant (within 1-2 minutes) overheat issues with CM3 - and as the original poster mentioned adding a heatsink does not help.

My last post on Mon May 14, 2018 7:56 pm had no response from RPi which is disappointing considering that this might be an issue with CM3 modules that's going to effect many users.

A couple of weeks ago I spoke with a Canadian manufacturer at the Munich HiFi show who are also using the CM3 and they are also experiencing the same overheat issue...

We would really appreciate some help with our issue - our production orders will be over 2K pcs per month if we can solve this overheat issue.
Last edited by John Westlake on Mon May 28, 2018 10:46 am, edited 2 times in total.

John Westlake
Posts: 72
Joined: Thu Nov 09, 2017 4:34 am

Re: CM3 always crashes within minutes under load

Sun May 27, 2018 8:00 pm

Also it has been suggested that the thermal performance difference between RPi3 and CM3 is due to the smaller PCB heatsinking area of the CM3 module - but this FAILS to explain why we still have thermal issues when a heatsink is added to the CM3 - I cannot believe that a BGA device mounted on standard FR4 PCB laminate (RPI3) has better thermal performance then a BGA with added heatsink but smaller PCB area.

hen
Posts: 11
Joined: Tue Apr 11, 2017 8:39 am

Re: CM3 always crashes within minutes under load

Mon Jun 04, 2018 10:59 am

Hi,
Do you have any update?

Thanks!

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1306
Joined: Sat Sep 10, 2011 11:43 am

Re: CM3 always crashes within minutes under load

Tue Jun 05, 2018 12:49 pm

Not yet, the factory has been on stop for a week... Just awaiting test logs...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

Return to “Compute Module”

Who is online

Users browsing this forum: gsh, wj_12 and 3 guests