GO-IOT
Posts: 3
Joined: Mon Nov 27, 2017 3:45 pm

Re: CM3 always crashes within minutes under load

Wed Jun 06, 2018 3:54 pm

Hi Gordon

We are now in "Production Stop" because of the CM3 overheating

Standard Raspian - No Heatsink or FAN are possible

Please help asap

KR

PH

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 9:07 am

I am facing exact same problems as described by previous posters. CM3 freezes when put under stress.

First i thought it was in the python code, then tried with gdb to get stack trace. It kills my gdb with segfault. Within second console hangs, and cannot ssh anymore into it.

When we dont put it under stress, no problems for ages.

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 9:20 am

cm3-1.jpg
cm3-1.jpg (215.67 KiB) Viewed 1435 times
cm3-2.jpg
cm3-2.jpg (245.42 KiB) Viewed 1435 times

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 9:35 am

[email protected]:~# vcgencmd get_config int
aphy_params_current=819
arm_freq=1200
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=400
desired_osc_freq=0x36ee80
disable_commandline_tags=2
disable_l2cache=1
disable_splash=1
display_hdmi_rotate=-1
display_lcd_rotate=-1
dphy_params_current=547
dtdebug=1
enable_uart=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
hdmi_force_cec_address=65535
hdmi_force_hotplug=1
ignore_lcd=1
init_uart_clock=0x2dc6c00
over_voltage_avs=0x30d40
overscan_bottom=48
overscan_left=48
overscan_right=48
overscan_top=48
pause_burst_frames=1
program_serial_random=1
sdram_freq=450

[email protected]:~# vcgencmd get_config str
device_tree=-

[email protected]:~# vcgencmd otp_dump
08:00000000
09:00000000
10:00000000
11:00000000
12:00000000
13:00000000
14:00000000
15:00000000
16:00280000
17:1220000a
18:1220000a
19:ffffffff
20:ffffffff
21:ffffffff
22:ffffffff
23:ffffffff
24:ffffffff
25:ffffffff
26:ffffffff
27:00002727
28:2f90922a
29:d06f6dd5
30:00a220a0
31:00000000
32:00000000
33:00000000
34:00000000
35:00000000
36:00000000
37:00000000
38:00000000
39:00000000
40:00000000
41:00000000
42:00000000
43:00000000
44:00000000
45:00000000
46:00000000
47:00000000
48:00000000
49:00000000
50:00000000
51:00000000
52:00000000
53:00000000
54:00000000
55:00000000
56:00000000
57:00000000
58:00000000
59:00000000
60:00000000
61:00000000
62:00000000
63:00000000
64:00000000
65:00000000
66:00000000

Linux 01-1234 4.14.34-v7+ #1110 SMP Mon Apr 16 15:18:51 BST 2018 armv7l GNU/Linux

[email protected]:~# free
total used free shared buff/cache available
Mem: 949460 251568 238088 119672 459804 528432
Swap: 0 0 0



Device tree

Code: Select all

/dts-v1/;
/plugin/;

/ {
  videocore {

    pins_cm3 {

      pin_config {

        [email protected] {
          polarity = "active_high";
          termination = "pull_down";
          startup_state = "inactive";
          function = "input";
        }; // pin

        // BANK 0 - USER GPIO //
        [email protected]  { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected]  { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected]  { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected]  { function = "i2c0";    termination = "pull_up";    }; // I2C 0 SDA
        [email protected]  { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected]  { function = "i2c0";    termination = "pull_up";    }; // I2C 0 SCL
        [email protected]  { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected]  { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected]  { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected]  { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "uart0";   termination = "no_pulling"; drive_strength_mA = < 8 >; }; // UART0 TX
        [email protected] { function = "uart0";   termination = "pull_up";    drive_strength_mA = < 8 >; }; // UART0 RX


        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "output";  termination = "pull_down"; startup_state = "active"; }; // LAN_RUN
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE

        // BANK 1 - USER GPIO//
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE WAS INPUT NO PULL
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE WAS INPUT NO PULL
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_up";    }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE
        [email protected] { function = "input";   termination = "pull_down";  }; // DEFAULT STATE

        // BANK 2 - DON'T TOUCH UNLESS YOU KNOW WHAT YOU'RE DOING //
        [email protected] { function = "uart1";   termination = "no_pulling"; drive_strength_mA = < 8 >; }; // UART1 TX
        [email protected] { function = "uart1";   termination = "pull_up";    drive_strength_mA = < 8 >; }; // UART1 RX

        [email protected] { function = "pwm";    termination = "no_pulling"; drive_strength_mA = < 16 >; }; // Right audio
        [email protected] { function = "pwm";    termination = "no_pulling"; drive_strength_mA = < 16 >; }; // Left audio
        [email protected] { function = "input";  termination = "no_pulling";    }; // HDMI_Hotplug

      }; // pin_config

      pin_defines {
        [email protected]_ENABLE { type = "external"; number = <1>; };
        [email protected]_SDA { type = "internal"; number = <3>; };
        [email protected]_SCL {type = "internal"; number = <5>; };
        [email protected]_RUN { type = "internal"; number = <21>;}; //LAN RESET_N
        [email protected]_CONTROL_ATTACHED { type = "internal"; number = <88>;}; // HDMI_Hotplug

      }; // pin_defines

    }; // pins_cm3

  }; // videocore

};

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 9:45 am

and a catched freeze
2018-06-08 11_40_55-Instellingen.png
2018-06-08 11_40_55-Instellingen.png (4.36 KiB) Viewed 1425 times
2018-06-08 11_41_07-01-1234_ ~.png
2018-06-08 11_41_07-01-1234_ ~.png (23.47 KiB) Viewed 1425 times

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 10:10 am

Update 1

The post of 6by9 suggest floating pins (SMPS_SDA and SMPS_SCL)

I added them to my device tree, big change is that voltage is not 1.4V but gets stable at volt=1.2438V
2018-06-08 12_10_11-01-1234_ ~.png
2018-06-08 12_10_11-01-1234_ ~.png (4.75 KiB) Viewed 1410 times
update 2

since adjusted device tree, its stressing more than 30 min so far, and still a live. Before it freezed within minutes.
Last edited by Mikeynl on Fri Jun 08, 2018 10:38 am, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7135
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 10:38 am

Mikeynl wrote:
Fri Jun 08, 2018 10:10 am
Small update:

The post of 6by9 suggest floating pins (SMPS_SDA and SMPS_SCL)
As just responded via PM, please ensure that if your dt-blob.bin is based off minimal-cm-dt-blob.dts, that it is after PR 812 was merged (5th Feb). Without that then you have no SMPS control.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 10:41 am

I see now indeed that my minimal file is different than what was on github...

But very strange, all docs and info would be from raspberry pi website, so have no idea where i got this file from .

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23338
Joined: Sat Jul 30, 2011 7:41 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 12:07 pm

Mikeynl wrote:
Fri Jun 08, 2018 10:41 am
I see now indeed that my minimal file is different than what was on github...

But very strange, all docs and info would be from raspberry pi website, so have no idea where i got this file from .
It's possible that you got the file before the PR that fixed the issue, or simply moved to the CM3 without updating the blob?

Can I suggest anyone seeing the overheating issue double check their dtblob to ensure they have the correct settings, as shown above. Not having these set will result in the chip overvolting, which WILL leading to overheating. Can you please report back here with results so we know whether this fixes the overheating being reported.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Fri Jun 08, 2018 1:58 pm

jamesh wrote:
Fri Jun 08, 2018 12:07 pm

It's possible that you got the file before the PR that fixed the issue, or simply moved to the CM3 without updating the blob?
100% not. We just started 6 weeks ago with developing cm3 board

Anyway, problem solved :)!

Mikeynl
Posts: 27
Joined: Sat Nov 11, 2017 1:36 pm

Re: CM3 always crashes within minutes under load

Sun Jun 10, 2018 8:45 pm

jamesh wrote:
Fri Jun 08, 2018 12:07 pm
Can you please report back here with results so we know whether this fixes the overheating being reported.
Hi James, when running full blast it hangs around 80 degrees. We ordered dim size heatsinks to glue on top. For us heating was not the problem, only freezing of the cm3. That is now 100% solved with adjustment of device tree file.

metalmajor
Posts: 34
Joined: Fri Oct 20, 2017 9:40 pm

Re: CM3 always crashes within minutes under load

Mon Jun 18, 2018 2:32 pm

For the other blobs there is no problem, correct? I am using the dt-blob-dualcam.dts file as a basis.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23338
Joined: Sat Jul 30, 2011 7:41 pm

Re: CM3 always crashes within minutes under load

Mon Jun 18, 2018 4:08 pm

metalmajor wrote:
Mon Jun 18, 2018 2:32 pm
For the other blobs there is no problem, correct? I am using the dt-blob-dualcam.dts file as a basis.
As long as you have the fix from in here https://github.com/raspberrypi/documentation/pull/812 you should be OK.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

metalmajor
Posts: 34
Joined: Fri Oct 20, 2017 9:40 pm

Re: CM3 always crashes within minutes under load

Mon Jun 18, 2018 5:54 pm

yes, I meant that people starting from the other ".dts" files were already safe from this bug? As far as I know, it seems the dt-blob-dualcam.dts contains these lines already so I probably should be okay. I also ran stress-ng and the system kept running ok when doing the cpu tests, so...

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23338
Joined: Sat Jul 30, 2011 7:41 pm

Re: CM3 always crashes within minutes under load

Mon Jun 18, 2018 7:38 pm

metalmajor wrote:
Mon Jun 18, 2018 5:54 pm
yes, I meant that people starting from the other ".dts" files were already safe from this bug? As far as I know, it seems the dt-blob-dualcam.dts contains these lines already so I probably should be okay. I also ran stress-ng and the system kept running ok when doing the cpu tests, so...
If the lines are there, the power controller should be working, so thinks shod indeed work fine.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Return to “Compute Module”