User avatar
with ice cream
Posts: 161
Joined: Mon Jul 30, 2012 7:25 am

Kernel panic intepretation help needed

Mon Apr 29, 2019 8:49 am

Every once in a while (no discernible pattern) my Pi 3B+ (4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019) starts panicking:

Code: Select all

Apr 29 10:27:43 localhost kernel: [1027157.753964] INFO: task kworker/2:1:16365 blocked for more than 120 seconds.
Apr 29 10:27:43 localhost kernel: [1027157.753979]       Tainted: G         C      4.14.98-v7+ #1200
Apr 29 10:27:43 localhost kernel: [1027157.753984] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 29 10:27:43 localhost kernel: [1027157.753993] kworker/2:1     D    0 16365      2 0x00000000
Apr 29 10:27:43 localhost kernel: [1027157.754023] Workqueue: events_freezable mmc_rescan
Apr 29 10:27:43 localhost kernel: [1027157.754058] [<807a2ed0>] (__schedule) from [<807a3538>] (schedule+0x50/0xa8)
Apr 29 10:27:43 localhost kernel: [1027157.754076] [<807a3538>] (schedule) from [<807a39b0>] (schedule_preempt_disabled+0x18/0x1c)
Apr 29 10:27:43 localhost kernel: [1027157.754097] [<807a39b0>] (schedule_preempt_disabled) from [<807a52b8>] (__mutex_lock.constprop.3+0x190/0x58c)
Apr 29 10:27:43 localhost kernel: [1027157.754115] [<807a52b8>] (__mutex_lock.constprop.3) from [<807a57d0>] (__mutex_lock_slowpath+0x1c/0x20)
Apr 29 10:27:43 localhost kernel: [1027157.754136] [<807a57d0>] (__mutex_lock_slowpath) from [<807a5830>] (mutex_lock+0x5c/0x60)
Apr 29 10:27:43 localhost kernel: [1027157.754154] [<807a5830>] (mutex_lock) from [<80640540>] (rpi_firmware_transaction+0x44/0xac)
Apr 29 10:27:43 localhost kernel: [1027157.754174] [<80640540>] (rpi_firmware_transaction) from [<806406a0>] (rpi_firmware_property_list+0xf8/0x208)
Apr 29 10:27:43 localhost kernel: [1027157.754191] [<806406a0>] (rpi_firmware_property_list) from [<80640814>] (rpi_firmware_property+0x64/0x84)
Apr 29 10:27:43 localhost kernel: [1027157.754209] [<80640814>] (rpi_firmware_property) from [<8063bfc4>] (bcm2835_sdhost_set_clock+0xb8/0x31c)
Apr 29 10:27:43 localhost kernel: [1027157.754225] [<8063bfc4>] (bcm2835_sdhost_set_clock) from [<8063c2d0>] (bcm2835_sdhost_set_ios+0xa8/0xf0)
Apr 29 10:27:43 localhost kernel: [1027157.754240] [<8063c2d0>] (bcm2835_sdhost_set_ios) from [<8061f520>] (mmc_set_initial_state+0x98/0xb8)
Apr 29 10:27:43 localhost kernel: [1027157.754259] [<8061f520>] (mmc_set_initial_state) from [<8061f584>] (mmc_power_up.part.8+0x44/0x108)
Apr 29 10:27:43 localhost kernel: [1027157.754275] [<8061f584>] (mmc_power_up.part.8) from [<806204e4>] (mmc_rescan+0x254/0x394)
Apr 29 10:27:43 localhost kernel: [1027157.754291] [<806204e4>] (mmc_rescan) from [<80137a40>] (process_one_work+0x158/0x454)
Apr 29 10:27:43 localhost kernel: [1027157.754307] [<80137a40>] (process_one_work) from [<80137da0>] (worker_thread+0x64/0x5b8)
Apr 29 10:27:43 localhost kernel: [1027157.754322] [<80137da0>] (worker_thread) from [<8013de24>] (kthread+0x13c/0x16c)
Apr 29 10:27:43 localhost kernel: [1027157.754341] [<8013de24>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Is there a way to find what is causing this?

User avatar
PeterO
Posts: 4954
Joined: Sun Jul 22, 2012 4:14 pm

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 8:52 am

I would start at "Workqueue: events_freezable mmc_rescan" which suggests to me a failing sdcard.

PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
with ice cream
Posts: 161
Joined: Mon Jul 30, 2012 7:25 am

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 8:58 am

Thanks, Peter, for the quick response. The Pi boots from a USB-connected SSD. The SSD is connected via a powered USB hub.There is no SD card in the slot.

User avatar
PeterO
Posts: 4954
Joined: Sun Jul 22, 2012 4:14 pm

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 9:01 am

with ice cream wrote:
Mon Apr 29, 2019 8:58 am
Thanks, Peter, for the quick response. The Pi boots from a USB-connected SSD. The SSD is connected via a powered USB hub.There is no SD card in the slot.
But unless you've disabled it, the kernel keeps polling the sd card slot to detect if a card has been inserted. Maybe there is a problem with the sdcard interface...

PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
with ice cream
Posts: 161
Joined: Mon Jul 30, 2012 7:25 am

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 9:10 am

PeterO wrote:
Mon Apr 29, 2019 9:01 am
Maybe there is a problem with the sdcard interface...
That's an interesting perspective. fstab doesn't reference any SD card partitions.

How would one disable the SD interface? How would this be reversible?

User avatar
rpdom
Posts: 15054
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 9:16 am

with ice cream wrote:
Mon Apr 29, 2019 9:10 am
PeterO wrote:
Mon Apr 29, 2019 9:01 am
Maybe there is a problem with the sdcard interface...
That's an interesting perspective. fstab doesn't reference any SD card partitions.

How would one disable the SD interface? How would this be reversible?
It's as simple as adding an entry to /boot/config.txt and rebooting. Reversing it is just removing that entry and rebooting again.

Code: Select all

# Prevent constant polling of SD card when USB booted
dtoverlay=sdtweak,poll_once=on

User avatar
with ice cream
Posts: 161
Joined: Mon Jul 30, 2012 7:25 am

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 9:21 am

rpdom wrote:
Mon Apr 29, 2019 9:16 am
It's as simple as adding an entry to /boot/config.txt and rebooting.
Thanks. For clarification: the boot process still checks the SD card first, even if it is disabled and only polling gets disabled? If USB booting fails I can still pop in a working SD card?

User avatar
PeterO
Posts: 4954
Joined: Sun Jul 22, 2012 4:14 pm

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 9:23 am

I believe this only changes the behaviour of the kernel once it is running, not the boot process.
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
with ice cream
Posts: 161
Joined: Mon Jul 30, 2012 7:25 am

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 9:31 am

PeterO wrote:
Mon Apr 29, 2019 9:23 am
I believe this only changes the behaviour of the kernel once it is running, not the boot process.
It makes sense. Enabling USB boot is something that is done once and sticks, disabling SD boot for good would be undesirable for most (unless we're talking hardening, but that's something for a different thread).

User avatar
with ice cream
Posts: 161
Joined: Mon Jul 30, 2012 7:25 am

Re: Kernel panic intepretation help needed

Mon Apr 29, 2019 9:46 am

OK, the documentation explains that "poll_once=on" applies after boot. I'll try this and see if there are no more panics.

User avatar
with ice cream
Posts: 161
Joined: Mon Jul 30, 2012 7:25 am

Re: Kernel panic intepretation help needed

Sat May 04, 2019 11:31 am

Well, I'm back. I disabled SD card polling as discussed above. Now the panics happen again:

Code: Select all

May  4 10:48:57 localhost kernel: [420987.124314] INFO: task kworker/3:1:7047 blocked for more than 120 seconds.
May  4 10:48:57 localhost kernel: [420987.124326]       Tainted: G         C      4.14.98-v7+ #1200
May  4 10:48:57 localhost kernel: [420987.124328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  4 10:48:57 localhost kernel: [420987.124332] kworker/3:1     D    0  7047      2 0x00000000
May  4 10:48:57 localhost kernel: [420987.124352] Workqueue: events dbs_work_handler
May  4 10:48:57 localhost kernel: [420987.124375] [<807a2ed0>] (__schedule) from [<807a3538>] (schedule+0x50/0xa8)
May  4 10:48:57 localhost kernel: [420987.124387] [<807a3538>] (schedule) from [<807a711c>] (schedule_timeout+0x1d0/0x3f4)
May  4 10:48:57 localhost kernel: [420987.124395] [<807a711c>] (schedule_timeout) from [<807a41bc>] (wait_for_common+0xc0/0x184)
May  4 10:48:57 localhost kernel: [420987.124403] [<807a41bc>] (wait_for_common) from [<807a42a0>] (wait_for_completion+0x20/0x24)
May  4 10:48:57 localhost kernel: [420987.124412] [<807a42a0>] (wait_for_completion) from [<80640564>] (rpi_firmware_transaction+0x68/0xac)
May  4 10:48:57 localhost kernel: [420987.124423] [<80640564>] (rpi_firmware_transaction) from [<806406a0>] (rpi_firmware_property_list+0xf8/0x208)
May  4 10:48:57 localhost kernel: [420987.124431] [<806406a0>] (rpi_firmware_property_list) from [<80640814>] (rpi_firmware_property+0x64/0x84)
May  4 10:48:57 localhost kernel: [420987.124440] [<80640814>] (rpi_firmware_property) from [<8061becc>] (bcm2835_cpufreq_clock_property.constprop.1+0x48/0x                                        5c)
May  4 10:48:57 localhost kernel: [420987.124448] [<8061becc>] (bcm2835_cpufreq_clock_property.constprop.1) from [<8061bf2c>] (bcm2835_cpufreq_driver_target                                        _index+0x4c/0xc0)
May  4 10:48:57 localhost kernel: [420987.124457] [<8061bf2c>] (bcm2835_cpufreq_driver_target_index) from [<80615dcc>] (__cpufreq_driver_target+0x250/0x580)
May  4 10:48:57 localhost kernel: [420987.124466] [<80615dcc>] (__cpufreq_driver_target) from [<8061a760>] (od_dbs_update+0xa8/0x174)
May  4 10:48:57 localhost kernel: [420987.124473] [<8061a760>] (od_dbs_update) from [<8061b24c>] (dbs_work_handler+0x3c/0x68)
May  4 10:48:57 localhost kernel: [420987.124481] [<8061b24c>] (dbs_work_handler) from [<80137a40>] (process_one_work+0x158/0x454)
May  4 10:48:57 localhost kernel: [420987.124488] [<80137a40>] (process_one_work) from [<80137da0>] (worker_thread+0x64/0x5b8)
May  4 10:48:57 localhost kernel: [420987.124495] [<80137da0>] (worker_thread) from [<8013de24>] (kthread+0x13c/0x16c)
May  4 10:48:57 localhost kernel: [420987.124505] [<8013de24>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
May  4 10:50:59 localhost kernel: [421110.004975] INFO: task kworker/3:1:7047 blocked for more than 120 seconds.
May  4 10:51:00 localhost kernel: [421110.004988]       Tainted: G         C      4.14.98-v7+ #1200
May  4 10:51:00 localhost kernel: [421110.004990] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May  4 10:51:00 localhost kernel: [421110.004997] kworker/3:1     D    0  7047      2 0x00000000
May  4 10:51:00 localhost kernel: [421110.005018] Workqueue: events dbs_work_handler
May  4 10:51:00 localhost kernel: [421110.005060] [<807a2ed0>] (__schedule) from [<807a3538>] (schedule+0x50/0xa8)
May  4 10:51:00 localhost kernel: [421110.005070] [<807a3538>] (schedule) from [<807a711c>] (schedule_timeout+0x1d0/0x3f4)
May  4 10:51:00 localhost kernel: [421110.005080] [<807a711c>] (schedule_timeout) from [<807a41bc>] (wait_for_common+0xc0/0x184)
May  4 10:51:00 localhost kernel: [421110.005088] [<807a41bc>] (wait_for_common) from [<807a42a0>] (wait_for_completion+0x20/0x24)
May  4 10:51:00 localhost kernel: [421110.005098] [<807a42a0>] (wait_for_completion) from [<80640564>] (rpi_firmware_transaction+0x68/0xac)
May  4 10:51:00 localhost kernel: [421110.005109] [<80640564>] (rpi_firmware_transaction) from [<806406a0>] (rpi_firmware_property_list+0xf8/0x208)
May  4 10:51:00 localhost kernel: [421110.005116] [<806406a0>] (rpi_firmware_property_list) from [<80640814>] (rpi_firmware_property+0x64/0x84)
May  4 10:51:00 localhost kernel: [421110.005125] [<80640814>] (rpi_firmware_property) from [<8061becc>] (bcm2835_cpufreq_clock_property.constprop.1+0x48/0x                                        5c)
May  4 10:51:00 localhost kernel: [421110.005134] [<8061becc>] (bcm2835_cpufreq_clock_property.constprop.1) from [<8061bf2c>] (bcm2835_cpufreq_driver_target                                        _index+0x4c/0xc0)
May  4 10:51:00 localhost kernel: [421110.005145] [<8061bf2c>] (bcm2835_cpufreq_driver_target_index) from [<80615dcc>] (__cpufreq_driver_target+0x250/0x580)
May  4 10:51:00 localhost kernel: [421110.005155] [<80615dcc>] (__cpufreq_driver_target) from [<8061a760>] (od_dbs_update+0xa8/0x174)
May  4 10:51:00 localhost kernel: [421110.005162] [<8061a760>] (od_dbs_update) from [<8061b24c>] (dbs_work_handler+0x3c/0x68)
May  4 10:51:00 localhost kernel: [421110.005173] [<8061b24c>] (dbs_work_handler) from [<80137a40>] (process_one_work+0x158/0x454)
May  4 10:51:00 localhost kernel: [421110.005180] [<80137a40>] (process_one_work) from [<80137da0>] (worker_thread+0x64/0x5b8)
May  4 10:51:00 localhost kernel: [421110.005188] [<80137da0>] (worker_thread) from [<8013de24>] (kthread+0x13c/0x16c)
May  4 10:51:00 localhost kernel: [421110.005200] [<8013de24>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)

Return to “Advanced users”