Issues since kernel got updated to 4.19.93

[rosioara]

I have a Rasp 4b 4GB version and I'm normally using it to run tvheadend with USB dvb-c or dvb-s tuners.

Everything worked pretty normal before mid-January. However, I did the 'apt update / app full-upgrade' and noticed that kernel was updated to 4.19.93. Since then, the USB devices are no longer usable. Below is the dmesg output.

If I downgrade the kernel to previous stable versions from November or September (e.g. sudo rpi-update 4b2c270) the issues are gone.

Anything I am missing? I tried also the VL805 000137ad firmware but no change.

Code: Select all

[   37.431597] xhci_hcd 0000:01:00.0: xHCI host not responding to stop endpoint command.
[   37.447618] xhci_hcd 0000:01:00.0: Host halt failed, -110
[   37.447630] xhci_hcd 0000:01:00.0: xHCI host controller not responding, assume dead
[   37.447683] xhci_hcd 0000:01:00.0: HC died; cleaning up
[   37.457815] usb 1-1: USB disconnect, device number 2
[   37.457835] usb 1-1.4: USB disconnect, device number 3
[   37.458119] Unable to handle kernel NULL pointer dereference at virtual address 0000004c
[   37.458158] pgd = 1d6d2ce3
[   37.458179] [0000004c] *pgd=80000000004003, *pmd=00000000
[   37.458215] Internal error: Oops: 207 [#1] SMP ARM
[   37.458236] Modules linked in: bnep hci_uart btbcm serdev bluetooth ecdh_generic 8021q garp stp llc evdev cx24116 vc4 dvb_usb_dw2102 dvb_usb drm_kms_helper dvb_core v3d gpu_sched brcmfmac brcmutil drm drm_panel_orientation_quirks sha256_generic snd_soc_core cfg80211 snd_compress snd_pcm_dmaengine syscopyarea sysfillrect sysimgblt fb_sys_fops snd_bcm2835(C) snd_pcm rfkill raspberrypi_hwmon snd_timer hwmon snd bcm2835_codec(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) v4l2_common videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev media vc_sm_cma(C) rpivid_mem uio_pdrv_genirq uio fixed ip_tables x_tables ipv6
[   37.458476] CPU: 0 PID: 68 Comm: kworker/0:2 Tainted: G         C        4.19.95-v7l+ #1292
[   37.458503] Hardware name: BCM2835
[   37.458538] Workqueue: usb_hub_wq hub_event
[   37.458585] PC is at dw2102_disconnect+0x24/0x70 [dvb_usb_dw2102]
[   37.458612] LR is at usb_unbind_interface+0x84/0x26c
[   37.458633] pc : [<bf95b1fc>]    lr : [<c07916e8>]    psr: 60000013
[   37.458655] sp : ef2d7c98  ip : ef2d7cb8  fp : ef2d7cb4
[   37.458676] r10: dc4bb478  r9 : bf965ac0  r8 : dc4bb420
[   37.458697] r7 : 00000000  r6 : 00000000  r5 : dc4bb400  r4 : dc4bb420
[   37.458719] r3 : dbad1000  r2 : 40000093  r1 : 00000000  r0 : dc4bb400
[   37.458743] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   37.458767] Control: 30c5383d  Table: 1ac6e380  DAC: fffffffd
[   37.458789] Process kworker/0:2 (pid: 68, stack limit = 0xfb3a783a)
[   37.458811] Stack: (0xef2d7c98 to 0xef2d8000)
[   37.458832] 7c80:                                                       bf95b1d8 dc4bb420
[   37.458862] 7ca0: dc4bb454 bf965ac0 ef2d7cfc ef2d7cb8 c07916e8 bf95b1e4 ef2d7cdc ffffffed
[   37.458891] 7cc0: ef2aa878 ef2aa800 00000000 dc4bb400 ef2d7cfc dc4bb420 dc4bb454 bf965ac0
[   37.458920] 7ce0: 00000000 00000034 dc4bb200 dc4bb478 ef2d7d24 ef2d7d00 c07086f8 c0791670
[   37.458949] 7d00: 00000007 efa63830 dc4bb420 c106f97c c1153dd4 dc4bb428 ef2d7d34 ef2d7d28
[   37.458978] 7d20: c07087b8 c070857c ef2d7d54 ef2d7d38 c07070ac c07087a4 0000000b dc4bb454
[   37.459007] 7d40: dc4bb420 c1004d88 ef2d7da4 ef2d7d58 c0703800 c0706fdc 00000002 ef2aa878
[   37.459035] 7d60: ef2d7d84 ef2d7d70 c07964ac c0703a08 00000002 8aea40ae ef2d7da4 ef2aa800
[   37.459064] 7d80: 00000001 00000000 ef8f6000 dc4bb400 dc4bb200 00000000 ef2d7dcc ef2d7da8
[   37.459093] 7da0: c078ed5c c07036ac 00000004 ef2aa800 ef2aa878 ef2aa8ac dc4ba800 dc4bb200
[   37.459121] 7dc0: ef2d7e04 ef2d7dd0 c07854f8 c078ecb8 ef2d7dec ef2d7de0 c09c910c 00000004
[   37.459150] 7de0: ef2aa400 ef2aa478 ef2aa4ac ef23d400 dc4ba000 00000000 ef2d7e3c ef2d7e08
[   37.459179] 7e00: c078567c c0785480 eff214c0 ef2d8080 c1004d88 00000001 ef23d200 ef23b400
[   37.459207] 7e20: eff25f00 c1004d88 ef23d304 00000000 ef2d7e5c ef2d7e40 c0785714 c0785480
[   37.459236] 7e40: ffffffed ef23b400 ef23d020 ef23d300 ef2d7efc ef2d7e60 c0786e38 c07856c4
[   37.459265] 7e60: ef2d7eb4 ef2d7e70 c024d1d8 c0e9bad8 c09ce5b4 eff21040 eff24300 ef8f6180
[   37.459293] 7e80: 40000013 00000000 ef2d7eb4 ef23d020 c0305a7c c0306c80 40000013 c09ce410
[   37.459322] 7ea0: ef8f61c0 eff21040 ef23b400 00000000 ef23b4ac c0e9bad8 ef23d000 ef23d200
[   37.459351] 7ec0: eff25f00 00000000 ef23d304 8aea40ae ef2d7efc ef2c0580 ef23d300 eff21040
[   37.459379] 7ee0: eff25f00 00000000 ef23d304 00000000 ef2d7f34 ef2d7f00 c023e028 c0786aa0
[   37.459408] 7f00: 40000093 eff21040 eff21040 eff21040 ef2c0594 eff21040 eff21058 c1003d00
[   37.459437] 7f20: 00000008 ef2c0580 ef2d7f7c ef2d7f38 c023e36c c023dec4 ef2d7f5c ef2d7f48
[   37.459465] 7f40: c09ce5b4 c1003d00 c1096816 ef2d6038 ef2d7f7c ef2c28c0 ef2c2680 00000000
[   37.459494] 7f60: ef2c0580 c023e310 ef2c28dc ef911e74 ef2d7fac ef2d7f80 c02446a0 c023e31c
[   37.459523] 7f80: c02043d8 ef2c2680 c0244568 00000000 00000000 00000000 00000000 00000000
[   37.459551] 7fa0: 00000000 ef2d7fb0 c02010ac c0244574 00000000 00000000 00000000 00000000
[   37.459579] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   37.459608] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   37.459677] [<bf95b1fc>] (dw2102_disconnect [dvb_usb_dw2102]) from [<c07916e8>] (usb_unbind_interface+0x84/0x26c)
[   37.459722] [<c07916e8>] (usb_unbind_interface) from [<c07086f8>] (device_release_driver_internal+0x188/0x228)
[   37.459762] [<c07086f8>] (device_release_driver_internal) from [<c07087b8>] (device_release_driver+0x20/0x24)
[   37.459800] [<c07087b8>] (device_release_driver) from [<c07070ac>] (bus_remove_device+0xdc/0x108)
[   37.459836] [<c07070ac>] (bus_remove_device) from [<c0703800>] (device_del+0x160/0x35c)
[   37.459870] [<c0703800>] (device_del) from [<c078ed5c>] (usb_disable_device+0xb0/0x1e0)
[   37.459905] [<c078ed5c>] (usb_disable_device) from [<c07854f8>] (usb_disconnect+0x84/0x244)
[   37.459942] [<c07854f8>] (usb_disconnect) from [<c078567c>] (usb_disconnect+0x208/0x244)
[   37.459978] [<c078567c>] (usb_disconnect) from [<c0785714>] (hub_quiesce+0x5c/0xa0)
[   37.460013] [<c0785714>] (hub_quiesce) from [<c0786e38>] (hub_event+0x3a4/0x11d8)
[   37.460051] [<c0786e38>] (hub_event) from [<c023e028>] (process_one_work+0x170/0x458)
[   37.460085] [<c023e028>] (process_one_work) from [<c023e36c>] (worker_thread+0x5c/0x5a4)
[   37.460119] [<c023e36c>] (worker_thread) from [<c02446a0>] (kthread+0x138/0x168)
[   37.460151] [<c02446a0>] (kthread) from [<c02010ac>] (ret_from_fork+0x14/0x28)
[   37.460177] Exception stack(0xef2d7fb0 to 0xef2d7ff8)
[   37.460199] 7fa0:                                     00000000 00000000 00000000 00000000
[   37.460228] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   37.460255] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   37.460282] Code: e5903074 e1a05000 e2833a01 e5936bb8 (e596404c) 
[   37.460310] ---[ end trace 4940516b154503e7 ]---
[   37.479638] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   37.480160] cx24116_firmware_ondemand: Waiting for firmware upload(2)...
[   39.731605] cx24116_cmd_execute() Firmware not responding
[   39.731631] cx24116_firmware_ondemand: Writing firmware to device failed
[   39.731669] cx24116_firmware_ondemand: Firmware upload failed
[   39.731687] cx24116_cmd_execute(): Unable initialise the firmware
[   39.831991] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   39.832113] cx24116_firmware_ondemand: Waiting for firmware upload(2)...
[   42.081605] cx24116_cmd_execute() Firmware not responding
[   42.081631] cx24116_firmware_ondemand: Writing firmware to device failed
[   42.081669] cx24116_firmware_ondemand: Firmware upload failed
[   42.081687] cx24116_cmd_execute(): Unable initialise the firmware
[   45.212342] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   45.212469] cx24116_firmware_ondemand: Waiting for firmware upload(2)...
[   47.461603] cx24116_cmd_execute() Firmware not responding
[   47.461630] cx24116_firmware_ondemand: Writing firmware to device failed
[   47.461668] cx24116_firmware_ondemand: Firmware upload failed
[   47.461687] cx24116_cmd_execute(): Unable initialise the firmware
[   53.552347] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   53.552501] cx24116_firmware_ondemand: Waiting for firmware upload(2)...
[   55.801603] cx24116_cmd_execute() Firmware not responding
[   55.801629] cx24116_firmware_ondemand: Writing firmware to device failed
[   55.801666] cx24116_firmware_ondemand: Firmware upload failed
[   55.801685] cx24116_cmd_execute(): Unable initialise the firmware
[   60.892417] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   60.892572] cx24116_firmware_ondemand: Waiting for firmware upload(2)...
[   63.141604] cx24116_cmd_execute() Firmware not responding
[   63.141631] cx24116_firmware_ondemand: Writing firmware to device failed
[   63.141669] cx24116_firmware_ondemand: Firmware upload failed
[   63.141687] cx24116_cmd_execute(): Unable initialise the firmware
[   68.232398] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   68.232547] cx24116_firmware_ondemand: Waiting for firmware upload(2)...
[   70.481603] cx24116_cmd_execute() Firmware not responding
[   70.481628] cx24116_firmware_ondemand: Writing firmware to device failed
[   70.481666] cx24116_firmware_ondemand: Firmware upload failed
[   70.481685] cx24116_cmd_execute(): Unable initialise the firmware
[   76.532440] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   76.532598] cx24116_firmware_ondemand: Waiting for firmware upload(2)...
[   78.791604] cx24116_cmd_execute() Firmware not responding
[   78.791630] cx24116_firmware_ondemand: Writing firmware to device failed
[   78.791671] cx24116_firmware_ondemand: Firmware upload failed
[   78.791690] cx24116_cmd_execute(): Unable initialise the firmware
[   83.922615] cx24116_firmware_ondemand: Waiting for firmware upload (dvb-fe-cx24116.fw)...
[   83.922736] cx24116_firmware_ondemand: Waiting for firmware upload(2)...

[rosioara]

Similar issue here: https://archlinuxarm.org/forum/viewtopi ... 65&t=14172

[jdb]

Can you try rpi-update firmwares to see which is the first "bad" version? Keep the VL805 eeprom image at 0137ad.

Use git hashes from here: https://github.com/Hexxeh/rpi-firmware/commits/master

and do sudo rpi-update <hash> - there's only been 10 or so.

[rosioara]

sudo rpi-update 16ce612 - this already produces the issue (18 Nov) Linux raspberrypi 4.19.84-v7l+ #1278 SMP Mon Nov 18 14:37:06 GMT 2019 armv7l GNU/Linux

sudo rpi-update afbea38 - just before, this works fine (14 Nov)

From Stable, 4b2c270 (which I think it's the same as afbea38) works fine.

[jdb]

On the Nov 18 firmware, an you add force_turbo=1 to config.txt and test?
If that works, then can you replace force_turbo=1 with over_voltage=2?

[rosioara]

force_turbo=1 - works fine
over_voltage=2 - works fine

What would be the issue? Hardware related? I never did overclocking.

[dom]

What would be the issue? Hardware related? I never did overclocking.

The firmware commit identified was the main one that reduces power by reducing voltage when idle.
It seems like there is a clock required by USB that is not happy (on some boards) at the reduced voltage.

Do you have anything non-default in config.txt?
Can you try with over_voltage=1?
What does "vcgencmd get_config over_voltage_avs" report?

[rosioara]

over_voltage=1 still works fine.

over_voltage_avs=-17500, I don't know how to read this value...

No non-standard settings in config.txt, only today I found about it

[dom]

over_voltage=1 still works fine.

Could you try this firmware without the over_voltage line?

[quakstarpoke]

I also faced the same issue after I've set up samba.

I assume it may be this configuration that triggered this issue, but I am not quite sure.

Also it may be worth noting that several days ago I installed Openmediavault in my RPI 4b (4GB) as a NAS with samba turned on. Everything worked fine. But then yesterday I decided not to use OMV anymore and replace the OS as Raspbian Buster with desktop. After I configured samba, nothing happened and everything worked well. It was until next day that I found out there was no HDMI signal output and my keyboard and mouse did not response. So I plug out and in my power connector. Then I found out this issue when I read kernel logs, as following.

Note time 03:50:03, which is when I unplug the usb of my wireless mouse and keyboad. And time 05:16:45, which I think is when the issue came up. During this period of time I was asleep. Then time 15:17:05 is when I plug out and in my power connector.

Code: Select all

Jan 22 03:36:02 raspberrypi kernel: [   13.702233] Bluetooth: BNEP filters: protocol multicast
Jan 22 03:36:02 raspberrypi kernel: [   13.702246] Bluetooth: BNEP socket layer initialized
Jan 22 03:36:02 raspberrypi kernel: [   13.750633] Bluetooth: RFCOMM TTY layer initialized
Jan 22 03:36:02 raspberrypi kernel: [   13.750677] Bluetooth: RFCOMM socket layer initialized
Jan 22 03:36:02 raspberrypi kernel: [   13.750704] Bluetooth: RFCOMM ver 1.11
Jan 22 03:50:03 raspberrypi kernel: [  806.441086] usb 1-1.4: USB disconnect, device number 3
Jan 22 05:16:45 raspberrypi kernel: [ 6008.101951] xhci_hcd 0000:01:00.0: WARNING: Host System Error
Jan 22 05:16:52 raspberrypi kernel: [ 6015.129767] usb 2-2: Disable of device-initiated U1 failed.
Jan 22 05:16:52 raspberrypi kernel: [ 6015.129862] usb 2-2: Disable of device-initiated U2 failed.
Jan 22 05:16:52 raspberrypi kernel: [ 6015.129881] usb 2-2: Could not disable xHCI U2 timeout, bus schedule bandwidth may be impacted.
Jan 22 05:16:52 raspberrypi kernel: [ 6015.719629] usb 2-2: device not accepting address 2, error -108
Jan 22 05:16:53 raspberrypi kernel: [ 6016.429625] usb 2-2: device not accepting address 2, error -108
Jan 22 05:16:54 raspberrypi kernel: [ 6017.139630] usb 2-2: device not accepting address 2, error -108
Jan 22 05:16:54 raspberrypi kernel: [ 6017.849630] usb 2-2: device not accepting address 2, error -108
Jan 22 05:16:55 raspberrypi kernel: [ 6017.929802] usb 2-2: USB disconnect, device number 2
Jan 22 05:17:00 raspberrypi kernel: [ 6023.179683] xhci_hcd 0000:01:00.0: Abort failed to stop command ring: -110
Jan 22 05:17:00 raspberrypi kernel: [ 6023.179721] xhci_hcd 0000:01:00.0: xHCI host controller not responding, assume dead
Jan 22 05:17:00 raspberrypi kernel: [ 6023.179730] xhci_hcd 0000:01:00.0: HC died; cleaning up
Jan 22 05:17:00 raspberrypi kernel: [ 6023.180492] usb 1-1: USB disconnect, device number 2
Jan 22 05:17:00 raspberrypi kernel: [ 6023.181933] sd 0:0:0:0: [sda] Synchronizing SCSI cache
Jan 22 05:17:00 raspberrypi kernel: [ 6023.181982] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=0x00
Jan 22 05:17:00 raspberrypi kernel: [ 6023.350171] usb usb2-port2: couldn't allocate usb_device
Jan 22 15:17:05 raspberrypi kernel: [    0.000000] Booting Linux on physical CPU 0x0
Jan 22 15:17:05 raspberrypi kernel: [    0.000000] Linux version 4.19.93-v7l+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1290 SMP Fri Jan 10 16:45:11 GMT 2020

if anyone could explain to me how to solve this I would really appreciate it.

[rosioara]

over_voltage=1 still works fine.

Could you try this firmware without the over_voltage line?

Just tried, the issues are back again:

Code: Select all

[   89.591592] xhci_hcd 0000:01:00.0: xHCI host not responding to stop endpoint command.
[   89.607641] xhci_hcd 0000:01:00.0: Host halt failed, -110
[   89.607652] xhci_hcd 0000:01:00.0: xHCI host controller not responding, assume dead
[   89.607691] xhci_hcd 0000:01:00.0: HC died; cleaning up
[   89.607770] usb 1-1-port4: cannot reset (err = -110)
[   89.608020] usb 1-1-port4: couldn't allocate usb_device
[   89.608063] usb 1-1: USB disconnect, device number 2

[stripe]

Hi all
exactly the same issues here with the 4 B 4 GB, usb ports not working after todays kernel upgrade, not even mouse or keyboard, so I can only mount and access the sdcard on another machine to make any changes, or even re-flash to a working image, can historic images of rasbian be downloaded?
any ideas on how to try other firmware if I have no control over the machine, I could add ssh to the /boot and ssh in as a last resort
I made no changes to /boot
I updated firmware to reduce overheating around 2 months ago, sorry dont know version
thanks
Don

[rosioara]

Take the SD card out to a PC. In the Boot partition add an empty file named "ssh". Then in the same partition edit the file "config.txt" and add there over_voltage=1 (if not ok you can also try 2). This workaround works fine for me, let's hope for a structured approach.

I ordered a second 4b and I will test today to see if it's hardware related or not.

[jamesh]

Take the SD card out to a PC. In the Boot partition add an empty file named "ssh". Then in the same partition edit the file "config.txt" and add there over_voltage=1 (if not ok you can also try 2). This workaround works fine for me, let's hope for a structured approach.

I ordered a second 4b and I will test today to see if it's hardware related or not.s

Looks like the setting used for power might be too aggressive for some Pis. Not all or we would have seen problems in testing, this software has been in rpi-update for a month or so, so tried by a lot of people.

[rosioara]

Looks like the setting used for power might be too aggressive for some Pis. Not all or we would have seen problems in testing, this software has been in rpi-update for a month or so, so tried by a lot of people.

Not sure if this helps, but my Pi 4b was purchased immediately after release, around July-August last year. Perhaps these ones encountered issues. I'll test in the evening with a new hardware ordered yesterday, I hope it's a newer HW revision.

I'm also quoting a post from here: https://archlinuxarm.org/forum/viewtopi ... 65&t=14172

I moved the SD card to a second PI4 (and attached the USB2 device), and booted up, and it never hits the xHCI problem. It works fine, regardless if I plug the USB2 device into USB2 or USB3 ports, and regardless of the linux image (old image versus new image).

The original PI4 always hits the xHCI problem when running a newer kernel image: sometimes immediately, and sometimes within a few hours, but always always always.

[stripe]

Mine was also purchased on release day in the UK, (hope this helps)

edited to add: over_voltage=1 seemed to do the job here, will keep an eye on it and report any further problems to this thread

thanks everyone

edit 2: I can test any new images bugfixes/releases if needed, as I always have a spare sdcard and plenty of time

[jamesh]

This will be independant of revision or age. All chips come out of the fab with slightly different properties. There's a bell curve of those properties, a few are very close to the edges of the curve, and looks like the power management strategy is slightly too aggressive for those devices right at the limits. This applies to all the chips on the board, not just the SoC, and it could be a combination. Unfortunate we didn't see anything in testing, but this likely means it's very rare. Note, these are my personal thoughts on the matter, not an official Pi statement, and I may well be wrong!

[rosioara]

As other colleagues said, happy to test and provide feedback.

I noticed the issue with rpi-update but I did not raise this thinking it's not a stable kernel and it will be fixed later.

[jamesh]

I noticed the issue with rpi-update but I did not raise this thinking it's not a stable kernel and it will be fixed later.

Ouch. We cannot fix stuff we don't know about....!

[Roy Cormier]

Same problem here with a USB3 gigabit ethernet adapter. So far force_turbo=1 and over_voltage=2 seem to have fixed it.

[rosioara]

As promised, I tested the same setup with a brand new 4b board that I purchased today. Surprise (or not) this one works fine with the newest kernel, no USB issues like the other one.

The new board is revision 1.2, I ran apt update / apt full-upgrade and rpi-eeprom-update -a to bring it up-to-date.

The result for over_voltage_avs=-23750

[rosioara]

Perhaps, could you help me understanding the following output results from two different boards?

Board 1: 4b rev 1.1 purchased July 2019, showing USB issues (disconnects): over_voltage_avs=-17500

Board 2: 4b rev 1.2 purchased January 2020, no USB issues: over_voltage_avs=-30000

[jamesh]

Nothing to do with board revisions, this is just down to variations in soc manufacture.

[rosioara]

Understood, but what the values exactly mean?

[jamesh]

Understood, but what the values exactly mean?

Whist running, the firmware on the GPU monitors its internal status with regard to clocks and timings of the cores and sdram. Various things, including voltages, are adjusted slightly to ensure best performance and power usage. This I believe is one of them.