High CPU usage when booting from USB

[colonelpanic]

Ever since I set my pi up to boot from a USB flash drive, the system has been pegging the cpu at right around 0.25 24/7.

To troubleshoot I performed:

Code: Select all

echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
cat /sys/kernel/debug/tracing/trace_pipe > out.txt

and see a stream of entries for:

Code: Select all

kworker/1:2-65    [001] d...   296.694258: workqueue_queue_work: work struct=b9ff9878 function=set_brightness_delayed workqueue=b9c06300 req_cpu=4 cpu=1

Additionally, running:

Code: Select all

 watch -n1 -d cat /proc/interrupts

shows lots of constant interrupts for the line:

Code: Select all

62:     186502          0          0          0  ARMCTRL-level  64 Edge      dwc_otg, dwc_otg_pcd, dwc_otg_hcd:usb1

The kworker in the first column of the workqueue output is the same as the worker taxing the CPU, but I do not know how to troubleshoot further as I have found no other information on what the other fields mean. What brightness is being set? How can I stop this?

Additional info:
I am running a fully patched (including firmware) Raspbian Lite 9.4 system.
I do not have a GUI installed.
I am using this as a Pi-Hole and OpenVPN host.
This issue does not happen when booting the same system from the SD card.

[wh7qq]

Using htop on my B+ pi-hole, it is running at less than 5% pretty constantly. It is headless but otherwise a stock Raspbian (full) distro. That would seem to point at the openVPN hosting application or something it runs.

[colonelpanic]

The issue is resolved when I boot from the SD card instead. The filesystem on the USB is unchanged, but simply booting from SD instead of USB solves the problem. Something about booting from USB is triggering a lot of interrupt requests.

[jamesh]

The issue is resolved when I boot from the SD card instead. The filesystem on the USB is unchanged, but simply booting from SD instead of USB solves the problem. Something about booting from USB is triggering a lot of interrupt requests.

When you boot from SD card do you still have the USB drive attached?

[fanoush]

Not sure if it is the same thing but when booting from usb and having system being idle I saw kworker 2:0 and 2:2 eating about 8-10% CPU always one or the other, not both at once. Also I noticed green led blinking periodically. What stopped it was inserting microsd card! Then it stopped and both kworkers went down to 0% and green led blinking stopped. Below it can be seen both have quite high time consumed in total after 15 minutes system uptime.

Code: Select all

 1035 root       20   0     0     0     0 I  0.0  0.0  0:00.07 ├─ kworker/0:1
 1034 root       20   0     0     0     0 I  0.0  0.0  0:16.85 ├─ kworker/2:0
  739 root       10 -10     0     0     0 S  0.0  0.0  0:00.00 ├─ krfcommd
 ...
   65 root      -51   0     0     0     0 S  0.0  0.0  0:00.17 ├─ irq/92-mmc1
   64 root       20   0     0     0     0 I  0.0  0.0  0:41.24 ├─ kworker/2:2
   63 root       10 -10     0     0     0 S  0.0  0.0  0:00.00 ├─ SMIO
  

EDIT:
after inserting microsd it got mounted under /media/pi/boot,root automatically so after ejecting/unmounting via taskbar applet I tried to remove the card again and once the card was physically removed the green led blinking started again and one of kworker/2:x started eating cpu again. And this time it was kworker 2:1

Code: Select all

2427 root       20   0     0     0     0 I  0.0  0.0  0:00.08 ├─ kworker/0:2
 2426 root       20   0     0     0     0 R 11.2  0.0  0:02.91 ├─ kworker/2:1
 2412 root       20   0     0     0     0 I  0.0  0.0  0:00.17 ├─ kworker/0:1
 

So it looks like it checks for card insertion periodically? or something tries to touch /dev/mmcblk0pX devices?

[jerrm]

I am seeing the same behavior.

Minimal Stretch-Lite install from latest image, no GUI, nothing extra installed.

USB Boot without SD card and a kworker thread stays at 8-10%.

USB Boot with blank SD card installed - no kworker utilization.

[jerrm]

Has anyone else seen this CPU utilization behavior?

It seems surprising more haven't complained, but the utilization may be hidden in some "top" programs (htop at least) unless they are set to view kernel threads.

The Heisenberg principle sometimes applies to CPU monitoring, but the kworker utilization seems consistent and repeatable. The kworker thread does not give up it's CPU usage even when the system is maxed out under sysbench, memtester, or other similar.

Doesn't seem like we should need to buy an SD card just to get full performance if USB booting.

[odoy]

same here. thx for the hint with the empty sd card.

[RichardS]

I also got hit with this. It was a nightmare getting USB HD booting to work. I went through 3 SATA - USB adapters, and tried each with 2 different SATA Hard drives before I found ONE combination that would even boot - and now that it is booting, I also seem to be hit by exactly the same FLASHING GREEN LED issue, plus high cpu and sluggish performance reported by other above.

Is there a patch to fix this available upstream somewhere that I can apply, if not, is one even being worked on?

And, in the mean time, as a second related question, are there any settings I can apply to the config or command line configuration files that will improve USB boot compatibility (increased boot delay, etc) ???

Thanks

[HawaiianPi]

When USB booting I often leave a non-bootable SD card in the system, because it eliminates the 5 second delay, and as others have suggested, it also eliminates the kworker issue. When booting from SD card I used to keep a USB flash drive in the system for backing up configuration files, scripts and such, and now the SD card is used for that when booting from my USB SSD.

And, in the mean time, as a second related question, are there any settings I can apply to the config or command line configuration files that will improve USB boot compatibility (increased boot delay, etc) ???

Thanks

If you want more boot delay (more wait time for USB drives to get ready), there is another OTP bit for that. Add "program_usb_boot_timeout=1" to config.txt on a Raspbian SD card and boot it once in the system. After that the system will wait up to 5 seconds for USB drives to become ready (default is 2 seconds).

The new 3B+ model has an improved bootloader with better device compatibility, but other than that there is not much that can be done. If your system can't boot from a particular device, there is no way the system can read configuration files from that device.

[RichardS]

When USB booting I often leave an non-bootable SD card in the system for faster boot (eliminates the 5 second delay), and as others have suggested, it also eliminates the kworker issue. When booting from SD card I used to keep a USB flash drive in the system for backing up configuration files, scripts and such, and now the SD card is used for that when booting from my USB SSD.

Thanks for jumping in with that fast reply, I had noted that in the thread above, but was looking to completely eliminate the SD card and set up Raspbian to boot from one of those switch selectable read-only USB thumb drives, to create a totally bullet proof, just boot it up and it ALWAYS WORKS purpose built device that would not be subject to corruption issues.

If I add a blank SD card, and that card takes a power hit and becomes unreadable (not just Locked to read-only, but totally unreadable, returning an error, for the boot sector) would the Pi boot anyway, or just FREEZE?

I ask, because I have built such systems in the past based on SD cards, and every one of them had issues within a few months, despite using high quality known genuine SD cards. I have heard that Raspbian and the Pi have improved quite a bit in regard to SD reliability, and have personally found the high endurance SanDisk microSD cards to be fairly good - so I might end up doing something like you mentioned eventually, and using a microSD to create a writable "home" folder - but first I want to get the application working in a totally read-only environment without relying on an SD card.

[LTolledo]

did some testing on my own... USB device is 2.5" 1TB HDD in Orico enclosure USB3.0, microSD is class10 16GB formatted to maximum.

USB device boot, no microSD in slot, boots normally, green activity LED flashes regularly every 1.48 to 1.58 secs.

USB device boot, blank partition microSD in slot, boots normally, green activity LED no regular flashing

USB device boot, blank FAT32 formatted microSD in slot, boots normally, green activity LED flashes about once between 10 to 40 secs (and I get 1 extra FAT32 disk space in the microSD)

USB device boot, blank NTFS formatted microSD in slot, DID NOT BOOT.

USB device boot, blank EXT4 formatted microSD in slot, boots normally, green activity LED flashes erratically between 8 successive short pulses 1.2secs in between to about 0.5sec pulse 30secs in between (and I get 1 extra EXT4 disk space in the microSD).

these are my findings, you may get different results/reading on your test...

[RichardS]

did some testing on my own... USB device is 2.5" 1TB HDD in Orico enclosure USB3.0, microSD is class10

. . .

USB device boot, blank FAT32 formatted microSD in slot, boots normally, green activity LED flashes about once between 10 to 40 secs (and I get 1 extra FAT32 disk space in the microSD)

USB device boot, blank NTFS formatted microSD in slot, DID NOT BOOT.

USB device boot, blank EXT4 formatted microSD in slot, boots normally, green activity LED flashes erratically between 8 successive short pulses 1.2secs in between to about 0.5sec pulse 30secs in between (and I get 1 extra EXT4 disk space in the microSD).

these are my findings, you may get different results/reading on your test...

Thanks! That's really good info.

Just a few quick questions. Was that on a Raspberry Pi 3B or 3B+ ?

... and when you say 1 extra FAT or EXT4, do you mean that the Pi is creating an additional phantom partition on the microSD that was not there originally?

If so, that seems very strange doesn't it? I wonder what kind of info do you suppose it's trying to log?

[PeterO]

If so, that seems very strange doesn't it? I wonder what kind of info do you suppose it's trying to log?

It's not logging, it's polling the sd slot to see it you've inserted an sd card. If you USB boot you can hot plug sd-cards but I believe the PI doesn't use a "card inserted" physical switch so the kernel has to poll the slot by trying to read the card.

PeterO

[LTolledo]

Just a few quick questions. Was that on a Raspberry Pi 3B or 3B+ ?

Raspberry Pi 3 model B V1.2

... and when you say 1 extra FAT or EXT4, do you mean that the Pi is creating an additional phantom partition on the microSD that was not there originally?

If so, that seems very strange doesn't it? I wonder what kind of info do you suppose it's trying to log?

Oh maybe I got the wordings wrong.... I meant that the formatted partition in the microSD (FAT32, EXT4) can be accessed as an additional mounted volume.

Haven't tried removing/inserting the microSD with RPi still powered ON, but I think it may be possible....

[jerrm]

It's not logging, it's polling the sd slot to see it you've inserted an sd card. If you USB boot you can hot plug sd-cards but I believe the PI doesn't use a "card inserted" physical switch so the kernel has to poll the slot by trying to read the card.

But it is an insane amount of CPU usage for such a simple task. There may well be hardware deficiencies that requires such intense CPU usage, but that doesn't mean the current state of things is acceptable.

Ideally we would have tunables to disable the scan or increase the interval.

At minimum the USB boot documentation should make folks aware of the issue and the easiest work around (blank SD).

[PaultheElder]

It appears to me that to boot Raspberry Pi from a USB device brings tremendous advantages regardless of any modest gain or loss in performance. For one, the opportunity for individuals within goups to effectively own their personalised operating environment / workspace and transfer it from, say home to education place (where they could use any of a suite of Pi's) on a USB stick, is obvious. Such use alone must place this issue well within the Foundation's worthy goals.
Please, this whole 'problem' could be resolved to everyone's benefit if those visionary folks at Pi Towers would reveal fully how they intend this feature can be utilized to best advantage, thereby solving the "mysteries of blank SD cards and flashing green LED's" which are a diversion from the real purpose of the Pi.

[jerrm]

USB Boot is nearly two years old and is now enabled by default on the 3B+. I doubt there are any grand revelations coming. The advantages and uses are self evident.

A 10% hit to performance is more than modest. It essentially negates the the CPU improvement between the 3B and 3B+. It also has a small (about 1C no heatsink or fan) but measurable impact on idle CPU temp, probably indicating power consumption is also impacted.

It adds $5 to the cost if someone wants the advantages of USB boot without taking a performance hit.

It shouldn't take benchmarking or searching forums to know there is an issue.

None of this is world ending news, but not being documented is inexcusable.

[fanoush]

I have reported it as a bug here https://github.com/raspberrypi/linux/issues/2567

[fanoush]

looks like there is a fix coming, there will be a device tree overlay parameter to set in config.txt to detect the card only once at boot time and disable the polling

[fanoush]

Looks like there are currently two ways to avoid the card polling and high kworker cpu usage. Using newer sdtweak.dtbo from github issue linked above and adding one line to /boot/config.txt

Code: Select all

dtoverlay=sdtweak,poll_once

This marks the slot as non-removable which disables any polling for changes. Another way is unbinding microsd slot driver at runtime. This basically disables microsd slot completely. If you have card mounted then unmount it before trying this.

Code: Select all

root@raspberrypi:~# echo -n "3f202000.mmc" >/sys/bus/platform/drivers/sdhost-bcm2835/unbind

One can rebind the driver again to reenable the slot and/or scan for card changes.

Code: Select all

root@raspberrypi:~# echo -n "3f202000.mmc" >/sys/bus/platform/drivers/sdhost-bcm2835/bind

The 3f202000.mmc value is valid for Pi3B+, correct name of mmc interface may be different on different pi models (?) and can be see when lisiting /sys/bus/platform/drivers/sdhost-bcm2835 folder before unbinding.

Code: Select all

root@raspberrypi:~# ls -l /sys/bus/platform/drivers/sdhost-bcm2835/
total 0
lrwxrwxrwx 1 root root    0 May 29 22:31 3f202000.mmc -> ../../../../devices/platform/soc/3f202000.mmc

[jerrm]

Below is a crude systemd unit to unbind the driver if no card is present at boot. No doubt a lot more checks should be built in for differing configurations, but it suffices for my needs - simple one file fix I can make part of my standard config (assuming no issues come up with further testing).

Tested ONLY on Pi3B and Pi3B+ !

Code: Select all

# !! **  USE AT YOUR OWN RISK  ** !!
#
#
# This is a crude systemd unit to disable the sd card slot if no
# sd card is installed at boot.  Thanks to @fanoush.
#
# See https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=211912#p1321458
#
# Nothing is changed if the expected device name is not found.
#
# Nothing is changed if an sd card appears present in the slot.
#
# Paste this text into a file named /etc/systemd/system/disable-mmc-rescan
# execute "chmod 644 /etc/systemd/system/disable-mmc-rescan"
#
# Enable the service with "systemctl enable disable-mmc-rescan" and reboot.
#
# To re-enable the sd slot after boot - stop the service - "systemctl stop disable-mmc-rescan"
#
#
# !! **  USE AT YOUR OWN RISK  ** !!
#
# Only basic testing has been done on Pi3B and Pi3B+ 
# under Raspbian Stretch-Lite and Jessie-Lite(Pi3B only) current as of 06/01/2018.
#
# No claims are made your system will remain stable or even boot.
#
# !! **  USE AT YOUR OWN RISK  ** !!

[Unit]
Description=Disable MMC Rescan
DefaultDependencies=no

[Service]
Type=oneshot

ExecStart=/bin/bash -c " dpath=/sys/bus/platform/drivers/sdhost-bcm2835 ; \
                         dname=3f202000 ; \
                         dfull=$( /bin/ls -d $dpath/$dname.{mmc,sdhost} 2> /dev/null) ; \
                         [ a$dfull = a ] && { \
                           echo $dname 'device not present. no changes made.' ; \
                           exit 1 ; \
                         } ; \
                         dlink=$( /usr/bin/basename $dfull ) ; \
                         [ ! -L $dpath/$dlink ] && { \
                           echo $dlink 'link not found. no changes made.'; exit 1 ; \
                         } ; \
                         [ -b /dev/mmcblk0 ] && { \
                           echo 'sd card found - no changes made' ; \
                           exit 1; \
                         } ; \
                         echo $dlink > /var/run/%n ; \
                         echo -n $dlink > $dpath/unbind ; \
                         echo 'no sd card found - driver binding removed - sd card rescan disabled.' ; \
                       "
RemainAfterExit=true

ExecStop=/bin/bash -c "  dpath=/sys/bus/platform/drivers/sdhost-bcm2835 ; \
                         dname=3f202000 ; \
                         status=a$(/bin/systemctl is-system-running) ;  \
                         [ $status = astopping ] && { \
                           echo 'System Shutting Down - Skipping driver reload' ; \
                           exit 0 ; \
                         } ; \
                         dlink=$( /bin/cat /var/run/%n ) ; \
                         [[ ! a$dlink =~ a$dname.(mmc|sdhost) ]] && { \
                           echo 'valid device name not found.  no changes made.' ; \
                           exit 1 ; \
                         } ; \
                         echo -n $dlink > $dpath/bind ; \
                         echo 'sd card rescan enabled.' ; \
                      "
StandardOutput=journal

[Install]
WantedBy=sysinit.target

EDIT: Updated to work with Jessie.

[fanoush]

Nice. However having system set to 'stop' such service and initialize card at shutdown right before poweroff is not very optimal. Hopefuly the worst case is that the shutdown will be a tiny bit slower.

[LTolledo]

Looks like there are currently two ways to avoid the card polling and high kworker cpu usage. Using newer sdtweak.dtbo from github issue linked above and adding one line to /boot/config.txt

Code: Select all

dtoverlay=sdtweak,poll_once

This marks the slot as non-removable which disables any polling for changes.

If I deploy this, can the inserted microSD card be accessed as a mounted volume (if formatted)?

[fanoush]

If I deploy this, can the inserted microSD card be accessed as a mounted volume (if formatted)?

Yes, it will work with the poll_once option if card is already inserted at boot time. After boot it will detect no changes (card removal or insert) unless you unbind and bind the driver after the change.

And BTW you need to download updated sdtweak.dtbo from the link posted to github issue https://drive.google.com/file/d/1JOwJpv ... sp=sharing and replace your older one in /boot/overlays/ otherwise new options will have no effect.