User avatar
jbeale
Posts: 3570
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

NFS share: kworker hits 97% CPU. Can I limit it?

Wed Oct 01, 2014 2:33 am

I am using a python program (this one: https://github.com/jbeale1/PiCam1/blob/ ... iMotion.py ) to record 1080p video and JPEG stills, saving them to a shared folder on an Ubuntu laptop via NFS (multi-gig video files aren't suited to the limited space on the Pi's SD card). The connection is via ethernet on both ends, there is no wifi involved. This setup works well most of the time, with 'top' showing that my program r2.py occupies a steady 23% CPU. It is handling the camera video and stills via the picamera library . At intervals of about 35 seconds, there is about 6 seconds of network traffic as the buffered video stream goes over the network. At that time I see a raw datarate coming into the Ubuntu system around 6MB/sec (48 mbit/sec).

There is a problem that happens at random intervals of several minutes or more. The recorded video file is missing a few seconds. Running 'speedometer -r eth0' on the Ubuntu box (Acer C720), and 'top' on the R-Pi, shows that sometimes (always during the network activity of about 6 MB/sec) the kernel thread process 'kworker:0/0' CPU usage goes above 97% and at that time my video/still saving program "r2.py" falls below 1% CPU, even though my process is set to maximum priority with 'nice -20'. The below screen shot was taken at 17:53:43 and sure enough when I review the video stream recorded by the r2.py program with embedded time/date stamp, I see that the video from 17:53:41 to 17:53:45 was not recorded.

How can I prevent kworker from hogging all the CPU when NFS is transferring data? Is there some bandwidth throttling that I can do? Can I limit the damage by reducing the NFS buffer size, so the data is transmitted more frequently but for shorter times? The Pi's kernel is fairly recent:

Code: Select all

[email protected] ~ $ uname -a
Linux rp8 3.12.28+ #709 PREEMPT Mon Sep 8 15:28:00 BST 2014 armv6l GNU/Linux
Image

I wonder if it could be related to this bug: https://bugs.launchpad.net/ubuntu/+sour ... ug/1322406

elatllat
Posts: 1337
Joined: Sat Dec 17, 2011 5:05 pm

Re: NFS share: kworker hits 97% CPU. Can I limit it?

Wed Oct 01, 2014 3:17 am

Try 'man tc'.

That's definitely bad buffer usage. There is an issue like this in VLC (fixed in the nightlies). maybe use a 4TB usb HD, instead trying to fix that.
SBC with 32GB RAM: https://hardkernel.com

FAQ : https://raspberrypi.stackexchange.com

Unanswered: https://www.raspberrypi.org/forums/search.php?search_id=unanswered

User avatar
jbeale
Posts: 3570
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: NFS share: kworker hits 97% CPU. Can I limit it?

Wed Oct 01, 2014 5:36 am

Thank you for that tip! After consulting a page of 'tc' examples at http://www.cyberciti.biz/faq/linux-traf ... p-traffic/ I tried this command:

Code: Select all

sudo tc qdisc add dev eth0 root tbf rate 20mbit burst 10kb latency 70ms peakrate 30mbit minburst 1540
and sure enough, now my peak datarate coming into the Ubuntu box is around 2.4 MByte/sec, less than half the previous rate, and the sending process on the Pi is enjoying a much more steady share of the CPU. The kworker process is only going up to 8% CPU during the peaks. I'll have to wait until daylight when the camera data rate goes back up to know, but this may be a workable solution.

EDIT: spoke too soon. While the bandwidth and CPU looks good, I still get gaps. I think there is a problem in my code sometimes causing a deadlock in MMAL while the camera tries to save video, highres JPEG and lowres YUV all at once. Current guess is that caused the problem, and the kernel process going to 97% CPU was just the symptom of my program giving up and going to 0 %.

Return to “Troubleshooting”