8000 interrupts per second when idle!


140 posts   Page 1 of 6   1, 2, 3, 4, 5, 6
by rgh » Fri Jun 08, 2012 3:36 pm
Hi, on my Pi, running Debian, 'vmstat' tells me it is getting 8000 interrupts per second, and /proc/interrupts tells me they are from the usb. That seems excessive on an idle system. My Ubuntu desktop is seeing about 2400, but they are probably mostly timer interrupts for scheduling given that it is running X and lots of other stuff.

Do others see that? Is it something that can be improved on with driver changes?
Posts: 219
Joined: Fri Nov 25, 2011 3:53 pm
by abishur » Fri Jun 08, 2012 3:47 pm
Do you have the network plugged in?

The network adapter is actually connected via the usb bus so if you have the network plugged in it wouldn't be too surprising to have a lot of interrupts on the usb hub would it?
Dear forum: Play nice ;-)
User avatar
Moderator
Moderator
Posts: 4221
Joined: Thu Jul 28, 2011 4:10 am
Location: USA
by rurwin » Fri Jun 08, 2012 4:04 pm
8,000 packets per second would be an extremely active network.
User avatar
Moderator
Moderator
Posts: 2890
Joined: Mon Jan 09, 2012 3:16 pm
by dom » Fri Jun 08, 2012 4:07 pm
That is normal everyone sees the same. Even when usb/network is completely idle. It's a feature of the Synopsys dwc_otg driver (it has many features).

If you are not using USB or network (i.e. very few people) it is possible to kill the driver and stop the interrupts. It does gain up to 20% performance.

There's been discussions about replacing the ISR with an assembly coded FIQ, or a slower timer interrupt when USB activity is low, but this really needs someone who knows what they are doing.
Moderator
Moderator
Posts: 3859
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by jecxjo » Fri Jun 08, 2012 4:24 pm
@dom thanks for the link to the mailing list. I knew there had to be a a RPI related kernel developers group, just didn't know where.
IRC (w/ SSL Support): jecxjo.mdns.org
Blog: http://jecxjo.motd.org/code
ProjectEuler Friend: 79556084281370_44d12dd95e92b1d9453aba2bdc94101b
User avatar
Posts: 157
Joined: Sat May 19, 2012 5:22 pm
Location: Ames, IA (USA)
by shirro » Fri Jun 08, 2012 4:27 pm
The funny thing is that even with 8000 interrupts per second the Pi still misses some of my key presses. You would think it could just do a poll every hundredth of a second and see I am not pressing a key anymore :-)

And @dom, would it be correct to say this is a feature of the driver or the hardware? I get the impression, perhaps wrongly, that we have the equivalent of a windows gdi printer rather than the fancier pcl/postscipt model. So the driver has to make up for the hardwares lack of smarts. So even a complete driver rewrite would face the same barrage of interrupts and all that can be done is to service them faster?
Posts: 248
Joined: Tue Jan 24, 2012 4:54 am
by dom » Fri Jun 08, 2012 5:01 pm
@shirro
(My knowledge on this is pretty limited. We bought in a block of hardware and got with it the dwc_otg linux driver. No one at Broadcom knows in detail how it works).
Yes, the USB hardware is "dumb" and needs to be told what to do (by the core) every frame. More advanced USB hardware blocks would handle the simple responses in hardware, and so produce less load on the CPU (esp. when idle).

My guess is that for the idle case, the ISR goes off, we read a register to say "has anything interesting occurred" and usually return without doing any more.
Going through the generic linux interrupt handlers probably means this takes tens of thousands of cycles. An assembly coded FIQ routine could probably reduce this by a factor of ten or more.
(but this is all speculation)
Moderator
Moderator
Posts: 3859
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by dom » Fri Jun 08, 2012 5:09 pm
I captured some benchmarks:

Quake with USB working: 22.5fps
Quake with USB killed: 26.8fps

XBMC with USB working: 27fps
XBMC with USB killed: 32fps

I was killing USB with (as root)
echo 1 > /sys/devices/platform/bcm2708_usb/bussuspend

So ~20% better.
Moderator
Moderator
Posts: 3859
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by obarthelemy » Fri Jun 08, 2012 6:22 pm
To me, USB is one on the great mysteries of life. If only firewire....
Posts: 1399
Joined: Tue Aug 09, 2011 10:53 pm
by tufty » Fri Jun 08, 2012 6:47 pm
Wow. The USB driver is taking 20% of cpu time?

:boggle:
Posts: 1330
Joined: Sun Sep 11, 2011 2:32 pm
by jamesh » Fri Jun 08, 2012 7:08 pm
Unfortunately it's a less than efficient driver, as provided by the chip supplier. The link above to the kernel mailing list is worth a read.
Moderator
Moderator
Posts: 10528
Joined: Sat Jul 30, 2011 7:41 pm
by mahjongg » Fri Jun 08, 2012 7:23 pm
Wow, then there is a potential 20% speed increase possible if we can fix this driver!

If that is impossible, then perhaps for games we can design a non USB keypad, using the GPIO's. :mrgreen:
User avatar
Moderator
Moderator
Posts: 4495
Joined: Sun Mar 11, 2012 12:19 am
by liz » Fri Jun 08, 2012 10:24 pm
jamesh wrote:Unfortunately it's a less than efficient driver, as provided by the chip supplier.


The IP core supplier's USB drivers, actually. We're hoping to get something better in place asap - and as James says, the kernel mailing list is worth a read.
User avatar
Foundation
Foundation
Posts: 3898
Joined: Thu Jul 28, 2011 7:22 pm
by johnbeetem » Fri Jun 08, 2012 11:58 pm
Here's my understanding of USB:

The problem with USB is that the host has to poll all devices to see if there's any work that needs to be done. A device cannot interrupt the host to tell the host that it needs service, so the host must poll each device often enough or risk losing receive data.

Intelligent host software figures out from the individual device specs how often it needs to poll each device. This can be very infrequent for a keyboard or mouse. For a 100Mb/s Ethernet device, the polling rate depends on how large the Ethernet's receive buffer is. The smaller the buffer, the more often the host needs to poll the device to ensure there is no receive data loss.

A good USB driver optimizes all this. It sounds like the one currently being used performs excessive polling to make sure it doesn't miss any receive data.
User avatar
Posts: 938
Joined: Mon Oct 17, 2011 11:18 pm
Location: The Coast
by shirro » Sat Jun 09, 2012 12:39 am
No expert but google tells me usb frames happen every 1ms. USB 2 divides them up into 8 microframes to increase performance. So 8000 of them a second. The thing isn't being driven by events as you might expect but by a clock.

I am guessing you can't slow the clock down or the usb device doesn't just lose data (I expect silicon can't be spared for huge buffers) but wouldn't even know what to do next. If I am just using the keyboard and the mouse I wonder if it can't be configured to go USB1 mode and check 1000 times per second?

I used to have a Sinclair ZX80 which couldn't display video and process things at the same time. It seemed a reasonable compromise to own a computer at the time. At least now I know I can wrap a demo program in a script to disable and enable usb. It brings back old times.

Rather more annoying, somewhere along the line I suspect it is missing one of those 8000 interrupts just as I am lifting my finger off a key. PS/2 interfaces can't be overly complicated can they? All those GPIO pins. Interrupt driven possible? Would buy a kit.
Posts: 248
Joined: Tue Jan 24, 2012 4:54 am
by rgh » Sat Jun 09, 2012 9:59 am
OK, thanks for the info and the link to the kernel mailing list. Interesting reading.
Posts: 219
Joined: Fri Nov 25, 2011 3:53 pm
by Morgaine » Sat Jun 09, 2012 10:54 am
dom wrote:(My knowledge on this is pretty limited. We bought in a block of hardware and got with it the dwc_otg linux driver. No one at Broadcom knows in detail how it works.)

Never put yourselves at the mercy of closed software. It always bites you eventually.
Intolerance is a failure of education. Education is predicated on tolerance of the uneducated.
User avatar
Posts: 141
Joined: Mon Mar 12, 2012 1:13 am
by shirro » Sat Jun 09, 2012 1:34 pm
Morgaine wrote:Never put yourselves at the mercy of closed software. It always bites you eventually.


The dwc_otg driver is many things but it isn't closed. The source is here and here. Though reading what has been written it is probably fair to say nobody really understands it well enough to rewrite it without additional documentation.
Posts: 248
Joined: Tue Jan 24, 2012 4:54 am
by johnbeetem » Sat Jun 09, 2012 2:46 pm
shirro wrote:The dwc_otg driver is many things but it isn't closed. The source is here and here. Though reading what has been written it is probably fair to say nobody really understands it well enough to rewrite it without additional documentation.

This is a problem with a lot of FLOSS. While the source code is there, it can be hard to do anything with it without a lot of exploration, detective work, and reverse engineering. It often seems easier (and more fun) to write something new rather than adapt something that exists.
User avatar
Posts: 938
Joined: Mon Oct 17, 2011 11:18 pm
Location: The Coast
by reggie » Sat Jun 09, 2012 3:20 pm
I think that's the issue here isn't it though? Lack of proper documentation, all well and good having source code but without documentation you're probably stumbling around in the dark slightly and having to use other strategies to find documentation. Like looking at other targets that use synopsys drivers, or arasan drivers for the emmc chip and hoping that they're using the same controller. But it ends up being incredibly galling when the only documentation they give you for the pi alludes to you using other documentation that simply isn't available. How are kids/teachers going to be expected to try and overcome issues if people that do know what they're doing struggle?
Posts: 151
Joined: Fri Aug 26, 2011 11:51 am
by obarthelemy » Sat Jun 09, 2012 11:15 pm
I think the idea is that experts will solve these issues for them. Once the faulty drivers are fixed, nobody needs to worry about them anymore, except people who actually want to ^^
Posts: 1399
Joined: Tue Aug 09, 2011 10:53 pm
by Morgaine » Sun Jun 10, 2012 6:36 am
reggie wrote:I think that's the issue here isn't it though? Lack of proper documentation, ...

Yep, I agree fully there. Without reasonable documentation on hardware APIs, it's in the lap of the gods how good FOSS code can ever be, even with very dedicated and competent reverse-engineering effort. You simply cannot guess with any high degree of certainty what low-level operations are actually doing even when you see them happening before your very eyes with debug equipment.

No hardware API documentation == black mark.
Intolerance is a failure of education. Education is predicated on tolerance of the uneducated.
User avatar
Posts: 141
Joined: Mon Mar 12, 2012 1:13 am
by tufty » Sun Jun 10, 2012 7:27 am
obarthelemy wrote:I think the idea is that experts will solve these issues for them. Once the faulty drivers are fixed, nobody needs to worry about them anymore, except people who actually want to ^^

Th problem, however, is that the experts appear very unlikely to solve these issues, at least not by taking the vendor-supplied buggy-as-hell drivers as a starting point. It's almost evident that this is the case, given that the dwc drivers have been out there for a couple of years or more, and they are still, not to put a fine a point on it, broken.

Generally speaking, in linux-land, if it's not already been done, it's probably either a really stupid idea, or it's really, really, hard to do. This applies for GL/ES or OpenVG accelerated X, too.

Does my pessimism show in this?

Simon
Posts: 1330
Joined: Sun Sep 11, 2011 2:32 pm
by reggie » Sun Jun 10, 2012 2:10 pm
I Agree that once the drivers are fixed that no one needs to worry about them but this project is based around people learning to program on these devices, whether it's device drivers or controlling robots, it doesn't really matter, documentation is key. Coupled with the fact that whilst the original 10k was supposed to be in the hands of developers and tinkerers but is now in the hands of the masses means that there are a lot of people having bad experiences with their pi that won't understand why. This will include some of their target market that jumped the gun before the edu unit is released and adopted early.

It's an unfortunate symptom of the foundation doing their best for everyone, they're a victim of their own success here. It would be nice if the foundation could give us some clarity over documentation and whether there is any extra documentation we can have or they can talk to their partners about us having or if anyone at the foundation who worked on driver implementation is allowed to discuss any aspects of the code.

I do appreciate that there are NDAs in place and people can't just hand stuff over.
Posts: 151
Joined: Fri Aug 26, 2011 11:51 am
by Morgaine » Sun Jun 10, 2012 4:31 pm
In a nutshell, this SoC wasn't designed for use in a networked general purpose computer with multiple USB devices, and the continuous clock-driven polling of the USB tries to make up for it, but poorly and with high overhead.

Does Broadcom provide a more useful version of this device containing a built-in Ethernet MAC, which could be used in a future Pi?
Intolerance is a failure of education. Education is predicated on tolerance of the uneducated.
User avatar
Posts: 141
Joined: Mon Mar 12, 2012 1:13 am