mandersen
Posts: 3
Joined: Fri Jul 27, 2012 6:46 pm

High interrupt rate on embedded NIC

Tue Jul 31, 2012 8:53 pm

I see a stable, high interrupt rate on my Pi. Interrupt 75 is getting triggered no less than 8000 times per second. Here measured with nothing connected in the 2 USB ports:

Code: Select all

 pi@raspberrypi ~ $ grep dwc_otg /proc/interrupts ; sleep 10 ; grep dwc_otg /proc/interrupts 
 75:  104821321   ARMCTRL  dwc_otg, dwc_otg_hcd:usb1
 75:  104901729   ARMCTRL  dwc_otg, dwc_otg_hcd:usb1
pi@raspberrypi ~ $
As far as I understand, the NIC is the only wired to the USB lanes on the board. 'lsusb' says:

Code: Select all

pi@raspberrypi ~ $ sleep 5 ; lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. 
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. 
pi@raspberrypi ~ $ 
There is absolutely no ethernet traffic on the connected network that can warrant 8000 interrupts per second.

Is this a known problem? -NIC chip or driver broken?

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5608
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: High interrupt rate on embedded NIC

Tue Jul 31, 2012 9:21 pm


mandersen
Posts: 3
Joined: Fri Jul 27, 2012 6:46 pm

Re: High interrupt rate on embedded NIC

Tue Jul 31, 2012 10:46 pm

Since the interesting stuff was actually in the U-Boot mailing list and other thread got a bit side tracked with irrelevant USB voltage related problems/observations - I am summing up to the best of my abilities here:

Status of the 8000 interrupts/sec problem
The 'dwc_otg' driver for the USB root hub (and the SMC NIC) in the Pi is doing 8000 idle interrupts per second. Tests have shown that disabling the driver and its 8000 interrupts per second can increase the processing performance of the Pi by as much as 20%.

This 'dwc_otg' driver is provided by Synopsis (not Broadcom). The source code for the driver is open but it is included in the Linux kernel tree - nor likely to become so, because it does not live up to established standards.

The current version of the driver included in Raspbian 2012/07/15 has a so-called 'SOF_reduction' setting which can reduce the interrupt load - but also results in some unfortunate side effects that impact responsiveness.

Stephen Warren (and possibly others) are working on a replacement driver for the USB root hub - but it is not yet fully functional nor available in Raspbian.


Please correct me as required ;)

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5608
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: High interrupt rate on embedded NIC

Tue Jul 31, 2012 11:04 pm

@manderson
Sounds pretty accurate.
Additionally Gordon (gsh) has been working on the FIQ solution. Still 8000 interrupts per second, but the FIQ takes about 10% of the time that a full ISR takes, so the hope is the idle cost is reduced from 20% to 2%.

chrisw2
Posts: 106
Joined: Sat Apr 07, 2012 11:22 am
Location: Manchester, UK

Re: High interrupt rate on embedded NIC

Tue Jul 31, 2012 11:19 pm

@manderson

Seems a fair summary to me but I think you missed out a "not" as in:
"...is not included in the Linux kernel tree"

You could also have added that the hardware documentation is Synopsis private which probably makes rewriting/improving the code very hard even for experts.

This is what Greg Greg Kroah-Hartmann (see: https://en.wikipedia.org/wiki/Greg_Kroah-Hartman ) had to say about it here: http://lists.infradead.org/pipermail/li ... 00042.html
> > Oh, and why no documentation on the hardware, I thought we had it around
> > here somewhere?
>
> The documentation is Synopsys-private. Only people with certain types of
> Synopsys SolvNet accounts can see it (there are few around). However the
> Ralink RT3050 uses the same core, and there's a register map in the
> datasheet:
> http://www.tracermcc.ru/foto/bender/RT3 ... 8_0902.pdf
> AFAICS it doesn't look much like EHCI, but I'm not so familiar with that.

Without any open specs, I really can't work on the code, unfortunately :(
How hard can Broadcom lean on people to get this opened up so that we
can get working code?

drgeoff
Posts: 11125
Joined: Wed Jan 25, 2012 6:39 pm

Re: High interrupt rate on embedded NIC

Tue Jul 31, 2012 11:53 pm

On one hand it is heartening to at last see that expert people are aware of the problem and are trying to fix it. However there doesn't seem to be any mention of the packet loss problem that thexman has been moaning about for ages nor all the other USB problems with hubs, with non-working or kernel-panicking keyboards, with losing network connectivity upon starting X, etc. Are all of the problems stemming from this Synopsys hardware/software?

The disheartening sides are that 1) the thread on infradead.org goes back almost two months and 2) it contains a fair amount of pessimism about a fix given that the hardware details are not known.
Quis custodiet ipsos custodes?

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1583
Joined: Sat Sep 10, 2011 11:43 am

Re: High interrupt rate on embedded NIC

Wed Aug 01, 2012 5:48 am

As Dom say's I'm working on it, but can only do one thing at a time...

FIQ implementation works fine on my desk, but had trouble when Dom tried to use it! Need to work harder! But it did reduce the overhead from 8000 to 800 IRQ's per second.

Currently looking into the packet dropping issue with 1.0 / 1.1 devices on a 2.0 hub... This is caused by split transactions going wrong, but currently I'm finding it very difficult to track down and having to add my own debug interface because using kernel debug is way too slow!

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

vmp32k
Posts: 14
Joined: Fri Jul 27, 2012 3:05 pm

Re: High interrupt rate on embedded NIC

Wed Aug 01, 2012 12:52 pm

http://qkme.me/3qb78m :D

Good to know someone's working on it, I was kinda loosing faith in the usefulness of my Pi already. ;)

calebzulawski
Posts: 46
Joined: Tue Jan 31, 2012 6:00 am

Re: High interrupt rate on embedded NIC

Wed Aug 01, 2012 1:18 pm

I'm not sure that this is the proper place to post this... but is it possible to disable the NIC completely to totally disable this interupt? I only need USB for my project. Thanks!

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: High interrupt rate on embedded NIC

Wed Aug 01, 2012 1:24 pm

It's the usb driver that needs the interrupts. An usb frame is 1ms and it's divided into 8 microframes for usb 2.0. It seems to interrupt every micro frame. Gordon is now working on a method to quick determine if there is actual work to do, so that the interrupt handler kicks in less frequent. At least that's how I understand it. There is a script somewhere around here to disable the usb2ethernet driver. I doubt this will make much difference.

calebzulawski
Posts: 46
Joined: Tue Jan 31, 2012 6:00 am

Re: High interrupt rate on embedded NIC

Wed Aug 01, 2012 1:30 pm

Oh, I understand. Thanks for clarifying.

Return to “Troubleshooting”