USB FIQ testing....

[gsh]

So,

I've finally got something that I think works quite well with the devices I have. This fix should help remove any problems with keyboards repeating / dropping keys...

https://github.com/ghollingworth/linux.git

It may well break other things, such as isochronous endpoints (I've not put in any special cases yet) so I'd be interested if anyone can test with something like that...

Disclaimer:

To try it out, you're going to have to be able to rebuild your own kernel, if you can't do that then this is not the right time for you to try out the patch. Second, there are a couple of BUG()'s in there that are specifically there to catch the bad cases (such as ever getting a NYET response from a hub). This isn't actually a bug at all but is useful to know!

To build this, I'd suggest you turn on kernel debug (under kernel hacking in the menuconfig) and have a serial debug available...

Known issues:

1) Isochronous endpoints have not been tested and I know they are different but haven't been bothered yet to put any thought into why they would fail, so they probably do!
2) There are two main bug's you could get rid of it you hit them, the first is the NYET bug on line 266 of dwc_otg_hcd_intr.c. You can safely remove this with no loss of packets. The second is the NYET ERR bug on line 291, if you hit this then you have dropped a packet and is indicative of the type of packet loss we were seeing on keyboards... It doesn't mean it's the end of the world, but I would like to know about it! (remove the BUG and see if its better anyway)


So to confirm, have a play and see if it fixes some of your problems, if not try to understand which of the BUGs you are hitting there is further debug available if you have a serial console connected and understand the kdb interface...

Would be interested in knowing if there are any combinations of devices that fail with this software...

Thanks and have fun

Gordon Hollingworth

[gritz]

Excellent! Given the importance of the work on the USB I/O perhaps it might be an idea to make this topic a sticky for the duration of the test, so's it doesn't risk dropping into page 2 obscurity (if there's no replies for a day or two) as well as a note about not posting offtopic issues (like I am now, oops) just in case it gets really busy. Cheers.

[MattF]

Hi Gordon,

Would you expect this to work on a kernel tree that has been bumped up to 3.7.10?

I'm based on https://github.com/raspberrypi/linux/commits/rpi-3.6.y but haven't applied every single one of the latest commits. I also have aufs added in, and some critical ath9k fixes from wireless next.

Sadly I can't even get a kernel that boots, even with nothing connected on USB at all. Is there anything special I need in cmdline.txt ?

I've noticed you havent applied cc68e81acc17773029333dd3a0201ef347ab0b38 to your 3.8.y-temp tree yet.

Building your tree now to see if it breaks any of my peripherals.

Thanks, keep up the good work.

[MattF]

Hmm,

Having built your tree, even having a keyboard plugged in results in a very early panic in boot.
cmdline.txt :
dwc_otg.fiq_fix_enable=1 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait consoleblank=0 panic=10 panic_on_oops=1 usbcore.old_scheme_first=1 noapic usbcore.autosuspend=-1

config.txt.gz and screenshot of panic attached (sorry, not really setup for proper debug this evening, but more could be done if needed)

Perhaps I have something dumb in cmdline.txt or .config which is preventing boot.

Thanks

[gsh]

Thanks for having a go, I've got a full debug output from Dom as well and it looks like you're tripping over a start of frame interrupt BUG...

I'm going to have a look at it later and see if I can understand whats going wrong...

@MattF do you have a serial debug connected to the GPIO's ?

If you do then you can output the debug easier... (i.e. use a terminal and log the output to a file and post it...)

Also, have a look at the wiki on github.com/ghollingworth/linux later and I'll add some further information on how to get more debugging out...

Thanks

Gordon

[gsh]

Ah OK,

I believe there is was a screw up when merging the code back together when creating my new release...

I'm now also hitting the same problem, am hoping to get a look into it between other things, but am nose to the grindstone this week!

Will post when I've fixed the problem, add a watch on this thread to be notified of updates...

Gordon

[gsh]

OK,

I've updated the ghollingworth/linux tree to fix one of the bugs, I was hoping to catch any cases of NYET packet occuring, thinking that we shouldn't be seeing many of them now, but actually thats not the case!

So this release should be better, but still has lots of BUG_ONs inserted so I'd be interested in any crashes that do occur when you go plugging stuff in!

Thanks

Gordon

[MattF]

Hi,

No change to test this weekend I'm afraid as I'm away, but I haven't forgotten about it

[ddv2005]

Kernel crashes on opening USB audio device for play & record:

Code: Select all

[   65.738307] hcd->hub_port[0] = 0x0
[   65.741853] hcd->hub_port[1] = 0x0
[   65.745352] hcd->hub_port[2] = 0x8
[   65.748847] hcd->hub_port[3] = 0x0
[   65.752342] hcd->hub_port[4] = 0x0
[   65.755836] hcd->hub_port[5] = 0x0
[   65.759331] hcd->hub_port[6] = 0x0
[   65.762824] hcd->hub_port[7] = 0x0
[   65.766318] hcd->hub_port[8] = 0x0
[   65.769811] hcd->hub_port[9] = 0x0
[   65.773305] hcd->hub_port[10] = 0x0
[   65.776888] hcd->hub_port[11] = 0x0
[   65.780467] hcd->hub_port[12] = 0x0
[   65.784047] hcd->hub_port[13] = 0x0
[   65.787611] hcd->hub_port[14] = 0x0
[   65.791191] hcd->hub_port[15] = 0x0
[   65.794906] ------------[ cut here ]------------
[   65.799635] kernel BUG at drivers/usb/host/dwc_otg/dwc_otg_hcd.c:1327!
[   65.806301] Internal error: Oops - BUG: 0 [#1] PREEMPT ARM
[   65.811902] Modules linked in: spidev snd_usb_audio snd_pcm snd_page_alloc snd_hwdep snd_usbmidi_lib snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_timer snd_seq_device uvcvideo videobuf2_core evdev leds_gpio snd led_class videodev media videobuf2_vmalloc videobuf2_memops spi_bcm2708 i2c_bcm2708
[   65.841264] CPU: 0    Not tainted  (3.6.11+ #14)
[   65.847476] PC is at dwc_otg_hcd_allocate_port+0x10c/0x11c
[   65.854573] LR is at dwc_otg_hcd_allocate_port+0x10c/0x11c
[   65.861601] pc : [<c029c70c>]    lr : [<c029c70c>]    psr: 200001d3
[   65.861601] sp : dce7fe20  ip : 00000000  fp : c05370a9
[   65.876196] r10: dd98c830  r9 : dd98c860  r8 : dd9dc980
[   65.883068] r7 : dce7fe4c  r6 : 00000010  r5 : dd9dc980  r4 : dd98c920
[   65.891316] r3 : 00000015  r2 : 20000193  r1 : 00000001  r0 : 200001d3
[   65.899633] Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment user
[   65.908847] Control: 00c5387d  Table: 1da98008  DAC: 00000015
[   65.916567] Process auddemo (pid: 1620, stack limit = 0xdce7e268)
[   65.924717] Stack: (0xdce7fe20 to 0xdce80000)
[   65.931192] fe20: 00000002 00000003 dd98c800 dd9dc9a8 00000010 c029c810 00000001 dd98c838
[   65.943842] fe40: 00000000 00000000 00000000 00000000 bededdcc dd98c800 dd98c828 f8840000
[   65.956948] fe60: 00000008 dd916000 c052204c c05e937c c0542037 c029f478 dd98c800 f301080e
[   65.970533] fe80: 00000008 c02a0a18 c029d8a4 a0000193 00000000 00000000 00000000 00000020
[   65.984390] fea0: c0521ff4 c029d8b0 c029d8a4 c0274884 dd97eae0 c0078e3c bededb70 c25c4110
[   65.998339] fec0: ddbcd7e0 c0521ff4 dce7e000 00000000 dce7ff5c c000db88 dce7e000 00000000
[   66.012312] fee0: 00000000 c0078ff0 00000008 dd916000 c001a3cc dce7e000 c0521ff4 c007b354
[   66.026317] ff00: c007b2bc c0531120 00000020 c0078730 0000014a c000e8bc c00c3c34 80000013
[   66.040565] ff20: f200b200 c039ff14 00000005 dce7ff8c 00000100 dcef0d60 00000000 dce7ff8c
[   66.055039] ff40: dd81dec0 00000005 c000db88 dce7e000 00000000 00000000 00c5387d dce7ff70
[   66.069748] ff60: c00d36e8 c00c3c34 80000013 ffffffff 00020001 0180a9b0 bededb70 c25c4110
[   66.084639] ff80: 00000005 c00d36e8 dce7e000 00000000 00000000 0180a9b0 bededb70 0180aa00
[   66.099618] ffa0: 00000036 c000da00 0180a9b0 bededb70 00000005 c25c4110 bededb70 00020001
[   66.114711] ffc0: 0180a9b0 bededb70 0180aa00 00000036 b6e2993c b6e28734 b6e294f0 00000000
[   66.129815] ffe0: 0007ff07 bededb40 b6e14c94 b6d2c53c 20000010 00000005 00000000 00000000
[   66.144938] [<c029c70c>] (dwc_otg_hcd_allocate_port+0x10c/0x11c) from [<c029c810>] (dwc_otg_hcd_select_transactions+0xf4/0x35c)
[   66.163423] [<c029c810>] (dwc_otg_hcd_select_transactions+0xf4/0x35c) from [<c029f478>] (dwc_otg_hcd_handle_sof_intr+0x98/0xdc)
[   66.181903] [<c029f478>] (dwc_otg_hcd_handle_sof_intr+0x98/0xdc) from [<c02a0a18>] (dwc_otg_hcd_handle_intr+0x1fc/0x274)
[   66.199759] [<c02a0a18>] (dwc_otg_hcd_handle_intr+0x1fc/0x274) from [<c029d8b0>] (dwc_otg_hcd_irq+0xc/0x18)
[   66.216468] [<c029d8b0>] (dwc_otg_hcd_irq+0xc/0x18) from [<c0274884>] (usb_hcd_irq+0x30/0x40)
[   66.231950] [<c0274884>] (usb_hcd_irq+0x30/0x40) from [<c0078e3c>] (handle_irq_event_percpu+0x50/0x1b0)
[   66.248303] [<c0078e3c>] (handle_irq_event_percpu+0x50/0x1b0) from [<c0078ff0>] (handle_irq_event+0x54/0x84)
[   66.265091] [<c0078ff0>] (handle_irq_event+0x54/0x84) from [<c007b354>] (handle_level_irq+0x98/0x108)
[   66.281249] [<c007b354>] (handle_level_irq+0x98/0x108) from [<c0078730>] (generic_handle_irq+0x30/0x48)
[   66.297585] [<c0078730>] (generic_handle_irq+0x30/0x48) from [<c000e8bc>] (handle_IRQ+0x30/0x84)
[   66.313308] [<c000e8bc>] (handle_IRQ+0x30/0x84) from [<c039ff14>] (__irq_svc+0x34/0xc8)
[   66.328223] [<c039ff14>] (__irq_svc+0x34/0xc8) from [<c00c3c34>] (fget_light+0xd0/0x108)
[   66.343215] [<c00c3c34>] (fget_light+0xd0/0x108) from [<c00d36e8>] (sys_ioctl+0x1c/0x5c)
[   66.358201] [<c00d36e8>] (sys_ioctl+0x1c/0x5c) from [<c000da00>] (ret_fast_syscall+0x0/0x30)
[   66.373528] Code: e0844105 e5840124 eaffffe9 eb000b30 (e7f001f2)
[   66.383096] ---[ end trace 1815cd305f115d5c ]---
[   66.391114] Kernel panic - not syncing: Fatal exception in interrupt

I guess that your patch does not support more than 2 split transactions simultaneously on same hub but USB audio required it.

[gsh]

It should work, it just won't interleave the transactions...

i.e. rather than

Start A
Start B
Complete A
Complete B

It will do

Start A
Complete A
Start B
Complete B

I think its more likely that the problem is to be with the wrong scheduling (I've already found another bug here) I'm going to look into it this evening...

Gordon

[ddv2005]

It should work, it just won't interleave the transactions...
It will do

Start A
Complete A
Start B
Complete B

It does not work for USB audio because duration of each transaction about 80-100% of time. Both transactions needs more than 100% of time for one channel and it can't be combined to one channel.

[ddv2005]

Gordon,
Do you have documentation on USB controller or hardware USB analyzer? Because I suspect that not all host channels are equal. It is looks like hc#7 has more priority than hc#0. And if you schedule some transaction on hc#0 and another transaction to hc#7 then second transaction go first.
I asked Synopsis for documentation but no luck.

[jamesh]

Gordon,
Do you have documentation on USB controller or hardware USB analyzer? Because I suspect that not all host channels are equal. It is looks like hc#7 has more priority than hc#0. And if you schedule some transaction on hc#0 and another transaction to hc#7 then second transaction go first.

Yes, he does have documentation, but more importantly he has the RTL for the Synopsis USB hardware block. I believe he also has (or had) access to a USB analyser.

[M33P]

Gordon,
Do you have documentation on USB controller or hardware USB analyzer? Because I suspect that not all host channels are equal. It is looks like hc#7 has more priority than hc#0. And if you schedule some transaction on hc#0 and another transaction to hc#7 then second transaction go first.
I asked Synopsis for documentation but no luck.

I've seen an analogous situation to this while profiling individual HC transfer completions with multiple periodic transactions active (at USB2.0 speed). Certain HCs/transactions end up always being completed before others - and at variable points within each microframe.

If the hardware doesn't guarantee FIFO behaviour for SPLIT transactions then we're a bit boned for queuing multiple transactions into a hub's TT (which is itself a FIFO).

[ddv2005]

It should work, it just won't interleave the transactions..

Why you won't interleave the transactions? Is it hardware limitation or something else?

[gsh]

Basically:

To schedule something for a periodic endpoint (interrupt or isochronous) you have to make sure it gets queued in the frame before... Whereas if you want to schedule a bulk or control endpoint you can send it in the current frame and it'll go out immediately.

But what happens if I start two split transactions down the same TT in the same uframe, when I come to complete the transaction if the first one replies NYET I mustn't retry with the second one or it will fail with NYET and the data will be lost forever (the usual failure case of the hub!). But if I only queue the request for the first one then if that succeeds then the second one will fail because the complete won't go out in the same frame...

It's interesting that there's a difference between the channels, because although I said the above I've found I can send some interrupt transactions in the current microframe if I set the odd/even frame setting correctly... So I may re-evaluate this sometime in the future.


But anyway this shouldn't effect your Audio, are you saying you've got multiple isochronous transfers every frame?

Gordon

[ddv2005]

But anyway this shouldn't effect your Audio, are you saying you've got multiple isochronous transfers every frame?

I don't have hardware USB analyzer and I don't know what happened on low level but I did software USB capture:

Code: Select all

      1 0.000000000    4.1                   host                  USB      160    URB_ISOCHRONOUS out
      2 0.000037000    host                  4.1                   USB      1312   URB_ISOCHRONOUS out
      3 0.000372000    4.2                   host                  USB      176    URB_ISOCHRONOUS in
      4 0.000399000    host                  4.2                   USB      80     URB_ISOCHRONOUS in
      5 0.001113000    4.2                   host                  USB      176    URB_ISOCHRONOUS in
      6 0.001138000    host                  4.2                   USB      80     URB_ISOCHRONOUS in
      7 0.010363000    4.2                   host                  USB      176    URB_ISOCHRONOUS in
      8 0.010393000    host                  4.2                   USB      80     URB_ISOCHRONOUS in
      9 0.011114000    4.2                   host                  USB      176    URB_ISOCHRONOUS in
     10 0.011140000    host                  4.2                   USB      80     URB_ISOCHRONOUS in
     11 0.020367000    4.2                   host                  USB      176    URB_ISOCHRONOUS in
     12 0.020395000    host                  4.2                   USB      80     URB_ISOCHRONOUS in
     13 0.021115000    4.2                   host                  USB      176    URB_ISOCHRONOUS in
     14 0.021141000    host                  4.2                   USB      80     URB_ISOCHRONOUS in
     15 0.030368000    4.2                   host                  USB      176    URB_ISOCHRONOUS in
     16 0.030397000    host                  4.2                   USB      80     URB_ISOCHRONOUS in
     17 0.030763000    4.1                   host                  USB      176    URB_ISOCHRONOUS out
     18 0.030789000    host                  4.1                   USB      1520   URB_ISOCHRONOUS out

At each time it has 2 scheduled transactions simultaneous. As soon one transaction is finished software will schedule it again. It is mean that this transactions can't be interleaved.

It's interesting that there's a difference between the channels, because although I said the above I've found I can send some interrupt transactions in the current microframe if I set the odd/even frame setting correctly... So I may re-evaluate this sometime in the future.

If you have hardware USB analyzer ...could you just sniff the traffic between host controller and high speed HUB for any USB audio device? One for PC and another one for Pi. I guess that it can help to understand what's wrong with split transactions on Pi.

[gsh]

It depends upon what the scheduling rate is, interrupt and isochronous endpoints get scheduled continuously, but they only get sent at their requested rate. This then depends upon the audio rate that you have selected...

I think you're limit is about four endpoints each with a polling interval of 1. Then for each on we'd send the following (This absolutely requires perfect interrupt scheduling to achieve without dropping packets!)

Code: Select all

uFrame .7:    Split Start A
uFrame .0:
uFrame .1:    Split Complete A
              Split Start B
uFrame .2:
uFrame .3:    Split Complete B
              Split Start C
uFrame .4:
uFrame .5:    Split Complete C
              Split Start D
uFrame.6:
uFrame .7:    Split Complete D
              Split Start A

I was going to look into isochronous packet comms when I've finished with Interrupt.

Gordon

[gsh]

OK,

For those interested in testing the FIQ code I've updated it today and it all seems to work very very well, (in fact better than I expected it to work!)

Was able to run audio loopback with a daffodil usb device whilst simultaneously looping back USB serial and had pretty much every USB 1 device I own plugged into various hubs!

So I think its fairly stable, but of course wouldn't be in the slightest bit surprised to find that everyone else can even get it to boot (par for the course unfortunately!)

Gordon

[obcd]

thanks for the hard work.
Is it possible to get a short description of what needs to be done to obtain the stuff for testing, or does that exist somewhere and did I simply miss it?

[gsh]

Its the usual thing for building the kernel...

I'd suggest following the instructions on the http://elinux.org/RPi_Kernel_Compilation

But use my kernel sources from

https://github.com/ghollingworth/linux

It's well worth getting up to speed doing this, it'll take a couple of hours the first time, but after that it means you can play as much as you like!

I use ubuntu

Then:

> apt-get install gcc-arm-linux-gnueabi make ncurses-dev
> git clone https://github.com/ghollingworth/linux
> make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig
> make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-
> make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=<pathformodules> modules_install

Now you need to copy arch/arm/boot/Image to your FAT partition on the SDCard and overwrite kernel.img

> cp arch/arm/boot/Image /mnt/vfat/kernel.img

And copy the modules across to the ext4 partition

cp ./sd_install/lib/modules/* /mnt/ext4/lib/modules/

And then reboot...

Gordon

[gsh]

For those interested in testing, there is another update. I had realised that my testing wasn't working and that I was actually sending most of my USB packets on the wrong microframe...

Now fixed, it clearly does not work with ISOC based devices right now, that's my next step.

Gordon

[gsh]

Fixed Isochronous OUT endpoints and tested with Daffodil USB device (just did loopback)

Also fixed a problem with dequeuing URBs which has been a problem with the USB host driver forever... I think it's the reason you get the ethernet dropping out in big soak tests...

Gordon

[richardp]

USB_idle_trace.png (58.58 KiB) Viewed 22014 times

Here is a trace from a idle USB hub (with a sound adapter attached) The trace file is filled up with Incomplete transactions quite quickly. The Kernel being used is not the latest, but a week old (with FIQ, but not the latest change)

gsh, My offer still stands..
If you are in cambs, I can drop off the USB Analyser, thats if you would find it useful.
I am just groping in the dark as this is all new to me... I see a tons of NAK and Incomplete.

Richard

[gsh]

Not really,

I've got a very much more expensive one on my desk!

Thanks anyway

Gordon