gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1177
Joined: Sat Sep 10, 2011 11:43 am

Re: USB packet loss.

Fri Sep 14, 2012 8:19 am

thexman wrote:I'll boot up my 26 USB devices and run the scripts and see if it all falls over its the best test you can get its 60 odd io with a64 k script asking for 30 inputs and switching 30 out put channels over the USB twin hubs each 13 channels with no back fed power lets keep fingers crossed


Thanks

It wouldn't surprise me to find this fails, if you've got 26 devices with half having interrupt / isochronous endpoints then you're just going to end up with huge numbers of interrupts which will slow the box to nothing and probably make it crash as well...

I guess it depends if your script does it one at a time or all together...


Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1177
Joined: Sat Sep 10, 2011 11:43 am

Re: USB packet loss.

Fri Sep 14, 2012 8:24 am

kyelo wrote:As you may remember, I am trying to do VOIP using a Logitech H530 USB headset, Twinkle and Raspbian.

I just now did the rpi-update and have 3.2.27+ #148 PREEMPT, with the fiq_fix_enable and naq_holdoff_enable enabled by default. I am also using microframe_schedule=1, enable_llm=1, and sync_after_dma=0.

Before this new code, I was able (on a lucky day) to get 1-2 sec into the dialing of a call before a reset occurred ("usb 1-1.2.2: reset full-speed USB device number 6 using dwc_otg" in dmesg).

Now I MAY be able to get 3 seconds into dialing a call or an echo test before the reset occurs. I am still (after the update) getting about 2500 interrups/sec (down from 8500 interrupts /sec baseline).

And I am using a powered 4-port hub (the USB headset is plugged into the hub) .
What do you mean a reset occurs? This sounds like a power problem, the Pi doesn't spontaneously reset (reset is either a software triggered thing that happens when you type `reboot` or by the power management browning out...)

I would suggest the problem you have is power, have you tried this on a known good hub

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

samsamsam
Posts: 36
Joined: Fri Aug 17, 2012 11:36 am

Re: USB packet loss.

Fri Sep 14, 2012 10:07 am

I think he mean: "dmesg shows that USB device has been reset"

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1177
Joined: Sat Sep 10, 2011 11:43 am

Re: USB packet loss.

Fri Sep 14, 2012 1:30 pm

OK, may need to reproduce that on my desk, but the only issues I've seen like this were power ones...
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

User avatar
Burngate
Posts: 4926
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK
Contact: Website

Re: USB packet loss.

Fri Sep 14, 2012 4:07 pm

obcd wrote:So, has it ever been tested with a model A without hub? It would limit the usb devices to just 1. You could connect a wireless keyboard/mouse dongle and check if the missing/stuck key issue still exists.
I've just been doing that.

Model B Rev1 (power-moded), Start.elf from 13 Sep, no network connected
Bog-standard kb & mouse ok on Raspbian & RISC OS
Rapoo E9070 wireless kb & mouse stutters on both (and sloooow mouse on RISC OS)

Model A Rev1 (power moded) Start.elf from 13 Sep
Bog-standard kb & mouse ok on Raspbian & RISC OS
Rapoo E9070 wireless kb & mouse ok on both

Yet to try with a hub

thexman
Posts: 259
Joined: Sat Apr 07, 2012 2:18 pm

Re: USB packet loss.

Fri Sep 14, 2012 4:52 pm

gsh wrote:

It wouldn't surprise me to find this fails, if you've got 26 devices with half having interrupt / isochronous endpoints then you're just going to end up with huge numbers of interrupts which will slow the box to nothing and probably make it crash as well...

I guess it depends if your script does it one at a time or all together...


Gordon
Yup all at the same time working on a 1.2ghz Linux box perfectly fine so fingers crossed
one armed controls engineer, my grammar is bad but lets face it most keyboards don't suit a one armed man

kyelo
Posts: 64
Joined: Sun Oct 23, 2011 6:31 pm

Re: USB packet loss.

Fri Sep 14, 2012 8:35 pm

INTERESTING DISCOVERY!!!

I have found that I can do VOIP if the Logitech USB headset is plugged directly into one of the R_pi's 2 USB ports but not into a USB hub!! It is not the highest quality audio in the world, but it does work. And the call recipient could hear and understand me! (I am using the same sip provider I use on other computers).

Luckily, my Logitech keyboard and mouse use the same dongle, so keyboard, mouse and headset can be serviced by the 2 usb ports on the R_pi. Of course I could not simultaneously also have a USB thumb drive, USB hard drive or Wi-Fi.

The next step was leave the Logitech USB headset plugged into my R_pi, and into the other R_pi USB port plug a 4-port Belkin "unlisted" USB hub; and then I plugged the mouse/keyboard dongle into the hub. I still had VOIP.

However, when I plug the headset into a USB hub and make a VOIP call, 100% of the time I get failure of the VOIP (with not connecting the call or connecting the call but loss of the voice after a couple of seconds.)

And invaribly associated with this failed call is the dmesg "reset full-speed USB device number [number for Logitech USB Headset]" AND if the dongle for the keyboard/mouse is still plugged into the same USB hub as the headset I also get the dmesg "reset full-speed USB device number [number for keybosrd/mouse dongle]".

I should also mention that my TP1/TP2 is 5.95. I am not currently over-volting but had overvolt set to 6 earlier (just haven't got around to it on the current card). But I think I read somewhere that if you do set it once, that it triggers a "sticky bit" that lingers...is this correct?

So the possibility arises that if one is not using overvoltage there may not be enough power in the R_pi's USB port to drive a headset.

I have done the above with wired ethernet and have not tested it with wi-fi (which of course would have to go on the hub).

If indeed this is a USB driver issue, it would be nice to see it resolved so that one has a choice of where to plug the headset. But at least now I know how to do VOIP on my Pi!

drgeoff
Posts: 7223
Joined: Wed Jan 25, 2012 6:39 pm

Re: USB packet loss.

Sat Sep 15, 2012 9:49 am

kyelo wrote: So the possibility arises that if one is not using overvoltage there may not be enough power in the R_pi's USB port to drive a headset.
The 'overvoltage' only increases the voltage to the Broadcom CPU/GPU chip and should have no affect on the voltages and signal levels on the USB sockets.

Your TP1/TP2 voltage is higher than I would feel comfortable with.

obcd
Posts: 908
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Sat Sep 15, 2012 11:44 am

@NickT
The RTL8188CUS indeed shows approx. 50 % packet loss.
I had misinterpreted the results of nuttcp speed tests. While I ran the test on the Pi wireless IP, the Pi returned it's packets over the eth0 connection which was also still connected. Due to that, I always got the same 27 mbit speed. I thought it was an usb bandwidth limit, so never really asked myself questions about the Pi wifi.

nx5
Posts: 6
Joined: Wed Sep 12, 2012 9:16 pm

Re: USB packet loss.

Sat Sep 15, 2012 12:09 pm

obcd wrote:@NickT
The RTL8188CUS indeed shows approx. 50 % packet loss.
I had misinterpreted the results of nuttcp speed tests. While I ran the test on the Pi wireless IP, the Pi returned it's packets over the eth0 connection which was also still connected. Due to that, I always got the same 27 mbit speed. I thought it was an usb bandwidth limit, so never really asked myself questions about the Pi wifi.
I can corroborate I'm seeing the same amount of network packets dropped by an RTL8188CUS (even a bit above 50% in general). It seems to always drops them no matter what, even if you're just sending some keystrokes through ssh, half the packets are consistently dropped.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5097
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: USB packet loss.

Sat Sep 15, 2012 12:13 pm

kyelo wrote:I should also mention that my TP1/TP2 is 5.95. I am not currently over-volting but had overvolt set to 6 earlier (just haven't got around to it on the current card). But I think I read somewhere that if you do set it once, that it triggers a "sticky bit" that lingers...is this correct?
Why is your voltage 5.95V? What are you powering it from? This is outside of spec for USB devices, and may well be causing problems. 5.25V is the upper limit.

kyelo
Posts: 64
Joined: Sun Oct 23, 2011 6:31 pm

Re: USB packet loss.

Sun Sep 16, 2012 12:37 am

Dom said:
Why is your voltage 5.95V? What are you powering it from? This is outside of spec for USB devices, and may well be causing problems. 5.25V is the upper limit.
Motorola cell phone power source, 5ated at 5.1 V, 1000 milliamps. I was measuring with an entry-level analog RadioShack meter.

After above comments I took my R_pi in to a computer sales/service store and had it checked with their instruments which showed 4.91 V! Needless to say, I have already returned the meter I was using and gotten an upgrade.

They computer store personnel had never seen or heard of the Pi but were very impressed with it, and logged onto http://www.raspberrypi.org before I had left the store!

User avatar
yoctopuce
Posts: 21
Joined: Sat Mar 03, 2012 11:27 pm
Contact: Website

Re: USB packet loss.

Fri Sep 21, 2012 7:21 am

We have run some tests today to validate the tentative fixes.

First we have run rpi-update to get the latest fix. Here is my cmdline.txt and the kernel version

Code: Select all

root@raspberrypi:/home/pi# more /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
root@raspberrypi:/home/pi# uname -a
Linux raspberrypi 3.2.27+ #160 PREEMPT Mon Sep 17 23:18:42 BST 2012 armv6l GNU/Linux
As expected, Gordon fix does not completely solve the issue of packet loss in this case, but the packet loss seems to happen far less frequently (I have been able to stress the PI for several minutes before I get a missing packet).

Then we have added the dwc_otg.speed=1 parameter to force USB to work only in full speed mode (speed of USB 1.1). This solved the issue. This confirms that the PI has some trouble to handle USB 1.1 devices behind an USB 2.0 hub, but forcing the USB to work in full speed mode completely solves the packet loss for our USB 1.1 devices.

Our working config:

Code: Select all

root@raspberrypi:/home/pi# more /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 dwc_otg.speed=
1 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
root@raspberrypi:/home/pi# uname -a
Linux raspberrypi 3.2.27+ #160 PREEMPT Mon Sep 17 23:18:42 BST 2012 armv6l GNU/Linux
Last edited by yoctopuce on Fri Sep 21, 2012 7:50 am, edited 1 time in total.
We produce USB controllers, USB sensors as well as embedded USB hubs for DIY projects

milhouse
Posts: 613
Joined: Mon Jan 16, 2012 12:59 pm

Re: USB packet loss.

Fri Sep 21, 2012 7:29 am

yoctopuce wrote: Then we have added the dwc_otg.speed=1 parameter to force USB to work only in full speed mode (speed of USB 1.1).
Just need to ask, if this parameter is added will it slow down ethernet activity?

At the moment I'm only interested in running the Pi as an XBMC media center with infra-red dongle, maybe a keyboard/mouse, and of course ethernet. I do notice missed keystrokes and missed remote control button presses, so I'm wondering if this parameter will be a good solution for people in a similar situation, but only as long as it doesn't knacker the ethernet performance! :)

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5097
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: USB packet loss.

Fri Sep 21, 2012 9:20 am

milhouse wrote:
yoctopuce wrote: Then we have added the dwc_otg.speed=1 parameter to force USB to work only in full speed mode (speed of USB 1.1).
Just need to ask, if this parameter is added will it slow down ethernet activity?
Yes. You can only get up to 12 Mb/s.

milhouse
Posts: 613
Joined: Mon Jan 16, 2012 12:59 pm

Re: USB packet loss.

Fri Sep 21, 2012 9:57 am

dom wrote:
milhouse wrote:
yoctopuce wrote: Then we have added the dwc_otg.speed=1 parameter to force USB to work only in full speed mode (speed of USB 1.1).
Just need to ask, if this parameter is added will it slow down ethernet activity?
Yes. You can only get up to 12 Mb/s.
Bummer, OK thanks. I assume the idea is to fix USB1.1 support so that the workaround isn't necessary...how likely is this, do you think?

thexman
Posts: 259
Joined: Sat Apr 07, 2012 2:18 pm

Re: USB packet loss.

Fri Sep 21, 2012 10:21 am

yoctopuce wrote:We have run some tests today to validate the tentative fixes.

First we have run rpi-update to get the latest fix. Here is my cmdline.txt and the kernel version

Code: Select all

root@raspberrypi:/home/pi# more /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
root@raspberrypi:/home/pi# uname -a
Linux raspberrypi 3.2.27+ #160 PREEMPT Mon Sep 17 23:18:42 BST 2012 armv6l GNU/Linux
As expected, Gordon fix does not completely solve the issue of packet loss in this case, but the packet loss seems to happen far less frequently (I have been able to stress the PI for several minutes before I get a missing packet).

Then we have added the dwc_otg.speed=1 parameter to force USB to work only in full speed mode (speed of USB 1.1). This solved the issue. This confirms that the PI has some trouble to handle USB 1.1 devices behind an USB 2.0 hub, but forcing the USB to work in full speed mode completely solves the packet loss for our USB 1.1 devices.

Our working config:

Code: Select all

root@raspberrypi:/home/pi# more /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 dwc_otg.speed=
1 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
root@raspberrypi:/home/pi# uname -a
Linux raspberrypi 3.2.27+ #160 PREEMPT Mon Sep 17 23:18:42 BST 2012 armv6l GNU/Linux
Good news for us but bad if you want 100mb internet and usb 1.1 it seams now.
one armed controls engineer, my grammar is bad but lets face it most keyboards don't suit a one armed man

obcd
Posts: 908
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Fri Sep 21, 2012 12:48 pm

Brings us back to the question if it would we possible to force only some devices to use usb 1.1 full speed instead of usb 2.0 high speed?
Would an ftdi device and keyboard mouse work without issues when it's connected to an usb 1.1 hub?
We might be able to have high speed wired or wireless internet, and no issues anymore on our devices that don't need the usb 2.0 speed.
Maybe this is 2 much wishfull thinking? Let's see if we can locate such an usb hub on Ebay.

User avatar
jbeale
Posts: 3263
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: USB packet loss.

Fri Sep 21, 2012 4:34 pm

There are lots of USB 1.1 hubs advertised on ebay, and in fact the cheapest of those advertised and labelled as USB 2.0 480 Mbps I discovered are also in fact 1.1 hubs :-).

Image
( ^ Ceci n'est pas un hub USB 2.0 )

ddv2005
Posts: 23
Joined: Fri Jul 20, 2012 2:17 am

Re: USB packet loss.

Fri Sep 21, 2012 4:44 pm

gsh wrote:
kyelo wrote:As you may remember, I am trying to do VOIP using a Logitech H530 USB headset, Twinkle and Raspbian.
Before this new code, I was able (on a lucky day) to get 1-2 sec into the dialing of a call before a reset occurred ("usb 1-1.2.2: reset full-speed USB device number 6 using dwc_otg" in dmesg).
And I am using a powered 4-port hub (the USB headset is plugged into the hub) .
What do you mean a reset occurs? This sounds like a power problem, the Pi doesn't spontaneously reset (reset is either a software triggered thing that happens when you type `reboot` or by the power management browning out...)
I got the same ( reset full-speed USB device number 6 using dwc_otg ) with USB audio device on external MultiTT Hub. I found that is because periodic isochronous SPLIT transaction has failed.
If I connect USB Audio device directly to the board (with Y cable to provide enough power) then no resets but I get clicking sound on playback (on external hub I get clean sound).
Something wrong with SPLIT transactions and it need USB analyzer to find out what is happened on PHY level.

Another problem with USB audio is clock drift due packets lost. I use auddemo application from pjsip project to test clock drift. Without USB traffic I got clock drift about 64-100 samples per second on 16Khz. But with iperf in background it more than 1000 samples per SECOND (>60ms PER SECOND)
https://gist.github.com/3762600
==============================
Done. Result:
Recording result : interval (min/max/avg/dev)=10/19/11/1, burst=2
Playback result : interval (min/max/avg/dev)=8/12/10/0, burst=2
Clock drifts detected. Capture device is running 1298 samples per second faster than the playback device
==============================

Dmitry
Last edited by ddv2005 on Fri Sep 21, 2012 4:52 pm, edited 1 time in total.

MaxK1
Posts: 898
Joined: Sun Aug 26, 2012 11:34 pm

Re: USB packet loss.

Fri Sep 21, 2012 4:48 pm

jbeale wrote:There are lots of USB 1.1 hubs advertised on ebay, and in fact the cheapest of those advertised and labelled as USB 2.0 480 Mbps I discovered are also in fact 1.1 hubs :-).

Image
( ^ Ceci n'est pas un hub USB 2.0 )
I also bought one that looks a lot like that. The wiring inside was also interesting the white wire was
Vcc and the red was labeled "DN". It was sold at Microcenter as USB 2.0, but clearly wasn't. Ooops.
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

User avatar
jbeale
Posts: 3263
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: USB packet loss.

Fri Sep 21, 2012 4:59 pm

ddv2005 wrote:I got the same ( reset full-speed USB device number 6 using dwc_otg ) with USB audio device on external MultiTT Hub. I found that is because periodic isochronous SPLIT transaction has failed.If I connect USB Audio device directly to the board (with Y cable to provide enough power) then no resets but I get clicking sound on playback (on external hub I get clean sound). Something wrong with SPLIT transactions and it need USB analyzer to find out what is happened on PHY level. Dmitry
That agrees with what I've read elsewhere on this forum (problems with SPLIT transactions, and also ISOCHRONOUS endpoints). See for example:
http://www.raspberrypi.org/phpBB3/viewt ... 249#p78797
http://www.raspberrypi.org/phpBB3/viewt ... 75#p141987

ddv2005
Posts: 23
Joined: Fri Jul 20, 2012 2:17 am

Re: USB packet loss.

Fri Sep 21, 2012 5:31 pm

jbeale wrote: That agrees with what I've read elsewhere on this forum (problems with SPLIT transactions, and also ISOCHRONOUS endpoints).
I found that in debug mode I don't have any clock drift. Deep investigation shown that small delay (2-10 usec) at end of dwc_otg_hc_start_transfer fix the problem in non debug mode.
I did patch https://gist.github.com/3762318 that fix the problem (no more than 48 samples drift with iperf on background). To enable it just apply the patch and put dwc_otg.split_delay_dynamic=10 or dwc_otg.split_delay_static=10 to cmdline.txt. The difference between split_delay_dynamic and split_delay_static that split_delay_static always wait specified amount of time but split_delay_dynamic test registers and exit if split transaction finished quickly than specified amount of time.
In worst case this patch take 50ms(5%) per second in delays because every millisecond we have 5 (2 on playback:start+result and 3 on capture:start+continue to get data+continue to get data with NYET result) delays on 10usec. But with dynamic delay I got performance downgrade no more than 1-2%

But I still don't understand why it actually work and only USB analyzer can help. I would be very appreciated if somebody with USB analyzer run the tests and capture USB traffic on PHY level.

To reproduce the problem you need:
1. USB audio device (like VR-Fidelity http://www.ebay.com/itm/Portable-USB-Sp ... 519ca2f0b1)
2. auddemo test application: http://call-o-call.com/auddemo.zip
3. alsa config patch to enable plughw : https://gist.github.com/3762409
4. kernel patch to enable delays https://gist.github.com/3762318

Just run auddemo application and enter "d 10 10" then "t X X 16000 10 1" where X X is number of your USB audio device in list (usually 1 if you have broadcom audio enabled). For more impressive results just run iperf in background. Then enable the patch via "echo 10 >/sys/module/dwc_otg/parameters/split_delay_dynamic" and test again.

Dmitry

obcd
Posts: 908
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Fri Sep 21, 2012 6:42 pm

Brings us back to the question if it would we possible to force only some devices to use usb 1.1 full speed instead of usb 2.0 high speed?
Would an ftdi device and keyboard mouse work without issues when it's connected to an usb 1.1 hub?
We might be able to have high speed wired or wireless internet, and no issues anymore on our devices that don't need the usb 2.0 speed.
Maybe this is 2 much wishfull thinking? Let's see if we can locate such an usb hub on Ebay.
I just managed to answer my own question. It was wishfull thinking. My 4 port ftdi serial device buildin hub was already an usb 1.1 hub. So forcing the devices to run at full speed instead of high speed doesn't improve the way they work. Only dwc_otg.speed=1 seems to do the trick.

ddv2005
Posts: 23
Joined: Fri Jul 20, 2012 2:17 am

Re: USB packet loss.

Fri Sep 21, 2012 6:47 pm

obcd wrote: I just managed to answer my own question. It was wishfull thinking. My 4 port ftdi serial device buildin hub was already an usb 1.1 hub. So forcing the devices to run at full speed instead of high speed doesn't improve the way they work. Only dwc_otg.speed=1 seems to do the trick.
Exactly. It is because without dwc_otg.speed=1 built-in hub still work on high speed and USB controller must use split transactions (that does not work well).

Return to “Troubleshooting”

Who is online

Users browsing this forum: No registered users and 62 guests