samsamsam
Posts: 36
Joined: Fri Aug 17, 2012 11:36 am

Re: USB packet loss.

Wed Sep 12, 2012 9:29 pm

obcd wrote:Thanks for clearing that up.
Another thing I was wondering. Is it possible to force an usb device to enumerate at usb 1.1 instead of usb 2.0 (I have trouble with the high speed and full speed terms (which is which..))I was thinking about ftdi and prolific adapters. They don't need the usb 2.0 speed and might behave better at usb 1.1 speed?
You can force the whole bus to become usb 1.1, but that isn't very interesting for the other devices that could benefit from the usb 2.0 speed. It might help to figure out if they indeed behave better.
I know you did not ask me, but YES it is possible
add dwc_otg.speed=1 to cmdline.txt
And Yes with this option USB hubs works superb (at least for me) :)

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Wed Sep 12, 2012 10:03 pm

I read about the dwc_otg.speed=1 somewhere but I am glad I didn't had to search it up again.
I was hoping I could force a specific client into usb 1.1 mode. Like only the prolific adapter.
Doing so, I could leave my network adapter at usb 2.0.
Will check out tomorrow how well things work in usb 1.1 mode.
Thanks.

thexman
Posts: 259
Joined: Sat Apr 07, 2012 2:18 pm

Re: USB packet loss.

Thu Sep 13, 2012 6:51 pm

gsh wrote:Just to make it clear , the USB packet loss (with USB1.0 and USB1.1 devices) does NOT occur with the model A...

I ran a test before leaving work today running Yocupuce's test and it is still running now (it normally failed after a few seconds on the original stack)

Gordon


Omg where can I buy 200 model a units please need these last month. Now ?
one armed controls engineer, my grammar is bad but lets face it most keyboards don't suit a one armed man

itimpi
Posts: 1090
Joined: Sun Sep 25, 2011 11:44 am
Location: Potters Bar, United Kingdom
Contact: Website

Re: USB packet loss.

Thu Sep 13, 2012 6:55 pm

thexman wrote:
gsh wrote:Just to make it clear , the USB packet loss (with USB1.0 and USB1.1 devices) does NOT occur with the model A...

I ran a test before leaving work today running Yocupuce's test and it is still running now (it normally failed after a few seconds on the original stack)

Gordon


Omg where can I buy 200 model a units please need these last month. Now ?
Model A is not currently available for sale. The only date we have been given for availability is "before the end of the year".

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1445
Joined: Sat Sep 10, 2011 11:43 am

Re: USB packet loss.

Thu Sep 13, 2012 7:32 pm

Yep,

To get a model A what you need is to work with Eben he keeps dropping stuff like this on my desk everyday! You'd be amazed at the junk I collect! (I have four of the little micro Pi's that Eben showed when they first launched the foundation for example, complete with 5Mpix cameras)


So today I think I found a significant step in the right direction...

I was working on the USB problem with the FTDI serial devices, the problem here is the FTDI device just sits for long periods not responding to the host and then after a millisecond it replies with two bytes! Basically it looks like there's just a prehistoric micro-controller in there! Problem is during this time the device is NAKing everything from the host and the host is getting up to 3 interrupts per NAK (and we're getting four NAKs per uframe). This means that the Pi is getting killed with interrupts and probably explains all the strange problems with booting and slow performance when using these devices.

So the 'workaround' I was working on was to reduce the interrupt overhead by holding off retransmitting the IN packets that were getting NAK'ed. After doing this and testing that it reduced the interrupts to a more manageable level (11000 per second, still high but doesn't kill the Pi) I also noticed that I could no longer reproduce the dropping packet problem.

In the end to get the old packet loss problem I had to use Yoctopuce's demo application changed to run for 10 minutes then I had to plug in two hubs, a keyboard, FTDI serial cable, then plug in two ethernet adapters and then flood ping it from my PC...

Finally the problem re-occurred!!!

I think there may still be some minor things to work around, but I think this is likely to be a fairly solid release of the USB driver, I still haven't been able to test it with a very large selection of USB devices but it does look like we're getting somewhere!

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5370
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: USB packet loss.

Thu Sep 13, 2012 8:59 pm

Gordon's update is in the latest rpi-udpate kernel.

The fiq fix is now enabled by default, as is Gordon's new patch (you can disable it with dwc_otg.nak_holdoff_enable=0 if you suspect it of causing problems).
Let us know is anything new is broken, or anything is improved.

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Thu Sep 13, 2012 9:12 pm

Great to hear Gordon & Dom.
Today, I forced my Pi to full speed with the dwc_otg.speed=1 command.
I connected a Lindy 4 port usb2serial device. This is basically a 4 port usb hub and 4 ftdi chips connected to that usb hub. It's normal behavour was that it makes the Pi freeze as soon as you try to access one of the serial ports.
To my suprise, with the bus switched to full speed, I could query the device and use it to send data without issues. The interrupt rate was approx. 1000/sec. This makes sense if you consider a 1ms frame rate. I know ftdi has a program to adjust the settings of the devices, but I can't remember if it would be possible to force them full speed only. Reducing the whole system to full speed lowers the network troughput from 28 mbit/sec to 7mbit/sec.
I doubt this information is of any use, but decided to post it anyhow for completeness when people search these posts. I don't think you need high speed for ftdi adapters that use normal serial speeds like 9600 baud. Besides, those ftdi adapters have some RX and TX buffers.
I will try the new kernel tomorrow with the Lindy device and post my results.

SiriusHardware
Posts: 502
Joined: Thu Aug 02, 2012 9:09 pm
Location: UK

Re: USB packet loss.

Thu Sep 13, 2012 9:15 pm

dom wrote:Gordon's update is in the latest rpi-udpate kernel.

The fiq fix is now enabled by default, as is Gordon's new patch (you can disable it with dwc_otg.nak_holdoff_enable=0 if you suspect it of causing problems).
Let us know is anything new is broken, or anything is improved.
newbie here .. can you just clarify that this will eventually make it into the 'stable' release accessed by apt-get update? Or will I never get this fix at all unless I use Rpi-update at some point?

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5370
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: USB packet loss.

Thu Sep 13, 2012 9:19 pm

SiriusHardware wrote:newbie here .. can you just clarify that this will eventually make it into the 'stable' release accessed by apt-get update? Or will I never get this fix at all unless I use Rpi-update at some point?
Sure. Assuming the testing is positive it will make it into the next apt-get upgrade.

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Thu Sep 13, 2012 9:21 pm

At the moment, you will only get it with the rpi-update.
If all reports of this kernel are positive, and it's running without issues, they might push it out as latest stable, so you will get it with apt-get upgrade and apt-get update.
It might also become the version you can download for initial install on your sd card.
Looks like Dom ninja'd me... He's in a better position to know than I am.

User avatar
PeterO
Posts: 5142
Joined: Sun Jul 22, 2012 4:14 pm

Re: USB packet loss.

Thu Sep 13, 2012 9:28 pm

dom wrote:Gordon's update is in the latest rpi-udpate kernel.
Can I request that rather than this "latest kernel" statement you actually say which release it is inculded in. Otherwise someone coming to this thread at some point in the future has no way of knowing if they have this in their kernel or not.

PeterO.
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5370
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: USB packet loss.

Thu Sep 13, 2012 9:34 pm

PeterO wrote:Can I request that rather than this "latest kernel" statement you actually say which release it is inculded in. Otherwise someone coming to this thread at some point in the future has no way of knowing if they have this in their kernel or not.
My message is timestamped, so all information is present.
But, if it helps, e2466799a989f0b3bbc669f9e2c755280d667dc0 from github.com/Hexxeh/rpi-firmware has this update.

User avatar
PeterO
Posts: 5142
Joined: Sun Jul 22, 2012 4:14 pm

Re: USB packet loss.

Thu Sep 13, 2012 9:41 pm

dom wrote: My message is timestamped, so all information is present.
But, if it helps, e2466799a989f0b3bbc669f9e2c755280d667dc0 from github.com/Hexxeh/rpi-firmware has this update.
How does that relate to "Linux raspberrypi 3.2.27+ #143" ?
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

Aydan
Posts: 702
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: USB packet loss.

Thu Sep 13, 2012 9:45 pm

This update (#148) didn't solve my "staggered hub mass storage problem"
see here: http://www.raspberrypi.org/phpBB3/viewt ... 28&t=10355
So I suppose it's not related to dropped packets.
Here's some info:
USB key plugged into the 2nd level hub:

Code: Select all

[email protected]:~# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   306 MB in  2.00 seconds = 152.83 MB/sec
 Timing buffered disk reads:  16 MB in 31.71 seconds = 516.76 kB/sec
[email protected]:~# uname -a
Linux raspberrypi 3.2.27+ #148 PREEMPT Thu Sep 13 21:36:23 BST 2012 armv6l GNU/Linux
[email protected]:~# lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/3p, 480M
        |__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
        |__ Port 2: Dev 4, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 6, If 0, Class=HID, Driver=usbhid, 1.5M
            |__ Port 4: Dev 6, If 1, Class=HID, Driver=usbhid, 1.5M
        |__ Port 3: Dev 5, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 7, If 0, Class=hub, Driver=hub/4p, 480M
                |__ Port 3: Dev 8, If 0, Class=stor., Driver=usb-storage, 480M
USB key plugged into the 1st level hub:

Code: Select all

[email protected]:~# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   306 MB in  2.00 seconds = 152.93 MB/sec
 Timing buffered disk reads:  76 MB in  3.03 seconds =  25.09 MB/sec
[email protected]:~# lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/3p, 480M
        |__ Port 1: Dev 3, If 0, Class=vend., Driver=smsc95xx, 480M
        |__ Port 2: Dev 4, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 6, If 0, Class=HID, Driver=usbhid, 1.5M
            |__ Port 4: Dev 6, If 1, Class=HID, Driver=usbhid, 1.5M
        |__ Port 3: Dev 5, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 7, If 0, Class=hub, Driver=hub/4p, 480M
            |__ Port 3: Dev 9, If 0, Class=stor., Driver=usb-storage, 480M
Dev 4 is my KVM switch
Last edited by Aydan on Thu Sep 13, 2012 9:47 pm, edited 1 time in total.

User avatar
NickT
Posts: 271
Joined: Mon May 21, 2012 10:43 am
Location: UK

Re: USB packet loss.

Thu Sep 13, 2012 9:46 pm

I'm not sure if preempt 148 version (which is what I got when I rpi-updated just now) is supposed to fix the dropped packets when using the built in RTL8188CUS drivers, but it appears that it hasn't.

I posted some results that I had with preempt 143 yesterday on this page
http://www.raspberrypi.org/phpBB3/viewt ... &start=900.

The figures are pretty much the same with the latest 148 version.

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Thu Sep 13, 2012 9:47 pm

[email protected] ~ $ uname -a
Linux raspberrypi 3.2.27+ #148 PREEMPT Thu Sep 13 21:36:23 BST 2012 armv6l GNU/Linux
This is what the latest rpi-update brought me.

User avatar
PeterO
Posts: 5142
Joined: Sun Jul 22, 2012 4:14 pm

Re: USB packet loss.

Thu Sep 13, 2012 9:55 pm

obcd wrote:
[email protected] ~ $ uname -a
Linux raspberrypi 3.2.27+ #148 PREEMPT Thu Sep 13 21:36:23 BST 2012 armv6l GNU/Linux
This is what the latest rpi-update brought me.
That looks like it was built ~23mins before Doms timestamped message, so how do we tell if it is the kernel referred to by Dom as "the latest" or not ?
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Thu Sep 13, 2012 9:59 pm

The dropped packets you see in ifconfig are ethernet packets, not usb ones.
It could be due to them having a bad crc.
Maybe one wifi adapter is more sensitive than the other.
What's the distance between the adapter and your router? Some adapters even don't like it if they are 2 close to the router.
I am not saying it can't be related to the usb packet loss.
I think I also have an adapter with that chipset at work. I might test and see if I also notice packet loss. Previous tests I did showed the same network speed between the wired and the wireless connections. I assume a 50% packet loss would have halved the speed.

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Thu Sep 13, 2012 10:07 pm

We could look at github if anything else was changed in the kernel today. If that didn't happen, it's very likely they build a new kernel today to add gordon's patch.
e2466799a989f0b3bbc669f9e2c755280d667dc0 from github.com/Hexxeh/rpi-firmware has this update.
doesn't ring a bell to me either. Maybe it's a hash or something, and you can compare it with the one of your kernel. I really haven't a clue.

User avatar
jbeale
Posts: 3517
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: USB packet loss.

Thu Sep 13, 2012 10:24 pm

obcd wrote:Is it possible to force an usb device to enumerate at usb 1.1 instead of usb 2.0
I'm no expert, but isn't that what happens (effectively) if you plug it into a USB 1.1 hub? Although that does introduce another variable into the equation. USB 1.1 hubs are sold on ebay for about USD 1.18 shipped.

Also, the long hex number is surely a github SHA code. Github labels commits with a SHA-1 checksum or hash value of the code, which is long enough so that each commit across every project on the whole site has a pretty-much-guaranteed-unique number. See for example the hex numbers on the right hand side of this page: https://github.com/raspberrypi/linux/commits/rpi-3.2.27

See also: http://git-scm.com/book/en/Git-Internals-Git-Objects

thexman
Posts: 259
Joined: Sat Apr 07, 2012 2:18 pm

Re: USB packet loss.

Thu Sep 13, 2012 10:37 pm

I'll boot up my 26 USB devices and run the scripts and see if it all falls over its the best test you can get its 60 odd io with a64 k script asking for 30 inputs and switching 30 out put channels over the USB twin hubs each 13 channels with no back fed power lets keep fingers crossed


Thanks
one armed controls engineer, my grammar is bad but lets face it most keyboards don't suit a one armed man

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: USB packet loss.

Thu Sep 13, 2012 10:43 pm

Interesting. I didn't realise they still sold usb 1.1 hubs. To my knowledge, a device tries to enumerate high speed first, and when that fails it tries full speed. So an usb 1.1 hub would indeed force that behavour.

@thexman
would full speed be an option for you, or do you need fast ethernet for your setup?
The model A wasn't having packet loss, but that's with only 1 usb device connected. (no hub)
You will need 26 of them. Maybe Eben can make a special price for you.

thexman
Posts: 259
Joined: Sat Apr 07, 2012 2:18 pm

Re: USB packet loss.

Thu Sep 13, 2012 10:56 pm

Ahu well the ultimate was to run 100 devices across 8 hubs each hub being 13 devices of various io give upto 500 points of io

Meaning a total control system operating at 1 to 10 seconds I don't need speed I just need the ability to connect everything to one controller and one script one the system is stable the Pi is a perfect partner and the LAN connection is a bonus meaning I can connect multiple devices with 500 io points doing what ever I need it too.
one armed controls engineer, my grammar is bad but lets face it most keyboards don't suit a one armed man

kyelo
Posts: 64
Joined: Sun Oct 23, 2011 6:31 pm

Re: USB packet loss.

Fri Sep 14, 2012 12:42 am

As you may remember, I am trying to do VOIP using a Logitech H530 USB headset, Twinkle and Raspbian.

I just now did the rpi-update and have 3.2.27+ #148 PREEMPT, with the fiq_fix_enable and naq_holdoff_enable enabled by default. I am also using microframe_schedule=1, enable_llm=1, and sync_after_dma=0.

Before this new code, I was able (on a lucky day) to get 1-2 sec into the dialing of a call before a reset occurred ("usb 1-1.2.2: reset full-speed USB device number 6 using dwc_otg" in dmesg).

Now I MAY be able to get 3 seconds into dialing a call or an echo test before the reset occurs. I am still (after the update) getting about 2500 interrups/sec (down from 8500 interrupts /sec baseline).

And I am using a powered 4-port hub (the USB headset is plugged into the hub) .

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1445
Joined: Sat Sep 10, 2011 11:43 am

Re: USB packet loss.

Fri Sep 14, 2012 8:11 am

NickT wrote:I'm not sure if preempt 148 version (which is what I got when I rpi-updated just now) is supposed to fix the dropped packets when using the built in RTL8188CUS drivers, but it appears that it hasn't.

I posted some results that I had with preempt 143 yesterday on this page
http://www.raspberrypi.org/phpBB3/viewt ... &start=900.

The figures are pretty much the same with the latest 148 version.
I doubt the fix I made would make a difference here, the 8188 is a USB2.0 device and therefore the NAKing is handled in the hardware, this shouldn't give high interrupt numbers... I was going to look into this next though to find out what's going on there!

Thanks

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

Return to “Troubleshooting”