USB - the Elephant in our Room

[xtramural]

There is now an sdcard latency fix improvement that should be safer than the "missing_status" one. (Thanks to ddv2005). Can you try adding:
sdhci-bcm2708.enable_llm=1 sdhci-bcm2708.sync_after_dma=0

to cmdline.txt and check if it works okay. It should improve USB/audio latency.

Not sure if this helps but with those changes:

Code: Select all

[email protected]'s password: 
Linux raspberrypi 3.2.27+ #54 PREEMPT Wed Aug 22 13:22:32 BST 2012 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Type 'startx' to launch a graphical session

Last login: Tue Aug 28 18:45:30 2012 from 192.168.0.9
[email protected] ~ $ dmesg
...
[    0.000000] Kernel command line: dma.dmachans=0x3c bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708.boardrev=0x2 bcm2708.serial=0x50ba235d smsc95xx.macaddr=B8:27:EB:BA:23:5D dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait sdhci-bcm2708.enable_llm=1 sdhci-bcm2708.sync_after_dma=0
...
[    3.870329] usb 1-1.2: New USB device found, idVendor=046d, idProduct=c52e
[    3.886587] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.901476] usb 1-1.2: Product: USB Receiver
[    3.913666] usb 1-1.2: Manufacturer: Logitech
[    3.931760] input: Logitech USB Receiver as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2:1.0/input/input0
[    3.949577] generic-usb 0003:046D:C52E.0001: input: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-bcm2708_usb-1.2/input0
[    3.984297] input: Logitech USB Receiver as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2:1.1/input/input1
[    4.004652] generic-usb 0003:046D:C52E.0002: input,hiddev0: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-bcm2708_usb-1.2/input1
[    7.352209] mmcblk0: r/w command failed, status = 0x80000900
[    7.379018] end_request: I/O error, dev mmcblk0, sector 15564536
[    7.393119] Buffer I/O error on device mmcblk0, logical block 1945567
[   10.197136] mmcblk0: r/w command failed, status = 0x80000900
[   10.226880] end_request: I/O error, dev mmcblk0, sector 15564536
[   10.241051] Buffer I/O error on device mmcblk0p2, logical block 1930207
[   20.425686] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   20.876157] ### snd_bcm2835_alsa_probe c05eeaf8 ############### PROBING FOR bcm2835 ALSA device (0):(1) ###############
[   20.895732] Creating card...
[   20.907269] Creating device/chip ..
[   20.920147] Adding controls ..
[   20.931981] Registering card ....
[   20.948763] bcm2835 ALSA CARD CREATED!
[   20.975590] ### BCM2835 ALSA driver init OK ### 
[   25.374994] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   25.391111] mmc0: DMA IRQ 6 ignored - results were reset
[   29.560025] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0x4DE1
[   40.487467] Adding 102396k swap on /var/swap.  Priority:-1 extents:1 across:102396k SS

Wasn't happy to see the mmcblk0 errors/failures so I restored the cmdline.txt to it's default state and rebooted. For comparison:

Code: Select all

Linux raspberrypi 3.2.27+ #54 PREEMPT Wed Aug 22 13:22:32 BST 2012 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Type 'startx' to launch a graphical session

Last login: Tue Aug 28 18:48:28 2012 from 192.168.0.9
[email protected] ~ $ dmesg
...
[    0.000000] Kernel command line: dma.dmachans=0x3c bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708.boardrev=0x2 bcm2708.serial=0x50ba235d smsc95xx.macaddr=B8:27:EB:BA:23:5D dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
...
[    3.645623] usb 1-1.2: New USB device found, idVendor=046d, idProduct=c52e
[    3.660517] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    3.675225] usb 1-1.2: Product: USB Receiver
[    3.688591] usb 1-1.2: Manufacturer: Logitech
[    3.707297] input: Logitech USB Receiver as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2:1.0/input/input0
[    3.725305] generic-usb 0003:046D:C52E.0001: input: USB HID v1.11 Keyboard [Logitech USB Receiver] on usb-bcm2708_usb-1.2/input0
[    3.762888] input: Logitech USB Receiver as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2:1.1/input/input1
[    3.783002] generic-usb 0003:046D:C52E.0002: input,hiddev0: USB HID v1.11 Mouse [Logitech USB Receiver] on usb-bcm2708_usb-1.2/input1
[   20.806876] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   21.262176] ### snd_bcm2835_alsa_probe c05eeaf8 ############### PROBING FOR bcm2835 ALSA device (0):(1) ###############
[   21.281398] Creating card...
[   21.292678] Creating device/chip ..
[   21.305310] Adding controls ..
[   21.316812] Registering card ....
[   21.333405] bcm2835 ALSA CARD CREATED!
[   21.361042] ### BCM2835 ALSA driver init OK ### 
[   25.725011] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   25.740864] mmc0: DMA IRQ 6 ignored - results were reset
[   25.758944] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   25.774938] mmc0: DMA IRQ 6 ignored - results were reset
[   29.057437] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0x4DE1
[   38.293630] Adding 102396k swap on /var/swap.  Priority:-1 extents:1 across:102396k SS
[email protected] ~ $

[dom]

Wasn't happy to see the mmcblk0 errors/failures so I restored the cmdline.txt to it's default state and rebooted. For comparison:

Can you try the two options one at a time to determine which one causes the problem?

[xtramural]

Wasn't happy to see the mmcblk0 errors/failures so I restored the cmdline.txt to it's default state and rebooted. For comparison:

Can you try the two options one at a time to determine which one causes the problem?

Looks like the sdhci-bcm2708.sync_after_dma=0 is the culprit. The errors/failures happened with just that option added:

Code: Select all

[    7.328991] mmcblk0: r/w command failed, status = 0x80000900
[    7.354533] end_request: I/O error, dev mmcblk0, sector 15564536
[    7.368850] Buffer I/O error on device mmcblk0, logical block 1945567
[   10.846507] mmcblk0: r/w command failed, status = 0x80000900
[   10.870303] end_request: I/O error, dev mmcblk0, sector 122864
[   10.884504] Buffer I/O error on device mmcblk0p1, logical block 14334

[dom]

Looks like the sdhci-bcm2708.sync_after_dma=0 is the culprit. The errors/failures happened with just that option added:

Thanks for testing.

[xtramural]

Looks like the sdhci-bcm2708.sync_after_dma=0 is the culprit. The errors/failures happened with just that option added:

Thanks for testing.

No problem. Happy to help. I meant to mention that without the sdhci-bcm2708.sync_after_dma=0 option I get my usual dose of 'missed completions' but this seems the lesser of two evils:

Code: Select all

[   25.866576] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   25.882327] mmc0: DMA IRQ 6 ignored - results were reset
[   25.931813] mmc0: missed completion of cmd 18 DMA (512/512 [1]/[1]) - ignoring it
[   25.947740] mmc0: DMA IRQ 6 ignored - results were reset

[adambialas]

After i update rpi-update to :
Linux version 3.2.27+ ([email protected]) (gcc version 4.7.1 20120402 (prerelease) (crosstool-NG 1.15.2) ) #84 PREEMPT Tue Aug 28 18:11:56 BST 2012
my USB microphone stop recoding audio

[obcd]

What was your previous revision of your os on the Pi? How did you update to that version? It doesn't seem an official one? Some settings became default in the latest kernels, but they can be turned off with settings in the cmdline.txt file. Is your usb recording device still recognised? Maybe you have a kernel now without the driver for it.

[adambialas]

Is recognised ok , but after i use arecord i have only file 44 bytes long. Before this update snd_usb_audio was in the blacklist and i modprobe in rl.local , and audio recording was ok.

[zog]

I had another late night on monday testing the raspberry pi with the new kernel. The new usb drivers seem to be much better, but there are still problems with blue tooth. Sometimes I get hci0 tx time out errors if I have the blue tooth dongle plugged into a powered hub ( this is where a radio powered device belongs ) at boot. The system is more stable with the blue tooth radio plugged into the rpi. I can now communicate with my Samba3G modem using putty, I have not tried setting up a 3G connection yet.
I edited the config.txt file using pico, but on rebooting the config file had become truncated and corrupted, this happened more than once. I have reformatted my SD card and I will start from scratch again!
I can report that mouse, keyboard and blue tooth radio can all exist on the same hub without stopping the other one functioning although they still seem to be interfering with each other.

My hunch is that the cpu is overloaded, a bit like me it has to much to do and to little time to do it, so occasionally it drops the ball on the usb side.

Some some easy things can be done to reduce the load on the cpu side.
1) These are using a lower screen resolution as the pi is using the cpu to draw the desktop.
2) Move all drawing operations over to the GPU which will require a sustained programming effort by someone.
3) Throttle back the speed on the eth0 if possible to give more processor time to the cpu to indirectly improve the latency times on the usb side.
I tried over clocking my pi to 806Mhz, which did seem to increase the reliability of the pi and this definitely improved the blue tooth. I have read Yocotopus posts on the usb issues and he seemed to think the problem was within the BCM2835 chip and could be due to stray capacitance on the board somewhere etc. It therefore makes sense to adjust the clock frequencies to try to adjust the pi away from any problem resonant frequencies.
arm_freq=806Mhz
So my plan now is to modify my pi into a rpi extreme edition rpi-xd, by adding passive heat sinks to the chips and by seeing how fast I can make the pi go and seeing if it improves the overall situation. I know a lot of people think heat sinks are unnecessary, but I will feel happier sticking them on the pi before any extreme over clocking / over voltage experiments are carried out.
I could do with a thermal imaging camera to see which parts are getting over heated, but I am not sure where I can get or borrow one of those from.

I have also been wondering is it possible to adjust the pi’s clock programmatically to put it into low and high performance modes ?

[lintweaker]

The latest firmware+kernel and using sdhci-bcm2708.enable_llm=1 did improve audio playback using usb-audio . It's not flawless, there are still occasional dropouts/crackles which increase with the bitrate being played back.

So very close now! Keep it up!

BTW is there a complete list of all the sdhci parameters the number of options is quite bewildering?

[rspitz]

I got one or two of these eth0 error messages when I was using eth0 but I didn't notice any problems (although I wasn't transferring data heavily). As far as networking is concerned, the issue isn't that bad, after all, communications protocols will make sure the data gets transferred.

So you say that the eth0 error messages (Failed to read/Failed to write register index...) can be safely ignored? Well, that makes me feel a little better, but they still clutter up kern.log and dmesg, and I cannot really link them to any network activity, like file transfers started via cron.

[Roger.Morgan]

Just got my RPI and hooked it up. HDMI=>DVI cable, 1000ma ps, old hp usb keyboard, old ibm usb wheel mouse. Everything worked but the mouse. Tried Logitech mouse. No joy. HDMI output is fantastic. No issue with keyboard. Having read several forums I'm guessing this is the usb power issue. I'll try a usb hub next and I think somewhere in one of my old travel bags is a micro mouse that might have a lower power requirement. I did notice during the text boot the raspbian image seemed to recognize the logitech mouse but after the startx the mouse did not seem to work. I provide my experience purely so those who think since it didn't happen to them it must be a stupid user issue. ( Which doesn't mean I can't be a stupid user. I have LOTS of years of experience at that! )

[lazx888]

sdhci-bcm2708.enable_llm=1 fixed frequent disconnects of my Logitech Dinovo Edge keyboard/mouse.

Thanks dom/popcornmix/ddv2005 (https://github.com/raspberrypi/linux/co ... 5b8c8ef5c9)

[dom]

sdhci-bcm2708.enable_llm=1 fixed frequent disconnects of my Logitech Dinovo Edge keyboard/mouse.

Thanks dom/popcornmix/ddv2005 (https://github.com/raspberrypi/linux/co ... 5b8c8ef5c9)

Glad it helped. I've now flipped the switch so "low latency mode" is enabled by default.
I helped an issue with sticky keys for me when using keyboard/mouse/hub/wifi combination.

[dom]

FIQ patch is pushed. Get it from rpi-update. It is disabled by default, but add:
dwc_otg.fiq_fix_enable=1
to cmdline.txt to test. It should give about 10% extra performance to ARM when USB is not busy. It shouldn't change behaviour of USB.

[pygmy_giant]

So as a recap, am I correct in understanding that current advice is to:

1) Try a powered hub

2) add:

dwc_otg.microframe_schedule=1 sdhci-bcm2708.enable_llm=1 sdhci-bcm2708.sync_after_dma=0 sdhci-bcm2708.sync_after_dma=0 dwc_otg.fiq_fix_enable=1

to the end of /boot/cmdline.txt (with no newlines)

3) do

sudo rpi-update

to update the drivers (requires rpi-update from https://github.com/Hexxeh/rpi-update )

and that entering:

uname -a

should indicate that you have the latest version by returning "Linux raspi 3.2.27+ #66 PREEMPT Fri Aug 24 23:52:42 BST 2012 armv6l GNU/Linux" (at the time of writing)

?

The positive feed back from these remedies seem to indicate that were moving from:



towards:

[asb]

FIQ patch is pushed. Get it from rpi-update. It is disabled by default, but add:
dwc_otg.fiq_fix_enable=1
to cmdline.txt to test. It should give about 10% extra performance to ARM when USB is not busy. It shouldn't change behaviour of USB.

Or just via apt http://www.raspberrypi.org/phpBB3/viewt ... 0&p=164595

[bredman]

It looks like we no longer have to worry about the network connection failing when the startx command is entered...
http://www.raspberrypi.org/phpBB3/viewt ... 60#p163824

Well done to all on the good work.

[MattOwnby]

UPDATE: tried the "llm" command line mode and it _may_ have helped. I will know more later.

I've been working on an application that combines continous JPEG decoding using OpenMAX, continuous rendering using GLES2, and continuous serial I/O (mostly input) via FTDI USB chip. After several days of trying to improve performance, I have determined that my performance woes are due to OpenMAX and FTDI/USB running at the same time.

If I disable the serial I/O completely, I can decode images at the speed I want with "htop" reporting about 60% overall CPU usage on the system and my main thread using about 35%.

If I run another process that does nothing but calls read(fd, ...) on /dev/ttyUSB0 (ie read from serial input as fast as possible), my main JPEG decoding process jumps over 50% and the "htop" reports CPU usage as pegged as 90+ % or more and I am no longer able to decode JPEGs at full speed. Further investigation using "oprofile" shows that the OpenMAX calls block a lot longer when the serial/USB is active than they do otherwise.

So I've had two ideas of how I might improve performance here.

- Stop using FTDI/USB and try to hook into the serial port on the GPIO pins. The device on the other end of the serial connection is an AVR microcontroller which has voltages from 0-5V on its RX/TX pins so I would need some kind of hardware to do the voltage conversion in between. If I go to all this trouble, how likely is it that my performance problems will disappear? Is there something I can test in the software first (such as writing to the serial /dev entry) to see if performance is impacted?

- Build my own kernel so that I have a working vmlinux file and then use oprofile to see if there is some functions inside the kernel that are clearly dragging everything else down.

both of these options sound like a lot of work. Which one would give me the most bang for the buck?

And.. any other ideas?

Thanks!
--Matt

PS - I absolutely need the serial I/O because as I said I am talking to an AVR microcontroller on the other end.

[dom]

gsh is looking into the FTDI issue. It seems the device generates a continuous stream of NAKs which causes a very high interrupt rate (about 44k/s). He's investigating if we can delay dealing with the NAKs to throttle the rate so there may be a solution.

I think using the dedicated UART pins will definitely resolve the performance issue.

[mahjongg]

So I've had two ideas of how I might improve performance here.

- Stop using FTDI/USB and try to hook into the serial port on the GPIO pins. The device on the other end of the serial connection is an AVR microcontroller which has voltages from 0-5V on its RX/TX pins so I would need some kind of hardware to do the voltage conversion in between. If I go to all this trouble, how likely is it that my performance problems will disappear? Is there something I can test in the software first (such as writing to the serial /dev entry) to see if performance is impacted?

PS - I absolutely need the serial I/O because as I said I am talking to an AVR microcontroller on the other end.

If you are using a normal AVR microcontroller (i.e. an ATmega series microcontroller), firstly these can work with 3V3 too, but if you are forced to run them at 5V, know that for the UART signal from the AVR no conversion logic is needed, as the AVR understands signals above 3.0 volt already as high (0.6 x 5V). Concerning converting the 5V level to useable 3V3 levels, all that is needed are two resistors, to divide the 5V signal by two, (simplest solution) or 3/5th (two 4K7 resistors, or one 2K2, and one 3K3 resistor).

[samsamsam]

I also use 2xFTDI devices and I do not see this problems:

Code: Select all

[email protected]:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0  82088  86200  43120    0    0     0     0  132  195  2  3 95  0
 2  0      0  81920  86200  43120    0    0     0     0 3122 1255 18 36 46  0
 0  0      0  81968  86200  43120    0    0     0     0 3215  802 10 24 66  0
 0  0      0  81968  86200  43120    0    0     0     0 3138  392  3  2 95  0
 0  0      0  81968  86200  43120    0    0     0     0 3093  146  1  1 98  0
 0  0      0  81968  86200  43120    0    0     0     0 3101  167  0  3 97  0
 0  0      0  81968  86200  43120    0    0     0     0 3094  155  2  2 96  0
 0  0      0  81968  86200  43120    0    0     0     0 3156  293  1  3 96  0
 0  0      0  81968  86200  43120    0    0     0     0 3076  109  0  0 100  0
 0  0      0  81968  86200  43120    0    0     0     0 3088  103  1  2 97  0
 0  0      0  81968  86200  43120    0    0     0     0 3099  157  1  0 99  0
 0  0      0  81968  86200  43120    0    0     0     0 3118  225  2  3 95  0
 0  0      0  81968  86200  43120    0    0     0     0 3070  107  1  1 98  0
 1  0      0  81944  86200  43120    0    0     0     0 3185 1190 15 35 50  0
 0  0      0  82000  86200  43120    0    0     0     0 3118  732  5 27 67  0
 0  0      0  82000  86200  43120    0    0     0     0 3112  205  1  2 97  0
 0  0      0  82000  86200  43120    0    0     0     0 3106  250  1  1 98  0

[MattOwnby]

If you are using a normal AVR microcontroller (i.e. an ATmega series microcontroller), firstly these can work with 3V3 too, but if you are forced to run them at 5V, know that for the UART signal from the AVR no conversion logic is needed, as the AVR understands signals above 3.0 volt already as high (0.6 x 5V). Concerning converting the 5V level to useable 3V3 levels, all that is needed are two resistors, to divide the 5V signal by two, (simplest solution) or 3/5th (two 4K7 resistors, or one 2K2, and one 3K3 resistor).

Do you mean that the TX signal from the pi to the AVR's RX does not need to be increased from 3.3V to 5V?

(but that AVR's TX to the pi's RX would need to be decreased from 5V to 3.3V?)

[rurwin]

- Stop using FTDI/USB and try to hook into the serial port on the GPIO pins. The device on the other end of the serial connection is an AVR microcontroller which has voltages from 0-5V on its RX/TX pins so I would need some kind of hardware to do the voltage conversion in between.

The RaspPi can output sufficient voltage for the AVR to detect a correct signal; the threshold (V_IH) of a Mega16 for example is less than 1V. Anything over that will be detected as a high signal. The 5V output of the AVR needs to be chopped to 3V, but the AVR can output plenty of current, so a simple voltage divider would work fine. You need ten times the current through the divider as you are taking out of it. The AVR will supply up to 20mA, so say 300 and 200 ohm resistors, giving a current of 10mA. You could probably make that 3K and 2K and it would still work. It is so simple you don't even need a PCB, just make it part of the wiring. You don't need to be exact there either. I don't know what the threshold of the RaspPi is, but so long as a high voltage is under 3.3V, but not extravagantly under, it will work fine. I'd probably try it with 2V just to be safe and increase it to 3V (so turn the divider upside-down) if it didn't work.

[mahjongg]

If you are using a normal AVR microcontroller (i.e. an ATmega series microcontroller), firstly these can work with 3V3 too, but if you are forced to run them at 5V, know that for the UART signal from the AVR no conversion logic is needed, as the AVR understands signals above 3.0 volt already as high (0.6 x 5V). Concerning converting the 5V level to useable 3V3 levels, all that is needed are two resistors, to divide the 5V signal by two, (simplest solution) or 3/5th (two 4K7 resistors, or one 2K2, and one 3K3 resistor).

Do you mean that the TX signal from the pi to the AVR's RX does not need to be increased from 3.3V to 5V?

(but that AVR's TX to the pi's RX would need to be decreased from 5V to 3.3V?)

exactly! also like Ruwin said. No need to lower the 5V power of an AVR RISC chip, but if you want to no problem either!


interfacing an "arduino" style microcontroller to the PI is easy, and the Gertboard (for example) demonstrates it.

A simple tip, if the AVR chip has trouble with the 3V3 level of the PI (too much noise for example), then simply add a 10K pullup to the signal, this will lift up the signal to the AVR a fraction of a volt, enough to make the signal more stable without any negative effect to the PI. Normally completely unnecessary though.