M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: USB/serial converters very broken on Pi

Thu Nov 22, 2012 1:30 pm

markushx wrote:That patch (adapted and compiled for 3.6.1+) works for my device with an FT232BM. Thanks!
Really?

If it did really fix it, you would either get a stack dump or messages for data toggle errors in dmesg. Did either of these happen?

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Sun Nov 25, 2012 2:35 pm

I got the latest kernel source and added the patch code. I compiled the kernel using the RPI to do the compile.
Linux raspberrypi 3.2.27+ #2 PREEMPT Fri Nov 23 21:19:50 UTC 2012 armv6l GNU/Linux

I did NOT do a rpi-update as I wasn't sure if that would undo the patch. (should I have done the update?)

My program ran for about 24 hours (1432 xfers of 50kbytes) before any issues started to show up.
NOTE: The system DID NOT lock up.
I'm using a simple perl wrapper code that calls my program. The wrapper code starts displaying
random and non-ASCII characters.

#!/usr/bin/perl
$cnt=0;
while(1) {
sleep(2);
$filename = sprintf("data_%5.5d", $cnt);
system("./sr DATA/${file_name}");
printf("Captured data # %s\n", $filename);
$cnt++;
}


Once this started happening I dmesg for the DATA Toggle message from the patch and sure enough
the message got logged.

raspberrypi kernel: [84808.912275] ERROR::handle_hc_datatglerr_intr:1950: Data Toggle Error - on endpoint type BULK

I'll do a rpi-update and see how that goes

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Sun Nov 25, 2012 6:44 pm

rpi-update stated that everything is up to date.

nurquhar
Posts: 16
Joined: Thu May 10, 2012 7:11 pm
Contact: Website

Re: USB/serial converters very broken on Pi

Tue Nov 27, 2012 9:52 am

I started this thread. See http://www.raspberrypi.org/phpBB3/viewt ... 94#p220994

Basicaly I am "simple" user trying to use an FTDI based USB Serial converter which crashes the RPi on first access to the /dev/ttyUSB0 stream. I think this is what this thread is all about.

I have tried an rpi-update a couple of days ago to fix but it made no difference.

I did find that a PL2303 USB converter does work though.

I don't know how to do kernal patches etc so I don't think I can try any of the attempted fixes that appear above.

If/when the "clever" people here figured out what the problem and fix is there going to be away for simpletons like me to fix our RPi's ? Perhaps I will be able to do an rpi-update in week or two to fix my RPi :?:

Also I did try the D2XX drivers from the FTDI site. Followed the reame instructions and built all the examples. However they seem to work either. This is what I got :

Code: Select all

[email protected]:~/nau/ftdi/release/examples/EEPROM/read$ sudo ./read
./read: error while loading shared libraries: libftd2xx.so: cannot open shared object file: No such file or directory
[email protected]:~/nau/ftdi/release/examples/EEPROM/read$ ls -l /usr/local/lib/libftd*
-rw-r--r-- 1 root staff 241478 Nov 26 14:44 /usr/local/lib/libftd2xx.a
lrwxrwxrwx 1 root staff     34 Nov 26 14:46 /usr/local/lib/libftd2xx.so -> /usr/local/lib/libftd2xx.so.1.1.12
-rwxr-xr-x 1 root staff 229597 Nov 26 14:44 /usr/local/lib/libftd2xx.so.1.1.12
[email protected]:~/nau/ftdi/release/examples/EEPROM/read$]

caldimerda
Posts: 68
Joined: Tue Oct 09, 2012 9:41 am

Re: USB/serial converters very broken on Pi

Tue Nov 27, 2012 6:20 pm

There is a lot of helpful stuff on the forum relevant to this but it isn't very easy to find. This worked for me:

apt-get update
apt-get upgrade
wget http://goo.gl/1BOfJ -O /usr/bin/rpi-update && sudo chmod +x /usr/bin/rpi-update
apt-get install git-core
rpi-update
raspi-config (and update from the menu)

Plus adding dwc_otg.speed=1 to /boot/cmdline.txt which sets the USB to full speed instead of high speed and slows a bunch of things down - none of which really matters to me.

I had exactly the same symptom with two expensive usb--->serial cables. On the other hand the cheap cable worked without all these changes. Now all three cables behave the same - they work aside from CRC errors when reading a DS2438 on a 1-wire bus. I think someone has a fix for this but I don't think it's been incorporated on github.

Good luck.

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: USB/serial converters very broken on Pi

Tue Nov 27, 2012 8:40 pm

tedh wrote:I got the latest kernel source and added the patch code. I compiled the kernel using the RPI to do the compile.
Linux raspberrypi 3.2.27+ #2 PREEMPT Fri Nov 23 21:19:50 UTC 2012 armv6l GNU/Linux

I did NOT do a rpi-update as I wasn't sure if that would undo the patch. (should I have done the update?)

My program ran for about 24 hours (1432 xfers of 50kbytes) before any issues started to show up.
NOTE: The system DID NOT lock up.
I'm using a simple perl wrapper code that calls my program. The wrapper code starts displaying
random and non-ASCII characters.

#!/usr/bin/perl
$cnt=0;
while(1) {
sleep(2);
$filename = sprintf("data_%5.5d", $cnt);
system("./sr DATA/${file_name}");
printf("Captured data # %s\n", $filename);
$cnt++;
}


Once this started happening I dmesg for the DATA Toggle message from the patch and sure enough
the message got logged.

raspberrypi kernel: [84808.912275] ERROR::handle_hc_datatglerr_intr:1950: Data Toggle Error - on endpoint type BULK

I'll do a rpi-update and see how that goes
Oh snap - now we're getting somewhere

My instincts tell me that this was a genuine error (as is catered for in the USB spec) but the behaviour of the program afterwards is what is intriguing.

The "garbage" received after the error event says that the host is still properly receiving data and passing it up through the various levels of interrupt handling / kernel / peat / bog / brackish water / userland which is a marked improvement over the entire device freezing.

Unfortunately my somewhat blind patch doesn't say what happened during this event - was the DMA channel halted? Was it an IN or an OUT? What other halt bits were set? How do we recover cleanly from this apparently broken behaviour?

USB-serial converters are one application area where the Pi can be expected to run for weeks without user intervention - some examples are as a local Modbus/RTU master (for some sort of distributed control), data logger, telescope controller (which is what I want to do with it) and of course whatever you all have in mind.

This is where silicon documentation (such that it is) would be helpful.

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Tue Nov 27, 2012 10:42 pm

I'm going to add some additional debug messages to my code to help narrow down when, where, which direction the error is happening.
I'll post my results once I get them

michaeljquinn
Posts: 39
Joined: Sun Oct 21, 2012 8:19 am

Re: USB/serial converters very broken on Pi

Wed Nov 28, 2012 1:31 am

This is excellent progress.

My use case for the pi is as a solar PV monitor using the FTDI USB-RS485 adapter, and i envisage this running for months without intervention. I got some stability, with wired ethernet, but adding an RT5370 based wifi(a recommended) has broken this again.

I watch with interest :roll:

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Wed Nov 28, 2012 3:05 am

I'm in the process of adding addition debug messages to dwc_otg_hcd_intr.c to try and get more info
about the DataToggle error.

nurquhar
Posts: 16
Joined: Thu May 10, 2012 7:11 pm
Contact: Website

Re: USB/serial converters very broken on Pi

Wed Nov 28, 2012 1:27 pm

caldimerda wrote:There is a lot of helpful stuff on the forum relevant to this but it isn't very easy to find. This worked for me:

apt-get update
apt-get upgrade
wget http://goo.gl/1BOfJ -O /usr/bin/rpi-update && sudo chmod +x /usr/bin/rpi-update
apt-get install git-core
rpi-update
raspi-config (and update from the menu)

Plus adding dwc_otg.speed=1 to /boot/cmdline.txt which sets the USB to full speed instead of high speed and slows a bunch of things down - none of which really matters to me.

I had exactly the same symptom with two expensive usb--->serial cables. On the other hand the cheap cable worked without all these changes. Now all three cables behave the same - they work aside from CRC errors when reading a DS2438 on a 1-wire bus. I think someone has a fix for this but I don't think it's been incorporated on github.

Good luck.
Success :D
I did all the update stuff (again) and put "dwc_otg.speed=1" in /boot/cmdline.txt and now when I cat a file to my (/dev/ttyUSB0) FTDI RS485 converter the tx led's flicker and the RPi did not crash.
I suspect it was the dwc_otg.speed=1 which swung it.

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Thu Nov 29, 2012 1:02 am

no new news. The program has been running for about 20 hours with no failures

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Thu Nov 29, 2012 3:58 am

Here is the information I got from the debug message I added to the code.

Code: Select all

 ERROR::dwc_otg_hcd_handle_hc_n_intr:2237:   hcint 0x00000402, hcintmsk 0x00000006, hcint&hcintmsk 0x00000002
 ERROR::handle_hc_chhltd_intr_dma:2076: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error  hcint.d32= 0x00000402  intsts=0x06600009
 ERROR::handle_hc_chhltd_intr_dma:2117: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error   intsts=0x06600009
 ERROR::handle_hc_datatglerr_intr:1950: Data Toggle Error - on endpoint type BULK




 
 ERROR::dwc_otg_hcd_handle_hc_n_intr:2237:   hcint 0x00000402, hcintmsk 0x00000006, hcint&hcintmsk 0x00000002
 ERROR::handle_hc_chhltd_intr_dma:2076: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error  hcint.d32= 0x00000402  intsts=0x06600001
 ERROR::handle_hc_chhltd_intr_dma:2117: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error   intsts=0x06600001
 ERROR::handle_hc_datatglerr_intr:1950: Data Toggle Error - on endpoint type BULK

 
 ERROR::dwc_otg_hcd_handle_hc_n_intr:2237:   hcint 0x00000402, hcintmsk 0x00000006, hcint&hcintmsk 0x00000002
 ERROR::handle_hc_chhltd_intr_dma:2076: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error  hcint.d32= 0x00000402  intsts=0x06600001
 ERROR::handle_hc_chhltd_intr_dma:2117: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error   intsts=0x06600001
 ERROR::handle_hc_datatglerr_intr:1950: Data Toggle Error - on endpoint type BULK

 
 ERROR::dwc_otg_hcd_handle_hc_n_intr:2237:   hcint 0x00000402, hcintmsk 0x00000006, hcint&hcintmsk 0x00000002
 ERROR::handle_hc_chhltd_intr_dma:2076: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error  hcint.d32= 0x00000402  intsts=0x06600009
 ERROR::handle_hc_chhltd_intr_dma:2117: HANDLE_HC_CHHLTD_INTR_DMA::  DATA Toggle Error   intsts=0x06600029
 ERROR::handle_hc_datatglerr_intr:1950: Data Toggle Error - on endpoint type BULK

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: USB/serial converters very broken on Pi

Thu Nov 29, 2012 9:32 pm

The two clues here are that we are now down to a very low failure rate, and the causes have all boiled down to a common failure point.

I've seen only one such data toggle error and it appears tedh has to leave a serial converter running for hours before any crop up.

This leads me to believe that we are now in the realms of "real" toggle errors - created by incipient noise/cosmic rays/variations in the aether. The question now is how to handle them properly. In theory, a host controller shouldn't care about toggle errors as this is part of the defined automatic behaviour of a USB transaction. I wonder if it is particular to serial converters - they are transferring (by USB standards) very small amounts of data (a dozen bytes or so), which are never going to be spread over more than one packet.

I wonder if it isn't a fault of the converters themselves - do they transmit the data and then give up? Is the reason for the DMA channel being halted in the controller because of some other non-conformance by these converters resulting in a default "transfer is complete" behaviour but with an unhandled error condition.

So is it a quirk of the pi or of the converters - easiest substitiution would be the controller. Changing the dwc controller for an EHCI controller perhaps. Although annoyingly USB on the PC "just works".

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Fri Nov 30, 2012 1:54 am

Sorry if this get posted twice. I don't know if my previous post actually got submitted.

I agree with M33P that there can be other issues that could be causing the problem.
While my program is running I'm monitoring the RPI voltage as well as I have and XBee Pro connected to the RPI.

One thing that I have notice is that last 3 failures have all happen during the same spot in my code
Very short data transfer.
The Arduino sends a value of "255" to the RPI using the XBEE.
After the RPI detects the "255" then the RPI sends the char "s" to across the XBEE to the Arduino.
The Arduino then returns the number of bytes that need to be read back to RPI.

The RPI times-out waiting for the number of bytes to be returned. Once this happens the logs files
shows the Data-Toggle error.
There is a remote sensor connected to the Arduino and if the remote sensor doesn't respond
then the Arduino doesn't send anything back (I'm going to change this).

Did the time-out happen due to the Data-Toggle error?
or
Did it happen due to the remote sensor not talking to the Arduino?
Why haven't I seen this when running via the PC instead of the RPI?

Looks like I have some addition debugging to do with my system

Howard
Posts: 64
Joined: Sun Mar 04, 2012 7:38 pm
Contact: Website

Re: USB/serial converters very broken on Pi

Fri Nov 30, 2012 3:55 am

I am still getting frequent crashes with USB serial.

Code: Select all

Linux despatcher 3.2.27+ #285 PREEMPT Tue Nov 20 17:49:40 GMT 2012 armv6l GNU/Linux
cmdline.txt:
dwc_otg.lpm_enable=0 sdhci-bcm2708.sync_after_dma=0 sdhci-bcm2708.enable_llm=1 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
lsusb:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. 
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. 
Bus 001 Device 004: ID 058f:9254 Alcor Micro Corp. Hub
Bus 001 Device 005: ID 05ac:0306 Apple, Inc. Optical USB Mouse [Fujitsu]
Bus 001 Device 008: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 001 Device 009: ID 099a:610c Zippy Technology Corp. EL-610 Super Mini Electron luminescent Keyboard
I am doing loopback testing of 25 byte packets. 9600 baud, 10 per second, most tests are 500 packets. Each test opens and configures the port, send/rx data, close.
The Pi crashes fairly repeatedly every THREE such tests. On the third (occasionally fourth) test the port opens but the Pi crashes after a few (6-12) packets are handled. I don't think the length of each test run is significant. Most testing done directly connected, but same happens through hub.
I have no GPIO serial port so cannot identify the nature of the crash, all I see is total freeze.
PSU is 1.4A, USB polyfuses shorted, power twin fed to micro-USB and GPIO with short cable.

caldimerda
Posts: 68
Joined: Tue Oct 09, 2012 9:41 am

Re: USB/serial converters very broken on Pi

Fri Nov 30, 2012 11:20 am

tedh, M33P

Thanks for the work you guys (and others) are putting into this.

My app also transfers very little data. I have an old CM12 model X10 controller - so it's usually sending/receiving 1, 2 or 3 bytes for each event and there are at most maybe 20 events per minute. I wrote the app (Java, with rxtx for serial comms) 14 years ago and it's run at times (on windows and linux on i86) for 6, maybe up to 9 months without restarting the app or the pc.

On the raspi it's only gone 8 days between raspi reboots. This time the app ran 7 days, then I restarted it (config change) and it ran 1 day and then the USB fell over. When it hangs it knocks over the raspi and the CM12, which needs to be switched on and off as well as the raspi - and that took a frustrating hour to work out, I can tell you :-)

I'm using a dirt cheap USB-->serial that worked before all the kernel/firmware fixes, although I applied them anyway, plus bumping usb down to v1.1. My expensive cables needed the s/w upgrades. I'm not (alas) up to incorporating your debugging code and recompiling - but if there is ever something to download I'll give it a try.

I also have another pi, using an expensive cable to read from a DS 1-wire bus. That device has been up 9 days now and is still running.

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: USB/serial converters very broken on Pi

Fri Nov 30, 2012 10:43 pm

NB: Using the default Pi kernel (as got via rpi-update) will still crash horribly with a usb-serial converter if it hits a data toggle error. It is only by introducing some debug code that we downgrade the breakage from complete lockup to simply being unable to read anything sensible from the device.

I butchered my GPS device to create a loopback adapter using a suitably placed croc clip. It really is the most simple of usb-serial converters - 1x PL2303X and 1x level converter.

It seems I can get much more breakage out of the Pi when using loopback - using the default kernel, I managed to get a data toggle error (and crash) after a few tries with
echo derp > /dev/ttyUSB0
or
cat /dev/zero > /dev/ttyUSB0.

With the debug incorporated, I once managed to get something a bit different. I don't know how but I managed to create an interrupt storm on the Pi (by using the same/similar commands) resulting in nonresponsive ethernet and very very slow serial console. 99% of CPU time was sys (i.e. in kernel mode). This is perhaps the origin of the previously reported "less severe" breakage.

I haven't managed to get the Pi to crash yet with loopback and debug code on it.

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Fri Nov 30, 2012 11:06 pm

I'm going to modify my project as well.
I'm still going to use the XBEEs and Arduino but just to the few basic xfers only.

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Tue Dec 04, 2012 1:08 am

One question for everyone that is having issues.
when you are seeing problem are there more than one USB device plug into the RPI
This past weekend I was doing some testing. I only had the USB2Serial converter plugged in
and i did get some Serial read timeouts but I never got any Data Toggle errors.

Just wondering it we need multiple USB devices to get the Data Toggle errors

michaeljquinn
Posts: 39
Joined: Sun Oct 21, 2012 8:19 am

Re: USB/serial converters very broken on Pi

Tue Dec 04, 2012 1:23 am

I have the FTDI - RS485 adapter, and there is more stability with ethernet, rather than a RT5730 based wifi dongle. I definately still need the USB 1.1 setting though, leaving that out causes almost immediate lockup

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Tue Dec 04, 2012 2:17 am

michaeljquinn, Do you have a keyboard/mouse connected?

tedh
Posts: 21
Joined: Thu Nov 01, 2012 2:56 am

Re: USB/serial converters very broken on Pi

Fri Dec 07, 2012 3:26 am

I don't have any new information.
I currently working on connecting a FPGA to the USB bus to monitor what is happening when
the data-toggle error happens.

michaeljquinn
Posts: 39
Joined: Sun Oct 21, 2012 8:19 am

Re: USB/serial converters very broken on Pi

Fri Dec 07, 2012 8:56 am

tedh wrote:michaeljquinn, Do you have a keyboard/mouse connected?
nope - see below

got it running now, been rock solid 5 days straight, but yep, set on usb1.1
NO POWERED Hub, both wireless and FTDI adapter plugged straight in



interestingly, the lsusb -v tells me that the max power of the FTDI adapter is 300mW and the WIFI 450mW, maybe no much margin for power, but hey its running fine

Code: Select all

cmdline.txt
dwc_otg.microframe_schedule=1 dwc_otg.speed=1 dwc_otg.fiq_fix_enable=1 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
Linux raspberrypi 3.2.27+ #250 PREEMPT Thu Oct 18 19:03:02 BST 2012 armv6l GNU/Linux

Code: Select all

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 005: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter
cheers

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: USB/serial converters very broken on Pi

Sat Dec 08, 2012 8:06 pm

A thought occurs.

We have been waiting for data toggle errors to happen - what happens when we force them?

A key property of BULK packets is that they are typically 512 bytes in length - i.e. large.

What if we forced errors by injecting bogus data in the middle of a BULK packet?

This should be easy to do. We need to
a) detect a long USB packet whizzing along at 480MHz
b) at some suitable time afterwards (to prevent stomping on the headers) we intentionally corrupt the USB bus
c) We only do this once in a certain time frame to emulate the behaviour of a chance event

This should easily be constructable with even low-speed logic.

- AC couple the USB bus to a suitable radio frequency diode. Use this rectifier as what is essentially an AM envelope detector.
-RC-filter this to give a suitable rise/fall time that ignores short control/handshake packets but produces a logic 1 in the middle of a BULK packet.
- Using HC logic and a line driver produce a very short (0.5us) burst of "silence" on the USB bus
- Use a latch to disable subsequent detections.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1469
Joined: Sat Sep 10, 2011 11:43 am

Re: USB/serial converters very broken on Pi

Sun Dec 09, 2012 11:14 pm

I'd have to check, but think my USB analyser can do this much easier!

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

Return to “Troubleshooting”