j_bali4
Posts: 6
Joined: Sat Aug 01, 2020 3:20 pm

Pi4 USB messages dropping

Sat Aug 01, 2020 4:09 pm

Having issues on Only the Pi4 with the Intel Realsense T265. This is a device that can be used in robots and similar applications to track the position and orientation of the camera in space. It sends this position message as an interrupt transfer at 200Hz

Essentially, you can get data from this device no problem, but anywhere from 30 seconds to 5 minutes later the interrupt transfer data just stops. Bulk transfers continue just fine. Video streams from the cameras are not used, so are not being sent over USB.

Their process is:
A new initial request is generated.
This request gets forwarded to libusb.
When a response arrives the interrupt_callback gets called.
At the end of the callback function a new request is created.

This process breaks down because either SDK creates the request, but it gets stopped in the pipeline somewhere, or it creates the request, but the response gets stopped in the pipeline, which breaks this chain. After a period of time, the T265 device sends a message saying

Code: Select all

T265 FW message: 13042424847758289392: [0x�/2:721] Host not reading - stopping
which clearly gets through from device to application.

This problem only seems to occur on the Pi4, not the Pi3, nor Jetson, nor laptop/desktops. Does not matter if it a USB2 or USB3 port. Kernel 4.x or 5.x makes no difference, neither does the beta VL805 firmware. Libusb and Libuvc both exhibit this problem. All this together makes me think that it is something specific to the Pi4 hardware, which is why I hope that posting here might help, even though its a third-party device having the issues.

Device Info:

Code: Select all

Bus 001 Device 007: ID 8087:0b37 Intel Corp.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x8087 Intel Corp.
  idProduct          0x0b37
  bcdDevice           ff.ff
  iManufacturer           1 Intel(R) Corporation
  iProduct                2 Intel(R) RealSense(TM) Tracking Camera T265
  iSerial                 3 944222110052
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x003c
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           6
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              4 Bulk Data
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x0016
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000006
      BESL Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat          32 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2436
Joined: Thu Jul 11, 2013 2:37 pm

Re: Pi4 USB messages dropping

Mon Aug 03, 2020 2:18 pm

Can you capture an xhci trace up until the point where the firmware reports the failure using the method in the linked post?

viewtopic.php?t=250990&start=75#p1560630
Rockets are loud.
https://astro-pi.org


jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2436
Joined: Thu Jul 11, 2013 2:37 pm

Re: Pi4 USB messages dropping

Tue Aug 04, 2020 9:32 am

Either your driver is doing something a bit weird or the log is incomplete. I see thousands of interrupt URB submissions, but with no corresponding events to say that they've completed. Reported events only start hundreds of seconds later, but I can see the dequeue pointers for the endpoint advancing despite the lack of events.

I note that your driver is only submitting a single interrupt URB at a time, as the enqueue and dequeue pointers for the ring are never more than 0x10 apart. This is generally bad practice for high-speed endpoints, as you may miss a service interval if userspace gets delayed and doesn't resubmit a URB in time. With an endpoint interval of 1, this means you have a <125uS service window before you miss a microframe of data.

Does the firmware fall over faster if you add an artificial 30/50/70uS delay in the resubmit loop in start_interrupt()?
Rockets are loud.
https://astro-pi.org

j_bali4
Posts: 6
Joined: Sat Aug 01, 2020 3:20 pm

Re: Pi4 USB messages dropping

Tue Aug 04, 2020 2:57 pm

The driver is not well written, so its hard to read. Is it possible the corresponding completed events only appear once the driver fails? The driver is clearly getting data from the device during that time, so even without the completed event it receives.

To add the delay I will have to edit and rebuild, but I can tell you that if you are running another realsense device, like their depth camera, the failures are much more frequent.

Does the service window timing imply that the execution time of the start_interrupt() callback, from receiving to submitting a new one, must be under 125uS? Would this be easier if bInterrupt was changed from 1 to 2, or is it better to queue up the next request earlier then not worry about remaining execution time?

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2436
Joined: Thu Jul 11, 2013 2:37 pm

Re: Pi4 USB messages dropping

Tue Aug 04, 2020 3:26 pm

One alternate explanation for "missing" messages in the trace could be per-cpu buffer overflow of the tracer. Each is implemented as a FIFO so if e.g. CPU0 is handling all the interrupts then it will be getting orders of magnitude more spam than the others.

If a URB gets returned to the host application and is resubmitted more than 125uS later, then the transfer is guaranteed to not occur in the next microframe. A robust firmware implementation shouldn't report a fatal error for not being serviced in its desired interval.

If running two cameras simultaneously increases the frequency of failure then I'd say missing a service interval is a probable cause of firmware giving up. Adding random delays to the processing callback would prove this.
Rockets are loud.
https://astro-pi.org

j_bali4
Posts: 6
Joined: Sat Aug 01, 2020 3:20 pm

Re: Pi4 USB messages dropping

Tue Aug 04, 2020 3:40 pm

If its a driver issue, why does it seem that the pi3 and jetson nano, which are both slower cpus, don't also exhibit this problem?

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2436
Joined: Thu Jul 11, 2013 2:37 pm

Re: Pi4 USB messages dropping

Tue Aug 04, 2020 3:55 pm

I have no idea what hardware is on the Nano.

Pi 3 is USB2.0 only. What does lsusb report when the device is plugged in at USB2.0 speeds?
Rockets are loud.
https://astro-pi.org

j_bali4
Posts: 6
Joined: Sat Aug 01, 2020 3:20 pm

Re: Pi4 USB messages dropping

Tue Aug 04, 2020 4:12 pm

There is no difference in the behavior between the usb2 and usb3 ports.

USB2:

Code: Select all

Bus 001 Device 009: ID 8087:0b37 Intel Corp.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x8087 Intel Corp.
  idProduct          0x0b37
  bcdDevice           ff.ff
  iManufacturer           1 Intel(R) Corporation
  iProduct                2 Intel(R) RealSense(TM) Tracking Camera T265
  iSerial                 3 944222110052
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x003c
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           6
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              4 Bulk Data
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x0016
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000006
      BESL Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat          32 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)
USB3

Code: Select all

Bus 001 Device 011: ID 8087:0b37 Intel Corp.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x8087 Intel Corp.
  idProduct          0x0b37
  bcdDevice           ff.ff
  iManufacturer           1 Intel(R) Corporation
  iProduct                2 Intel(R) RealSense(TM) Tracking Camera T265
  iSerial                 3 944222110052
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x003c
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           6
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              4 Bulk Data
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x0016
  bNumDeviceCaps          2
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000006
      BESL Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat          32 micro seconds
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

j_bali4
Posts: 6
Joined: Sat Aug 01, 2020 3:20 pm

Re: Pi4 USB messages dropping

Thu Aug 06, 2020 3:11 am

So, adding in the delay has actually improved driver stability. Its not by any means fixed, but it seems to take significantly longer between failures with anything above 60us delay, although I have not tested above 120us. Still odd behavior, and even more so because its only occurring on the pi 4, but some progress is nice.

srapp
Posts: 1
Joined: Thu Aug 06, 2020 7:06 am

Re: Pi4 USB messages dropping

Thu Aug 06, 2020 7:13 am

I'm absolutely happy, that you picked up this issue here - so there is just coming up a little light at the end of the tunnel...
@j_bali4: Could you please describe, what you have done to add the delay? What OS are you currently running?

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2436
Joined: Thu Jul 11, 2013 2:37 pm

Re: Pi4 USB messages dropping

Fri Aug 07, 2020 9:34 am

j_bali4 wrote:
Thu Aug 06, 2020 3:11 am
So, adding in the delay has actually improved driver stability. Its not by any means fixed, but it seems to take significantly longer between failures with anything above 60us delay, although I have not tested above 120us. Still odd behavior, and even more so because its only occurring on the pi 4, but some progress is nice.
That's significant, but not in the way that I was expecting. I would have thought that delay would either permanently break it, or have no effect. It does still point to something timing-related in the interaction between controller and device but without e.g. a bus analyzer it's going to be nigh impossible to find out what.

Are these devices available at retail?
Rockets are loud.
https://astro-pi.org

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9539
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi4 USB messages dropping

Fri Aug 07, 2020 1:01 pm

jdb wrote:
Fri Aug 07, 2020 9:34 am
j_bali4 wrote:
Thu Aug 06, 2020 3:11 am
So, adding in the delay has actually improved driver stability. Its not by any means fixed, but it seems to take significantly longer between failures with anything above 60us delay, although I have not tested above 120us. Still odd behavior, and even more so because its only occurring on the pi 4, but some progress is nice.
That's significant, but not in the way that I was expecting. I would have thought that delay would either permanently break it, or have no effect. It does still point to something timing-related in the interaction between controller and device but without e.g. a bus analyzer it's going to be nigh impossible to find out what.

Are these devices available at retail?
It looks like Mouser have them - https://www.mouser.co.uk/ProductDetail/ ... hqgg%3D%3D
Or as a bundle with a D435 - https://www.mouser.co.uk/ProductDetail/ ... fI5Q%3D%3D
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Advanced users”