Kinect - Would it ever work with a Raspberry Pi?

[jmacey]

I've managed to compile the freenect libraries (didn't bother with the demos etc) as well as libusb from source. I've managed to get a hacked version of the tiltdemo.c program to work, by not using the sync commands but using a proper contex from the device. I will see if I can get video later today (going to have to write my own OpenGL / EGL wrapper) as the glut demos will not work under the pi.

I'm uploading a video of the tilt working to youtube now will add a link once done.

[practisevoodoo]

Can you post the code?

[jmacey]

Let me tidy it up a bit first, it's a major hack
Here is the video of it working http://www.youtube.com/watch?v=UfmiPEVQ ... e=youtu.be

[theunit]

jmacey, I found the freenect tiltdemo example to work fine without any hacks. I did write some code to grab a depth image from kinect and save it to file - this seemed like the most basic program - but I have not yet tested it. No-one has reported success of grabbing depth images from the Kinect with libfreenect yet (that I know of). I wish the Asus Xtion worked with freenect, OpenNI is so bulky, especially for the RPi.

[jmacey]

For some reason I didn't work for me but once I used the freenect_set_led rather than the sync versions it was all fine. Am working on a video demo now will report back later

[jmacey]

Here is my first blog post about getting the tilt stuff working, http://jonmacey.blogspot.co.uk/2012/07/ ... inect.html will start on the video stuff later.

[Dimi]

as long as you wont try to use the depth cam (maybe opengl components) the tilt demo works quite fine without haking it.

[jmacey]

So far I can get the depth and the video camera's to initialise ok, I can see the IR glowing, however there seems to be an issue that the freenect_set_video_callback and freenect_set_depth_callback don't properly register my callbacks and I just grab the initial zero buffer.

I'm going to see If I can dig deeper into the freenect source and see what is going on, I think the main issues are to do with sync speed, I'm getting data to and from as the debug is working ok

Code: Select all

Control cmd=0016 tag=0000 len=000a: 18
Control reply: 18
start_lines:    0
end_lines:      0
cropping_lines: 0
Control cmd=0004 tag=0001 len=000a: 18
Control reply: 18
dcmos_emitter_distance: 7.500000
dcmos_rcmos_distance:   2.300000
reference_distance:     120.000000
reference_pixel_size:   0.104200
Control cmd=0016 tag=0002 len=000a: 18
Control reply: 18
ax:                1657
bx:                29
cx:                6
dx:                466
ay:                13
by:                2683
cy:                588
dy:                30
dx_start:          890
dy_start:          11433
dx_beta_start:     10062
dy_beta_start:     130801
dx_beta_inc:       170
dy_beta_inc:       136
dxdx_start:        2095599
dxdy_start:        631
dydx_start:        693
dydy_start:        2096048
dxdxdx_start:      134215779
dydxdx_start:      134217220
dxdxdy_start:      134217092
dydxdy_start:      134217003
dydydx_start:      134217189
dydydy_start:      134214840
Control cmd=0016 tag=0003 len=000a: 18

However getting this (which is from the usb_libusb10.c file)

Code: Select all

iso_callback(): failed to resubmit transfer after successful completion: -1
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
iso_callback(): failed to resubmit transfer after successful completion: -1
iso_callback(): failed to resubmit transfer after successful completion: -1

Will dig deeper, btw the code I'm using is based on my blog post here http://jonmacey.blogspot.co.uk/2010/12/ ... esign.html

[jmacey]

So it does seem to be a sync / timeout error, turning on USB debug gives this (and I've changed the timeout values in the kinect lib to loads of different values to no avail)

Code: Select all

[Stream 80] Invalid magic 0000
libusb: 0.734725 debug [submit_iso_transfer] need 1 32k URBs for transfer
libusb: 0.735193 error [submit_iso_transfer] submiturb failed error -1 errno=4008
libusb: 0.735427 debug [submit_iso_transfer] first URB failed, easy peasy
iso_callback(): failed to resubmit transfer after successful completion: -1
libusb: 0.735697 debug [libusb_handle_events_timeout_completed] doing our own event handling
libusb: 0.735879 debug [handle_events] poll() 4 fds with timeout in 60000ms
libusb: 0.736095 debug [handle_events] poll() returned 1
libusb: 0.736298 debug [reap_for_handle] urb type=0 status=0 transferred=0
libusb: 0.736479 debug [handle_iso_completion] handling completion status 0 of iso urb 1/1
libusb: 0.736671 debug [handle_iso_completion] last URB in transfer --> complete!
libusb: 0.736849 debug [disarm_timerfd] 
libusb: 0.736987 debug [usbi_handle_transfer_completion] transfer 0x14840cc has callback 0x4023792c
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
[Stream 80] Invalid magic 0000
libusb: 0.737425 debug [submit_iso_transfer] need 1 32k URBs for transfer
libusb: 0.741147 error [submit_iso_transfer] submiturb failed error -1 errno=4008
libusb: 0.741431 debug [submit_iso_transfer] first URB failed, easy peasy
iso_callback(): failed to resubmit transfer after successful completion: -1
libusb: 0.741712 debug [libusb_handle_events_timeout_completed] doing our own event handling
libusb: 0.741907 debug [handle_events] poll() 4 fds with timeout in 60000ms
libusb: 0.742115 debug [handle_events] poll() returned 1
libusb: 0.742324 debug [reap_for_handle] urb type=0 status=0 transferred=0
libusb: 0.742856 debug [handle_iso_completion] handling completion status 0 of iso urb 1/1
libusb: 0.743074 debug [handle_iso_completion] last URB in transfer --> complete!
libusb: 0.743253 debug [disarm_timerfd] 
libusb: 0.743407 debug [usbi_handle_transfer_completion] transfer 0x1485b44 has callback 0x4023792c
libusb: 0.743603 debug [submit_iso_transfer] need 1 32k URBs for transfer
libusb: 0.744082 error [submit_iso_transfer] submiturb failed error -1 errno=4008
libusb: 0.744313 debug [submit_iso_transfer] first URB failed, easy peasy
iso_callback(): failed to resubmit transfer after successful completion: -1

Enough code for one day will look again tomorrow

[ianmacs]

Just a few confirmations:

1) Kinect, despite coming with its own power plug, needs to be connected to the pi using a powered USB hub. Else, only the motor will be listed in lsusb, and libfreenect will not find a kinect device.

2) Not all powered USB hubs will work. Tried 4 hubs (3 different models), only one did work. This is the model of which I have two instances, the other instance does not work, so I cannot recommend any model.

3) When driving the motor, the pi reports ethernet problems. dmesg shows "eth0: kevent 4 may have been dropped"

4) Setting LEDs works. Also, the infrared laser diode can be switched on using freenect_start_depth.

5) freenect_process_events never calls the registered depth callback. Depth images do not reach the pi.

So it seems that Kinect depth sensing will not work with the pi.

[hitcher404]

Have experienced similar....

libfreenect can control leds and motor, but seems the callbacks in examples never get run, just end up with empty arrays where there should be depth/rgb data.

Compiled OpenNI and SensorKinect on Raspbian, but again no progress, running SampleNiSimpleRead just gives:

Code: Select all

UpdateData: a timeout has occurred waiting for new data. 

I am still hopeful though, seems people have got it working with not only the beagleboard-xm, but also the more rpi similar plain beagleboard:

http://www.kinecthesia.com/2011/04/great-success.html

http://www.kinecthesia.com/2011/10/kine ... -come.html

[jmacey]

I actually hacked out all the threading code in the libraries, and it seems to be a USB bottleneck, I'm presuming it's just USB timeouts / async type problem.

There was an issue with the mac version and there had to be a patch for libUSB to make it work, my suspicion is that there is a similar problem with the pi, but I don't know enough about usb programming.

I will have another look next week when I get a chance.

[vannoo67]

This guys says he had success by "Manually build[ing] [Libusb] with the libusb0.1-compact"
see http://programmingwhilestupid.com/kinec ... pberry-pi/

Hope you have success with this. I have a Kinect sensor working with my PandaBoard, but I'd love to be able to replace it with a tiny inexpensive R-Pi.

[Dimi]

any prove for this?

[jmacey]

trying now but now luck so far

[jmacey]

A bit more detail.

I've built both libusb and libusb-compat, It seems that the compat lib is for using older USB devices, however when linking the kinect lib it only seems to use the libusb.

I've hacked things to use just the compat lib and it built but when running it complains that libusb_init is not found. This is part of the normal lib-usb library, so I seems that it needs both.

With both installed I still get the usual errors of iso_callback() failed and [stream 80] Invalid magic 0000 and no data passed to the image block.

I've left a message on the blog post to ask for more details, see what he says

Jon

[Dimi]

I have just got the ASUS Xtion sensor working on the raspberry pi using OpenNI......

thx to theunit, I could also get the Xtion with the Pi working, see the images attached.
the SimpleRead-Sample works much faster when its just in console mode (after booting) while the SimpleViewer-Sample is really slow.

I thought the Xtion (and the Kinect) would simply calculate everything on the hardware and then just send the results just to the windows output? But it seems, that the sample is calculating a lot of things by itself.

cheers
dimi

[theunit]

Nice work Dimi!

I see you got the SimpleViewer working, I thought there were issues with glut or something, it didn't work on mine. Did you do anything different? Which distro? I was using Debian squeeze, I am going to update to wheezy soon though.

[Dimi]

thx for your work in the first place!

yes the normal SimpleViewer isn´t compiling because of some glut errors, but I used the java-based one (org.OpenNI.Samples.SimpleViewer.SimpleViewer) this one is compiling and viewing just the depth-image. but again, reeeallyyy slow! maybe 1-2fps. and just when debian is started (with startx, obviously )
currently I am on squeezy, some things wasn´t working in wheezy, even it was quite a little bit faster.

[theunit]

I didn't even think to try the java samples. I would guess that the glut/c++ implementation would run a bit faster (if it worked).

[hitcher404]

Could either of you with Xtions post the output of lsusb -vv?

Many thanks.

[hitcher404]

Maybe if I explain my reasoning. The PrimeSense Sensor driver defaults to using bulk endpoint transfers when compiled on arm, from GlobalDefaults.ini:

Code: Select all

; USB interface to be used. 0 - FW Default, 1 - ISO endpoints, 2 - BULK endpoints. Default: Arm - 2, other platforms - 1
;UsbInterface=2

The output of lsusb -vv with kinect attached reports only isochronous endpoints:

Code: Select all

Endpoint Descriptor:       
 bLength                 7        
 bDescriptorType         5       
 bEndpointAddress     0x81  EP 1 IN        
 bmAttributes            1          
 Transfer Type            Isochronous          
 Synch Type               None          
 Usage Type               Data        
 wMaxPacketSize     0x0bc0  2x 960 bytes

Maybe the success with asus Xtion devices is because they support data transfers with less strict time constraints (bulk), where as the kinects iso transfers mean that on a device like the Pi, all frames end up being dropped as it can get no where near the 30 per second that seems the only option in both the libfreenect and PrimSense mod drivers? (Very small understanding of USB communication, may be totally wrong!)

Those with working Xtions, have you modified any of the settings in /usr/etc/primesense/GlobalDefaults.ini? What do the SamplesConfig.xml files you are using to run the samples look like, default? Apart from the usb transfer mode, I would be very interested in what resolution your settings files are causing, as again from GlobalDefaults.ini it seems when compiled on arm they differ:

Code: Select all

; 0 - QVGA, 1 - VGA, 4 - QQVGA. Default: Arm - 4, other platforms - 0
;Resolution=1

The avin2 SensorKinect driver breaks if you set any other resolution than VGA and fps other than 30.

I have tried modifying XnDeviceSensorIO.cpp in avin2s mod to use bulk transfers, always fails. Figure I was being optimistic thinking you could choose the transfer mode even after a particular endpoint had been enumerated/was set in firmware.

Also, from what I have seen so far, compiling libusb-1.0.x or libusb-compat from source has absolutely no effect, but I may be doing it wrong, seems it is only libusb-1.0.x that is used by both drivers, at least for the camera.

Thanks for any responses.

[Dimi]

Hey hitcher 404,
I didn´t modified GlobalDefaults.ini or the SamplesConfig.xml.
When i digg depper the SimpleRead-Sample, there is the full depth image information coming with 30fps and just the middle point is shown on screen. So in console (without startx) information are coming really fast.
And here is the ouput of my lsusb -v (-w isn´t possible)

Code: Select all

Bus 001 Device 005: ID 1d27:0600  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x1d27 
  idProduct          0x0600 
  bcdDevice            0.01
  iManufacturer           2 PrimeSense
  iProduct                1 PrimeSense Device
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           69
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              500mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           3
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0b70  2x 880 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x135c  3x 860 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x1040  3x 64 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       1
      bNumEndpoints           3
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               1
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0000
  (Bus Powered)

hope you can work with it.
cheers
dimi

[AjTech]

Hello everyone I have finally got my pi so i will see what i can do with the kinect running ont he pi.

Keep up the great work with kinect on pi.

[Dimi]

Hello everyone I have finally got my pi so i will see what i can do with the kinect running ont he pi.

Keep up the great work with kinect on pi.

Good Luck AjTech and keep us informed about your progress!