Getting Kinect V2 to work with Pi

[gpropf]

I'm a technical consultant and some of my projects use the RPi for a lot of things. I've got a client right now who sent me a Kinect V2 (the one for the Xbox One, not the 360) and wants me to write some code in the Processing language to run on the Pi for it.

I've been struggling to get this thing working though. There seems to be a lot of confusing and contradictory information out there and a lot of it seems to be from the 2010-2012 time window and contains lots of dead links and obsolete information. Can someone give me a basic idea of what to do just to get the thing talking to the Pi? Right now it would be a victory to just be able to run some of the sample code and see that it's working. I'm aware that there are two incompatible projects for it, freenect and a combination of OpenNI and SensorKinect. Also freenect seems to come in two versions with freenect2 being for the V2 Kinect and freenect for the V1. At least that's what I *think* is true.

My progress so far is basically that the Kinect shows up in the output of 'lsusb -v' so the kernel seems to see it on the USB bus when I plug it in. Running freenect-glview (a test program from the freenect package in Raspian) yields only

Code: Select all

freenect-glview 
Kinect camera test
Number of devices found: 0

So freenect doesn't even see the camera. I get the same problems on my much more powerful Ubuntu desktop. Any help is appreciated.

[Gavinmc42]

Just went looking for Kinect stuff too.
Found there is a Lazarus/FPC unit for it.
http://www.freepascal.org/~michael/arti ... kinect.pdf
Have not tested it on a Pi , don't have a Kinect 360 yet.

Are you talking Kinect 360 or Kinect One?
Don't think anyone has reverse engineered the Kinect One yet?
MS made it harder to use just to pi.. people off.
Ok did a google, maybe. Not much info out there.

It would be nice to use a Kinect and OpenGL ES for this.
https://www.youtube.com/watch?v=Fg3Xgn5OlTI

Perhaps someone could port 3D to the Picamera with IR filter and IR blaster.
Run it on a Pi Zero 1.3 as a USB 3D scanner.
Anyone know where to get a structured IR blaster lens?
Maybe a IR Led and negative of a dot plot?

Use two Zeros/cameras in stereo with the Projectors output?

[gpropf]

Are you talking Kinect 360 or Kinect One?

Kinect One

Don't think anyone has reverse engineered the Kinect One yet?

Supposedly they have and that's what freenect2 is for. Not sure what this Lazarus thing you mention is.

[Gavinmc42]

http://www.freepascal.org/
http://wiki.lazarus.freepascal.org/5dpo#About
It is abit old, FPC is now ver 3+.

I want to get the Kinect V1 working in Ultibo (baremetal OS) to get the speed up.
Which is why I was looking, can get them second hand sometimes.

Someone in the FPC forums is probably going to get the V2 to work?

USB3 only?
https://support.xbox.com/en-US/xbox-on- ... own-issues
https://social.msdn.microsoft.com/Forum ... inectv2sdk

Time of flight sensor?
http://www.tomshardware.com/reviews/mic ... 681-6.html

Maybe not, my fav site has a breakdown
https://video.search.yahoo.com/yhs/sear ... ction=view

Ok I'm a good googler, but there does not seem to be much out there, what happened?
http://www.businessinsider.com.au/why-m ... off-2015-9

https://en.wikipedia.org/wiki/Kinect_for_Xbox_One
Ok, starting to think this went to the same place as the Apple Newton, before it's time?

The USB3 requirement might just make it a no go for the Pi.
Plug it in and do lsusb?
Looking at the chips on that PCB probably going to need 1amp plus the rest.
https://support.xbox.com/en-US/xbox-on- ... own-issues

I will not be getting a One, would rather invest the time in trying to DIY with a Zero and Pi camera.

or just wait until an open source one appears?
https://en.wikipedia.org/wiki/Time-of-flight_camera

New image sensors for cameras are getting 3D tech like this.
Apple acquired Primesense then things went quiet.....

[gpropf]

The USB3 requirement might just make it a no go for the Pi.
Plug it in and do lsusb?

Now that you mention it, here's what my Pi 2 Model B says when you do that:

Code: Select all

pi@raspberrypi ~ $ sudo lsusb -v|grep -i nui
  iProduct                2 NuiSensor Adaptor      
  iProduct                2 Xbox NUI Sensor

Yes, this is the odd thing. It shows up! Even with only USB 2.0 the Pi can see it. Unfortunately, every attempt to actually do anything with it has failed so far. It doesn't even seem to be usable as a plain webcam.

Looking at the chips on that PCB probably going to need 1amp plus the rest.
https://support.xbox.com/en-US/xbox-on- ... own-issues

In my opinion power requirements and the sheer data rate of the Kinect V2 will probably always stifle efforts to make it work with a Pi. The problem is that power dissipation is proportional to data rate, though I'm not sure if the relationship is linear or logarithmic. The Pi probably doesn't have the CPU muscle to handle the KV2's data rate and couldn't dissipate the heat produced with only air cooling even if it did.

New image sensors for cameras are getting 3D tech like this.
Apple acquired Primesense then things went quiet.....

One of the most frustrating things about this Kinect stuff is the link-rot issue. Links from even 3 years ago to PrimeSense send you to some stupid 404 page at Apple.

At this point I'm giving up. I told my client that I have an Xbox 360 Kinect and we can probably make that work. Haven't heard back from her. Don't know if she's pissed off or just away. She does tend to disappear for days on end sometimes. Thanks for the links to the open source stuff. I'll be focusing more on that too. I'm also interested in possible OS approaches to things like augmented reality. I really don't want to see Occulus Rift and Microsoft dominating this stuff.

[Gavinmc42]

I'm was doing Kinect research because I was asked to see if this was possible on a Pi.
http://www.interactive-sandbox.com/

Was trying to google for a IR structured light projector to use with the Pi cameras.
Hey, there is already a projector of light.

So I am now looking about how to use stereo Pi cameras to get 3D data.
https://www.raspberrypi.org/blog/real-t ... te-module/
Compute module allows two cameras, might be a bit slow to calculated the 3D?
90ms sounds fast enough?
But the new compute3 might be fast enough to do two cameras plus HDMI output to projector?
If the Compute3 keeps the same price as the Compute1 then it is near enough to a Pi3 price.
Two Cameras adds more cost, similar to Kinect $$?

Two camera and compute for 3D vision- sight for the blind project?

[jpers]

I know this thread is a bit old, so I will try to elaborate on problem of combining Kinect V2 and Pi. I just set up the Kinect V2 on CentOS PC computer with USB3 ports. I am also salivating at the prospect of having ToF camera for raspberry pi for that price

First, the bad news:

• 1. Kinect V2 needs USB 3, since the data rates it needs surpass the bandwith of USB2.
  2. Kinect V2 driver & libraries need GPU (as CUDA, OpenCL, OpenGL - OpenGL ES is not supported) to perform some processing.

The possibly good news & speculation:

• 1. There is open source Linux library, called Freenect2. Not Freenect, Freenect2. The technology differences between Kinect V1 and Kinect V2 are so significant, that there are *separate* support libraries out there. GPU is not needed if you don't mind the framerate.
  2. The Kinect itself is sold as a human interface device, a controller, where high frame rates are a must. Purely from the technical standpoint, USB3 is not needed to transfer depth images only. Kinect V2's depth maps are of 512*424 16-bit pixels, that yields less than 0.5 MB per image, and approx 100 MBits/sec at 30 frames per second, well within USB2 spec. I don't know whether all data is transferred via USB3 at all times, but it is certainly possible for user not to subscribe to everything the device can possibly send out.

Now, the posible showstoppers & speculation:

• 1. Kinect V2 needs isochronous USB transfers. This means predefined latency, and the libraries definitely check for this. The question is whether the *device* checks for the latency? If it does via some kind of mandatory initialization routine, then the game ends here.
  2. If it does not, then the issue is perhaps only how to start and stop the transfer quickly enough for the USB connection to not get overloaded. Which may not be possible.

[Gavinmc42]

We have been buying up old Kinect 360's second hand.
EB Games has them sometimes as trade-ins.

I think the Kinect V2 is in the too hard right now basket.
Too much MS hidden stuff.
I first thought the V2 was going to be ToF but is it SL?
Info not friendly to hacker? Most info is from 2013
Not impossible?
https://blogs.technet.microsoft.com/mic ... -xbox-one/

Time of flight stuff is getting more common, use an array of these?
https://www.adafruit.com/product/3317

How long before they are off the shelf?
http://www.ti.com/lsds/ti/sensing-produ ... rview.page

ToF is going to be easier than structured light.
http://au.mouser.com/applications/time- ... -robotics/

ToF RGBA cameras with alpha giving the depth info?
It is starting to change the way movies are made now.

Just done a multi-spectral camera, really glad the software was someone else's problem.
Doing this stuff on Pi's is cool.

Playstation camera? Depth perception from the two cameras?

[jpers]

Now, the posible showstoppers & speculation:

• 1. Kinect V2 needs isochronous USB transfers. This means predefined latency, and the libraries definitely check for this. The question is whether the *device* checks for the latency? If it does via some kind of mandatory initialization routine, then the game ends here.
  2. If it does not, then the issue is perhaps only how to start and stop the transfer quickly enough for the USB connection to not get

Ok, checked this. Under Linux (centos 7), while the OS on USB2 port detects Kinect being plugged in, it does not report all the USB devices that the Kinect hardware provides (only the Kinect adapter box). Tested on same computer, first with USB3 enabled, then with USB3 disabled.

While it still may be possible to change this behaviour, it would certainly require kernel hacking at USB level (I am not sure whether negotiation takes place on the Kinect-OS level or somewhere deeper).

Anyway, here are relevant dmesg logs:

First, USB3 enabled:

Code: Select all

[   78.019568] usb 2-2: new high-speed USB device number 7 using xhci_hcd
[   78.186034] usb 2-2: New USB device found, idVendor=045e, idProduct=02d9
[   78.186038] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   78.186040] usb 2-2: Product: NuiSensor Adaptor
[   78.186045] usb 2-2: Manufacturer: Microsoft Corporation
[   78.187040] hub 2-2:1.0: USB hub found
[   78.187265] hub 2-2:1.0: 1 port detected
[   78.619805] usb 3-2: new SuperSpeed USB device number 4 using xhci_hcd
[   81.594166] usb 3-2: New USB device found, idVendor=045e, idProduct=02d9
[   81.594170] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   81.594172] usb 3-2: Product: NuiSensor Adaptor
[   81.594173] usb 3-2: Manufacturer: Microsoft Corporation
[   81.596634] hub 3-2:1.0: USB hub found
[   81.596766] hub 3-2:1.0: 1 port detected
[   82.054914] usb 3-2.1: new SuperSpeed USB device number 5 using xhci_hcd
[   82.066690] usb 3-2.1: New USB device found, idVendor=045e, idProduct=02c4
[   82.066694] usb 3-2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=4
[   82.066696] usb 3-2.1: Product: Xbox NUI Sensor
[   82.066697] usb 3-2.1: Manufacturer: Microsoft
[   82.066698] usb 3-2.1: SerialNumber: 005863664147

And with USB3 disabled:

Code: Select all

[  336.057228] usb 1-1.2: new high-speed USB device number 12 using ehci-pci
[  336.145106] usb 1-1.2: New USB device found, idVendor=045e, idProduct=02d9
[  336.145110] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  336.145112] usb 1-1.2: Product: NuiSensor Adaptor
[  336.145114] usb 1-1.2: Manufacturer: Microsoft Corporation
[  336.145879] hub 1-1.2:1.0: USB hub found
[  336.146800] hub 1-1.2:1.0: 1 port detected

EDIT: USB 1.x and USB2 negotiate the version/speed deeply at physical level (sending/receiving pulses, chirps, and then switching to higher speed). I cannot find details of USB2 - USB3 negotiation, but if we assume that the same method is used as with USB1 vs USB2 then it is pretty darn game over for Kinect v2 on raspberry pi, since no amount of SW hacking can convince the USB port to pretend that it is USB3 (assuming that kinect v2 insists on USB3, which now looks sure).

[aggarcia]

Raspberry Pi 4 now has usb3 ports. The game is on.

[Gavinmc42]

Raspberry Pi 4 now has usb3 ports. The game is on.

Yes those old Kinect 360s are hard to get now.
Been a while, how far has the reverse engineering on the Kinect One gotten?

Desktop AR Sandpit?
That Davis software needs OpenGL and I've had lots of fun playing with that since the Pi4 came out

A good place to start?
https://github.com/OpenKinect/libfreenect2
Hmm, preowned Kinect One $21AUD , Kinect 360 - $13.50AUD
Time to raid the small change jar

[cjastbury]

Hi
I have a working Pi4 4GB with Kinect V2, streaming colour video and colour image capture is working. Trying to see why the depth and Ir are not working

[CarlosGS]

Hi! I had the same problem, documented here: https://github.com/OpenKinect/libfreene ... -553855716

Seems like the freenect driver would need some modification, if RPi4 hardware permits it at all..