I’m sure that you’ve heard of the Kinect from Microsoft by now, the revolutionary new technology that recognises the human form and lets you control games and apps with it. When the Kinect was released, there was big news surrounding it, not due to what it could do, but what developers were doing with it. In this tutorial, you’ll learn how to create a Pong game that can be controlled by moving your hands, and a picture gallery that can be navigated in the style of Minority Report — all in the browser’s Flash Player with no drivers required, if you’re on the Mac.
The Kinect got hacked within a few weeks, and developers started using it to control things other than the Xbox 360. This was huge, suddenly every nerd’s dream of living out that famous scene from minority report was coming to life. But for us flash dev’s we couldn’t get in on the action without having to learn C++. That makes it tricky, for someone like myself who just wants to make games with it.
Well, luckily for us, we now can, thanks to something called TUIO and some very nice developers who got kinect recognition streaming out the data via TUIO.
This description of TUIO is taken directly for their site, tuio.org:
TUIO is an open framework that defines a common protocol and API for tangible multitouch surfaces. The TUIO protocol allows the transmission of an abstract description of interactive surfaces, including touch events and tangible object states. This protocol encodes control data from a tracker application (e.g. based on computer vision) and sends it to any client application that is capable of decoding the protocol.
That’s quite a description. Basically, what it does is allow the decoding of touch interfaces like touch tables, through blob recognition. What clever dev’s have done is transcode the Kinect data into “blobs” so that TUIO can recognise them. From there, Flash can read the data with the help of TUIO’s native as3 libraries.
What TUIO can also do is recognise gestures. This is quite a nice feature as it allows you to do things with the kinect as if it was a big imaginary touch table. You can shrink or expand objects using a larger, more elaborate version of the pinch gesture, it can regonise swipes and touch gestures too.
For basic implementations though, a Kinect plus TUIO lets you track hands, which is what I’ll be covering first since that is something I’ve most recently used with great success in my project Spaced Out.
Spaced Out is a very experimental game. It used the Kinect to track players’ hands which served in game as the equivalent of a mouse. You can track the position of hands on screen and assign a cursor to them. In my game I used this in conjunction with a brainwave reading headset to blow up enemies on screen by tracking the players hands and also by reading their concentration levels.
You can see a video of all this in action here:
Spaced out gameplay
Spaced Out Gameplay from jon reid on Vimeo.
Different TUIO Trackers
There are a few different TUIO trackers for the Kinect right now. I’ll explain briefly what each one does in a minute, but they all have their advantages and uses for different occasions. Being able to have the freedom to choose which one we can use is a great advantage for us.
The great thing about TUIO is that all of these will use the exact same AS3 code that I’m going to show you today. So for different uses, you can just run a different TUIO tracker and your game or app will need no rewriting whatsoever.
The downside to TUIO trackers for the kinect right now is that they are generally mac only. This is due to driver issues when the Kinect first got hacked. I’m sure though, that as time goes on that more developers will release TUIO trackers for windows, especially since the official Microsoft SDK for Kinect has been released on Windows. In fact, there is a TUIO client for Windows right now! But it’s still in beta and you cant control the depth of the camera so I had to stand very far back to get it to work. It’s not something thats as easy to use as the Mac versions.
So the first one, known as TUIOKinect, tracks blobs, which it determines are hands. I personally find this one great as it needs no setup or calibration from the user. A player can just walk up to your game and use it instantly. The downside is doesn’t know exactly what a hand is, it’s just guessing. so if you are standing too close to the Kinect, it could think that any part of you is your hand. Only a minor issue when working with the zero calibration.
The second tracker, OpenNI2TUIO, tracks the entire skeleton to determine which part of you is a hand. Brilliant, very accurate tracking of hands. This one however does require the “superman” pose, so it’s good for singular apps that you are using for lengths of time, or ones where you require more accurate recognition of hands.
The Windows version, if you are interested in playing around with that, is called Open Exhibits TUIO Kinect. I got this running pretty quickly; you have to follow the readme.txt that comes with it and install some drivers, but it does just about work. It doesn’t give you the same control over what it tracks like the other two do, but it tracks what it thinks are hands just like the Mac version of TUIO kinect. But you Windows guys can play along too.
Step 1: Downloading Software
For this tutorial, I’ll be using TUIOKinect, just because I like being able to sit at my desk and still use the kinect. Of course, you can use one of the other trackers, but if you follow along as I cover the basics, then hopefully using another tracker will become nice and easy.
Download one of the trackers from the links above, for mac, the trackers dont need any installing and run straight away. For Windows you’ll need to install some drivers. They cover that in their readme.txt so I won’t go over that here.
Step 2: Download the Bridge
Downloading TUIO kinect isn’t the only step, you need a way to send that data specifically to flash. There is a bridge that does just that and is very simple to run. udp-flashlc-bridge by Georg Kaindl is the software I’ll be using today. You can get that here:
NOTE, if you are running the Windows version, you don’t need this as it comes bundled with similar software.
Step 3: Unzip Flash Bridge
Once you have downloaded that, extract to somewhere like your desktop or your documents folder, somewhere thats easy to get to. I say this because we have to run this bridge via a terminal command.
Step 4: Rename the Folder
Rename the folder to “flashBridge”, without the quotes, just because its default name is very long winded.
Step 5: Run the Bridge
Lets do that now, we can start it now and it’ll be running for the rest of the tutorial. Once it’s up and running, you don’t need to worry about it again, it’s a very stable bit of software.
Open up terminal and enter the following, I use desktop here, but replace that with wherever you put your folder:
cd desktop/flashBridge ./udp-flashlc-bridge
Your terminal window should now show two extra lines and confirm that its running fine. If it isn’t, it’ll tell you, so make sure you go over the last couple of steps and confirm that your are following them exactly.
It should look something like this:
