by Montgomery Martin
(Based on techniques presented during Troika Ranch's Live-I Workshop 2015)
Configuring Processing for Kinect Tracking
In Part 2 of this tutorial, we will learn how to download, install, and configure Processing to receive full body motion data and infrared video from an Xbox Kinect motion sensor. This data is broadcast via OSC and Syphon, allowing easy transmission into Isadora to create interactive media.
You should have previously completed the part 1 of this tutorial before continuing here.
This tutorial uses Processing, an open-source programming language and software sketchbook created to teach artists, students, designers, and hobbyists how to code within the context of the visual arts.
No prior knowledge of Processing is necessary for this tutorial. Instead, we have included step-by-step instructions on how to setup Processing so that you can use it to provide communication between a Kinect camera and Isadora. The Processing website has a huge library of resources if you are interested in learning more.
Step 1: Download the Tutorial Files from TroikaTronix
The Processing Sketches (.pde files) and Isadora files required for this tutorial can be downloaded from http://troikatronix.com/files/isadora-kinect-tutorial-files.zip
IMPORTANT! Steps 2 and 3 vary based on your operating system.
Step 2 (Mac OS): Download and Install Applications and Drivers
- Install Processing v2.2.1. It can be found under the heading "Stable Releases" on the page at http://processing.org/download/
IMPORTANT! You must install Processing v2.2.1 because SimpleOpenNI has not been updated to support the latest version of Processing!
- Note that there is no installer for Processing. You simply unzip the downloaded file and the Processing app will appear. To ensure that Processing's "home" folder is installed, double-click the Processing application and then immediately quit.
- Download this version of SimpleOpenNI v1.96 (112Mb). It has been updated to work with the Kinect model 1473 camera. The versions currently available on the SimpleOpenNI repository only work with the earlier model 1414 version of the Kinect. After the download completes, double-click the .zip file to unzip it. A folder called SimpleOpenNI will appear, which we'll use in a later step.
Step 2 (Windows): Download and Install Applications and Drivers
- Install the 32 bit version Processing v2.2.1. It can be found under the heading "Stable Releases" on the page at http://processing.org/download/
IMPORTANT! You must install Processing v2.2.1 because SimpleOpenNI has not been updated to support the latest version of Processing. The 32-bit version is required to enable Spout video sharing support.
- Note that there is no installer for Processing. You simply unzip the file to a location your drive, and you'll see the Processing v2.2.1 appear. Inside is the Processing.exe application and its associated files. To ensure that Processing's "home" folder is installed, double-click the Processing application and then immediately quit.
- Download the Kinect SDK v1.7. The Kinect SDK is required for SimpleOpenNI to operate under Windows. After the download completes, run the Kinect SDK 1.7 Setup.exe installer.
- From the the Spout downloads page, click the download link under the Spout logo to download the Spout installer. After the download completes, run the SpoutSetup.exe installer.
Step 3 (Mac OS Only): Manually Add the SimpleOpenNI Library to Processing
Because we're using a custom version of SimpleOpenNI, you will need to manually install it into Processing's library folder.
- In Step 2a, you unzipped the file "SimpleOpenNI_1.96_osx_kinect_1473.zip" and a folder called SimpleOpenNI was created.
- Select the folder SimpleOpenNI and choose Edit > Copy
- Go to the Finder
- Choose Go > Go to Folder... and enter “~/Documents/Processing/libraries/" without the quotation marks. Click OK.
- Press Cmd-V or choose Edit > Paste menu to put "SimpleOpenNI" in this location.
Step 3 (Windows Only): Install Spout Related Files
- Open the Spout downloads page.
- Click the download link under the Spout logo to download the file.
- After the download completes, run the SpoutSetup installer. This will install the Spout libraries to your computer. The default location is C:\Program Files (x86)\Spout2
If you are running an Isadora version less than 2.6, you'll also need to install two FreeFrameGL plugins to be able to use Spout within Isadora. (Spout has been integrated into Isadora 2.6 and newer)
IMPORTANT! The Isadora Kinect Tutorial File provided uses the Spout FFGL files to allow backward compatibility. If you do not install these files you will need to replace the "SpoutReceiver2" actor with the new Isadora 2.6 actor "Spout Receiver" before the video-sharing will work.
- Navigate to C:\Program Files (x86)\Spout2\PROGRAMS\FREEFRAMEGL
- Select the files SpoutReceiver2.dll and SpoutSender.dll. Then press Ctrl-C or choose Copy from the toolbar.
- Navigate to C:\Program Files (x86)\Common Files\. If you don't see a folder called "FreeFrame" then create one in this location. Then double-click the FreeFrame folder to open it.
- Press Ctrl-V or choose Paste from the toolbar to paste the items you copied above into theFreeFrame folder.
Step 4: Add Libraries to Processing using the Library Manager
- Launch the Processing application. An empty sketch opens by default.
- From the menu bar, select Sketch > Import Library > Add Library…The “Library Manager" dialog appears.
- In the search field, enter “oscP5”. Select and install oscP5 by Andreas Schlegel. This library provides support for Open Sound Control (OSC) input and output.
- MacOS Only: In the search field, enter Syphon. Select and install Syphon by Andres Colubri. Syphon is a free Mac OS tool that allows different programs to share video.
- Close the “Library Manager” dialog and quit Processing.
Step 4b: (Windows Only): Add Libraries to Processing Manually
- Open the Spout Processing Project releases page.
- Download the latest release. (at the time of writing this tutorial, release 2.0.6)
Open the Zip file and copy the contained 'spout' folder to your Processing's libraries folder, located at C:\Users\USERNAME\Documents\Processing\libraries (where USERNAME is the current username logged into your Windows OS).
Open the simpleOpenNI google code download page.
Download the latest release of simpleOpenNI. (at the time of writing this tutorial, SimpleOpenNI-1.96.zip)
Open the Zip file and copy the contained 'SimpleOpenNI' folder to your Processing's libraries folder, located at C:\Users\USERNAME\Documents\Processing\libraries (where USERNAME is the current username logged into your Windows OS).
Step 5: Ensure all Libraries are Installed
Mac OS: Navigate to /Users/USERNAME/Documents/Processing/libraries
Windows: Navigate to C:\Users\USERNAME\Documents\Processing\libraries
You should see these folders:
- spout (Windows OS Only)
- Syphon (Mac OS Only)
Mac OS: In the Isadora Kinect Tutorial Files folder, go into the Isadora_Kinect_Tracking_Mac folder. Inside you should see
- Isadora Skeleton Test Win.izz
Windows: In the Isadora Kinect Tutorial Files folder, go into the Isadora_Kinect_Tracking_Win folder. Inside you should see
- Isadora Skeleton Test Win.izz
Step 6: Test the Tracking Setup
- Connect your Xbox Kinect motion sensor to your computer in an available USB port. Ensure the sensor is connected to a power source.
- Open the Processing application again. Close the empty sketch.
- Mac OS: In the Isadora Kinect Tutorial Files folder, go into the Isadora_Kinect_Tracking_Mac folder and open the file “Isadora_Kinect_Tracking_Mac.pde”
Windows: In the Isadora Kinect Tutorial Files folder, go into the Isadora_Kinect_Tracking_Win folder and open the file “Isadora_Kinect_Tracking_Win.pde”
- This file contains all the essential programming to transmit skeleton data from a single performer via OSC, as well as the infrared depth image via Syphon or Spout.
- Click the “Run” button in the upper left-hand corner of the Processing window.
- Processing begins compiling the code, and a new window appears.You should see the depth image from the Kinect camera appear, but take note that it may take some time. (Typically it was about 10-20 seconds, but one member of our team had to wait two minutes before the image appeared; be patient!)
- Ask your performer to step in front of the sensor. Almost immediately you should see their body highlighted in color (usually green or blue), with a small “skeleton” superimposed over the body. This graphic feedback is an indication that Processing is tracking successfully.
- Ask your performer to step out of the sensor. We are ready to configure Isadora to receive the skeleton data from Processing.
Step 8: Use the Isadora Example File To See Tracking in Action
- With your Processing sketch still running, open the demonstration file we've prepared for you. (Mac OS = Isadora Skeleton Test Mac.izz / Windows =Isadora Skeleton Test Win.izz)
- Choose Output > Force Stage Preview to show the Stage preview. You should see the depth image from the Kinect, which is being received via Spout or Syphon. (Note: the image is displayed at 50% brightness.)
- Now have your performer step into the tracking area. You should see fifteen dots tracking the performer's body. Each dot represents one of the fifteen body parts tracked by the Kinect. Note that these will get larger when the performer moves towards the camera. That's the result of the perspective; as in real life, when objects are closer, they seem bigger.
- The dots are being generated by 3D Particles actors contained in the fifteen User Actors called "Kinect Skeleton Point". A pair of Broadcaster actors send messages to those User Actors allowing you to change the way the particles are rendered.
- In the Broadcaster labeled "Set Particle Count," change the 'value' input to 50. You'll see that the movement of the performer now leaves trails. Then, in the Broadcaster actor labeled "Set Particle Y Gravity," set the 'value' input to -100 or 100, as you prefer. Now the particles will flow downward or upward in response to a simulated gravitational force.
- Feel free to explore and modify the contents of the "Kinect Skeleton Point" to create your own variations on the sample patch we've provided.
That completes Part 2 of this tutorial. One last tip: Remember to STOP the Processing file by clicking on the 'stop' button, before closing the Processing application. We've seen a few crashes if you don't explicitly stop the Processing program first.