Microsoft Kinect Gesture Recognition & 3D Skeleton Tracking

Kinect Sensor Overview

Skeleton points

With the latest Kinect v2 sensor, powerful gesture detection features were introduced that have some really exciting applications for Psychological studies. They open up the possibility not only to track where a participant is in 3D space but also to detect specific poses, gestures and movements (perhaps as responses to stimuli) to a good degree of accuracy. It is now possible to “train” a gesture database to recognise key gestures / movements and detect them in participant responses in real time.

The Microsoft Kinect sensors were originally developed for body and gesture tracking in 3D space for players using the Xbox gaming console. The sensors employ depth-sensing, colour camera, IR emitter / IR camera (for low-light use) and a microphone array. The concept was that you can use your body as the controller for games. Microsoft coined the phrase “You Are the Controller”.

Kinect Sensor v1

Kinect v2 Sensor

The original concept was so successful that a vibrant user community developed ways to adapt the original Kinect sensor to work with a PC, allowing basic tracking of skeleton points in 3D space. All coding had to be done by hand and was very laborious and often flaky. Microsoft eventually embraced the user community as a growing development area and finally released their own software development kit (SDK) for VB.NET / C#.NET. The SDK allowed for 20+ body skeleton points to be tracked accurately in 3D space at 30 frame per second but did not include any facilities for gesture recognition (this had to be done manually by analysing the 3D points over time).

This emerging technology was used to develop early applications within the department including a project (Martyn Atkins / Marina Wimmer) to measure discus throwing responses (as 3D skeletal data) in the Xbox sports game. This employed two Kinect v1 sensors – one connected to the Xbox for the in-game play and another connected to a PC running a custom-written program developed by the Tech team which collected 3D skeletal data at 30Hz for post-experiment analysis.

Read more about the Kicking Action Response project

Kinect v2 Sensor

Kinect v2 SensorsWith the launch of the Xbox One, a new Kinect v2 sensor was launched to improve tracking accuracy and introduce a new integrated gesture detection engine via a new v2 SDK. With a total cost of £160 for a v2 sensor and adaptor kit for Windows PCs, you get a lot of powerful technology for a bargain price.

This powerful gesture detection technology has already been used to improve the original “discus throwing” experiment and has also been employed in a simple action-response football kicking game (Martyn Atkins / Kim Schenke) where participants respond to an on-screen actor kicking a ball at them by kicking the ball back or saving the ball with a block.

For more details on how this technology can be applied in your research, contact Martyn Atkins in the Psychology Technical Office, Link 109.

 Further Reading / Links

One Comment

  1. Pingback: Conference success | Psychology Tech Office

Comments are closed