KinectDTW – Kinect Gesture Recognition Open Source Project – ready for download!

Posted: July 30, 2011 in Dissertation, Kinect
Tags: , , , , , , , , , , , , ,

This evening I finally published the gesture recording and recognition project I’ve been working on. With the help of the Kinect community, especially a member who goes by the name of Rhemyst, we have produced a library which introduces developers to vector-based gesture recognition.

May of the approaches I’ve seen elsewhere use specific positional tracking to recognising gestures – i.e. tracking a hand and matching its movement profile against a series of coordinates or something. This is great, of course, and can actually offer very good recognition. But the Dynamic Time Warping approach is more flexible in that it can be very easily programmed by a novice. It’s great for rapid prototying and, with the help of the community, I hope this can grow into a production-capable recognition engine. It’s not quite there yet, though…

So what are you waiting for? Grab a copy of the first release of KinectDTW from Codeplex now!

kinectdtw.codeplex.com

Please share your recorded gestures and recognition parameters with the community so that we can all learn and benefit from your experience!

Steve

About these ads
Comments
  1. Owen Doonan says:

    I am a member of the Massachusetts State Assistive Technology Council and specialize in developing methodologies for assisting professionals and executives with significant disabilities to return to meaningful and satisfying work in my private consulting practice. I feel that Kinect is a significant tool with a high potential for enabling people with disabilities to enjoy a better quality of life when incorporated with other technologies.

    For instance, what if american sign language could be “gesture captured” and used to translate ASL into spoken language and used by an avatar to translate spoken language into ASL. Past this, is the potential of video cams visual recognition software to be worn and used by the visually impaired to see and interact with their environment ” like seeing a verbal discription as they walk along the sidewalk.

    Regards,
    Owen Doonan
    Chairman, THE-Group
    mentor-one@usa.net

    • rymixxx says:

      I love your thinking. I truly feel like Kinect is ‘version one’ of the new way of controlling computers, at least for the casual and (some elements of) the assistive audience. We’ve got a long way to go, sure but t’s a good start.

      I don’t know if you’ve seen this, but someone already did a good job with ASL and Kinect: http://kinecthacks.net/american-sign-language-recognition-using-kinect/ it uses a different framework from the one I used, but it’s definitely worth a look.
      Regards,
      Steve

  2. einstein009 says:

    I really want to know detail what u make, u have any post in details….give me some links….i really like this project, i need to build a project like you but recognizing only hand gesture…..

  3. [...] di gesture: non è ancora implementato, esiste una libreria open source che lo fa (forse questa, ma è una cosa sulla quale progettano di lavorare a breve, perchè in sostanza l’SDK di [...]

  4. mauliksoni says:

    I found following problems with this toolkit -
    1) in real world, you can not ask user to record the gesture and then recognize it.
    2) even i record my own gesture, and distance with kinect or room setup changes the toolkit fails to recognize the gesture accurately.

  5. jeffery says:

    I need to do what owen is doing any chance I will be able to capture american sign language without a 100 lines of code? Because i want the gesture to be saved to file under 32 fps because its hard to get 32 fps gesture by myself and then compare against the gesture their doing dynamically or save it to file also. Can you explain to me how to go about such a task.
    The goal is to compare the two recorded gestures and then use a tts(text-to-speech) engine to speak the word their using in sign language. Trouble is i would have to record the initial sign language gesture to file then compare against what their doing.

    • Owen Doonan says:

      I envisioned video capture and gesture translation using a library of asl gestures approach that would convert to text and then using standard text to speech software or braille software convert to an accessible format. The use of text reduces the need for large audio files.

      The second translation of speech to asl in order to hold a conversation would reverse use of the library to display an avatar gesture that translates the speech to text “Dragon Dictate etc” or text input to the respective gesture output for the avatar to render.

      The library could be developed using asl interpreters to input gestures in a learning mode for capture and cataloging to words and phrases. I see the process as a bit like teaching “Dragon” to recognize speech so it can be outputed to plain text. By using several translators and a traditional asl alphabet that could provide a wider gesture response for a given word or phrase. I hope this helps you in envisioning the process a bit better. I originally was looking at video capture before Kinect came along so my view was to simply video capture a library of real time translation and pair it to the traditional symbols.

      Thanks for caring;-)
      Owen

  6. Quora says:

    Is there any hand gesture recognition implementations that can work under various light intensities as well as complex background?…

    Complex backgrounds in general are not an issue: complex means more textured, more textured means easier to discern from the background. The worst case scenario for many vision algorithms is uniform background.  Low light conditions are a more difficul…

  7. Celine says:

    Hi , I am trying with your code . I would like to check with you , where can I download the reference for microsoft research kinect ?

  8. jeffery says:

    I found out that kinect sdk v1.5 will allow recording sensor data so maybe when can record sensor data to hard drive and into database for initial tests with what word were detecting and presto were done. That would be faster but the person currently signing would be compared against a database and I have a good idea of how to do this when I see the help docs for v1.5 commerical sdk coming out in may.

  9. jeffery says:

    Heres my project page: http://kinectsignlanguage.codeplex.com just follow it or send me a message to add you because im almost done with my kinectmultipoint project and when I get free time will focus in on it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s