Developing and establishing a natural user interface based on Kinect sensor with artificial neural network

Abstract

In this study, a real-time human motion classification system is developed by using Artificial Neural Networks (ANN) with negative correlation learning (NCL), to control windows games and applications. The system uses two Microsoft Kinects to extract human motion features. The x, y, z values of human skeleton joints are obtained from two Kinects using Microsoft Kinect Library and Microsoft c#. The data obtained from these devices are processed using noise reduction, feature extraction and classification modules. Four feature vectors; hand shape, hand locations, hand movement and hand distance are extracted for every human action and histograms of these feature vectors are used for classification. Kalman filter is used for noise reduction. Hand shapes are located and extracted using skeleton hand joints data, Kinect dept camera image and Kinect RGB camera image. Hands feature vectors are extracted with moment invariant method. The neural network is used as a classifier. The test results show that the developed and established system can be successfully used in real time recognition of human motion. This system is flexible and open for future extensions.