University:
Boston University
Course:
CAS CS 501 | Computer Science Practicum
Academic year:

2022
Views:

72

Pages:

27
Author:

anglintiomkd

HAND GESTURE RECOGNITION USING KINECT Contents 1. Introduction………………………………………………………………………….. 1 2. Review of literature………………………………………………………………….. 1 3. Methods……………………………………………………………………………….. 2 4. Experimental results…………………………………………………………………. 12 5. Conclusions…………………………………………………………………………… 13 6. References…………………………………………………………………………….. 13 7. Appendix……………………………………………………………………………… 15 List of Figures Fig. 1 Overall workflow 2 Fig. 2 Workflow of hand extraction 3 Fig. 3 Original depth map 4 Fig. 4 Rescalin 4 Fig. 5 Background elimination 5 Fig. 6 Extraction 6 Fig. 7 Parameters optimization of the Shi-Tomasi corner detector 7 Fig. 8 Median filtering of the image 8 Fig. 9 Hand contour 9 Fig. 10 Hand contour with approximation polygon 9 Fig. 11 Hand contour with detected convexity points 10 Fig. 12 Hand contour with filtered convexity points 11 List of Tables Table 1 Confusion matrix for performance evaluation 12 1 1. Introduction The Kinect 3-D camera, with its depth sensing capability, has given birth to various exciting projects in human-machine interaction. One potential application is in the “wet lab” environment where a technician with gloves on can enter numbers into a computer without physically touching the computer, thus avoiding the procedure of removing and putting on gloves. In this project, a robust algorithm, which can recognize various arrangements of fingers of both hands showing digits from 0 to 5 using a Kinect, will be developed. 2. Review of literature It has been proposed in the literature that covariance matching can serve as a robust and computationally feasible approach to action recognition.[1],[2] This approach, which involves computing the covariance matrices of feature vectors that represent an action, can potentially be useful in our 2-D hand gesture recognition problem as well. Nonetheless, this stochastic approach requires careful selection of features in order to achieve high classification rate. In addition to covariance matching, another approach has been proposed in the literature by Jmaa and Mahdi.[3] Instead of building a dictionary of covariance matrices, this approach is based on analyzing three primary features extracted from an image: location of fingers, height of fingers, and the distance between each pair of fingers. A histogram is used to represent the detected fingers in order to extract the features for digit recognition. By considering the three geometric features, this approach is able to return the appropriate hand-digit without precomputing a dictionary as in the case of covariance matching. Although covariance matching is much more flexible in that it can be used in a variety of recognition tasks, the approach requires careful selection of feature vectors and its computational cost is potentially high. On the other hand, the solution Jmaa and Mahdi proposed is limited to only hand-digit recognition. However, the implementation of this approach is relatively easier and requires less processing. In addition to the Jmaa and Mahdi method, there are numerous applications that use polygon approximation to detect convexity points for hand-digit 2 recognition.[4],[6] The combined approach will be more accurate, especially in hand-digit recognition system. 3. Methods Overall workflow The overall workflow of our project is composed of 5 steps. The figure below illustrates the flowchart. The first step is to setup the programming environment, including OpenNI, OpenCV, Kinect and Visual C++. Step two (image capture) through step five (hand digit recognition) corresponds to the program we developed. Figure 1 - Overall workflow 3 Image capture and extraction Kinect can capture depth information by projecting an infrared dots pattern and its subsequent capture by an infrared camera. With this feature, we can easily get the shape of hand while discarding other information. The approach is described below. The depth information that is captured will be converted into a gray scale image. The image does not contain any color information. When someone operates the Kinect, the person’s hands should be in the front, so that the Kinect can extract the hands by judging the depth. All these are in real-time. The steps are described below. Figure 2 - Workflow of hand extraction Original depth map The first step is to obtain the image from the depth camera. With the OpenNI and OpenCV, we can obtain the images in 640*480 resolution at 30fps. Each frame is converted into a matrix in uint8 gray scale. However, the raw image is barely recognizable due to low color contrast, and thus the whole image is dark. 4 Figure 3 - Original depth map Rescaling To improve the raw image, we rescaled it. By adjusting the scale factor, we make the sensitive range to be from the minimum depth to about 1 meter from the Kinect. When the user is operating the Kinect correctly, this maximum depth should reach some point at the user’s arm. The hand is in the sensitive range, so it’s in gray scale, but the body and the background will be considered as the same, which is the maximum of depth, and thus not visible. Figure 4 - Rescaling 5 Background elimination The white background and the body are not necessary for extraction, so we performed a digital-negative operation on the image to make the image more optimal for subsequent steps. The shadows around the hand and the body, due to the positional disparity between the RGB camera and the depth sensor, can severely affect the performance of the recognition, so we have to eliminate them. Based on the previous step, the shadow at this point is white, while the background is black. The gray part is the hand, which is in the sensitive range. We can eliminate the shadows by thresholding the gray level. Figure 5 - Background elimination Extraction To perform extraction, we have to improve our image further. We captured the top point of the palm of the hand, and kept the pixels which represent the depth up to 24 units from that point. Otherwise, they will be eliminated. All pixels in this range, will be converted into white (intensity thresholding). Finally we extracted a hand shape image for recognition. 6 Figure 6 - Extraction Corner detection The original approach to hand-digit recognition is to use corner detection to build the relationship between corners and hand-digit gestures. Two common corner detection algorithms have been investigated: Harris corner detector and Shi-Tomasi corner detector. In Harris corner detector, the sum of squared differences (SSD) between a window in the image and its shifted-version is first computed: which can be reduced to the following using Taylor expansion: Then the selection criteria score is computed by: In the above equation, A corresponds to the Harris matrix from the SSD, and k is the Harris detector free parameter. On the other hand, the selection score of the Shi-Tomasi 7 corner detector is computed by finding the minimum of the eigenvalues to the Harris matrix A: In this project, Shi-Tomasi detector was chosen for testing. By varying the threshold score and the minimum distance between corners, the detector was able to return just a single corner for each raised finger: Figure 7 - Parameters optimization of the Shi-Tomasi corner detector Although the Shi-Tomasi corner detector was able to return just one corner for each raised fingertip, there were multiple parameters that could potentially affect the output of the recognition. These parameters include the score threshold, the scaling factor of score, the block size of detector kernel, and the minimum distance between detected corners. Thus, we opted for another approach to the hand-digit recognition problem. 8 Convexity detection The final approach to the hand-digit recognition is convexity detection. This approach attempts to locate the convex points and concave points of the hand shape. Then according to the number of convex and concave points detected, a digit is predicted. This approach was chosen over the original approach because it requires less number of parameters, and it proves to be more robust in our application. The convexity detection approach consists of the following steps: 1. Pre-filtering of the image A median filter with a window size of 15 x 15 pixels was used to smooth the edges of the extracted image. This filtering operation can be written mathematically as: where f is the source image, g is the filtered image, and h is the uniformly-weighted median-filtering kernel. This step was necessary in order to reduce image noise and the number of unwanted convexity points. The corresponding OpenCV function is Figure 8 - Median filtering of the image 9 2. Tracing the hand contour

Hand Gesture Recognition Using Kinect

Related Documents

Recommended Documents

Description

Related Documents

Free up your schedule!

Take 5 seconds to unlock