Action Recognition Using Log-Covariance Matrices of Silhouette and Optical-Flow (Conclutions and Future Work)

95 Chapter 9 Conclusions and future work This chapter summarizes this thesis, and proposes ideas for future work. The main inspiration for this work was the rapid development of content-based video analysis. In the past decades, many researchers have been working in video processing. Most of the work has focused on low-level processing, such as motion detection, video segmentation, object tracking, etc. Although low-level processing can detect motion and isolate moving objects, it cannot classify motion. Thus, it is desirable to develop automatic methods to analyze video data with the goal of understanding the visual environment. Action recognition is a content-based technique that can provide semantic information about a video. It can lead to many applications, e.g., video surveillance, video search and human-computer interaction. The action recognition task is made complex by the complexity of the scene (multiple interacting moving objects, clutter, occlusions, illumination variability, etc.), the camera (imperfections, motion and shake, and viewpoint), and the complexity of actions (non-rigid objects and intra- and inter-class action variability). 9.1 Summary of contributions In this work, we studied a sub-problem of action recognition in which there is only one action in a video performed by an individual. We proposed a systematic action recognition framework, including action representation and action classification. An action is represented by log-covariance matrix of a bag of local features. These features 96 are associated with moving pixels, providing a coarse localized description of the action. Feature covariance matrices provide a compact representation since they extract second order statistics of local features and lie in a much lower dimensional space than the local features. In order to be effective, the selected local features should involve discriminative properties of motion dynamics. There are two examples of local features in this thesis, those based on silhouette tunnel and those based on optical flow. Silhouette tunnel describes the shape of an action and optical flow describes the motion dynamics of an action. Both of them include important information for classifying human actions. In action classification, we used two supervised learning classifiers: the nearest neighbor (NN) and sparse linear approximation (SLA) classifiers. In the SLA classifier, we approximate the logarithm of a query action covariance matrix by a sparse linear combination of the logarithm of training action covariance matrices using a fast linear program and determine the action label from sparse coefficients. Common to both classifiers is the novel idea that classification algorithms that have been developed for vectors can be re-purposed for covariance tensors by using a log-nonlinearity to map the convex cone of covariance matrices to the vector space of symmetric matrices. We also demonstrated how our approach can be used for sequentially detecting changes in actions in an adaptive unsupervised manner so as to parse a long video sequence into sub-sequences, each of which only includes a single action class. Then, we can employ our proposed approach to get action label of each sub-sequence. We used a non-parametric statistical framework to learn the distribution of the nearest-neighbor Riemannian distances between feature covariance matrices of video segments. Then, we used binary hypothesis testing to determine if new video segments include action changes. 97 We also proposed a method how to recognize human interactions, which is usually a more challenging problem than single action recognition. Given an interaction video, first we spatially divide it into individual actions using object tracking. Then, we estimated a ranked list for each individual. Each entry in this list reflects the chance of each action class to be the best class. Then, in the decision fusion step, we combined the ranked lists to obtain the interaction label. We tested the performance on UT-Tower interaction dataset, and our method turned out to perform very closely to state-of-the-art algorithms. Overall the contributions of this thesis can be summarized as follows: • We developed a new action recognition framework based on log-covariance matrices of bags of local action features; • We generalized the vector-based NN and SLA Classification algorithms to covariance matrices; • We discovered discriminative features for action recognition from object silhouette tunnels and optical flow; • We extended the feature covariance matrix framework to the unsupervised action change detection problem; • We developed a feature covariance based method for human interaction recognition. 9.2 Future work This section briefly describes a few directions that could extend our work. 98 9.2.1 Alternative features In this thesis, we introduced two examples of local features based on silhouette tunnel and optical flow. However, there may exist more discriminative features that can better represent motion dynamics. Recently, we have adapted our action recognition framework to perform real-time gesture recognition using the Kinect camera from Microsoft. Kinect provides robust and reliable skeleton model of a person in front of it, and we used parameters of this model as features for gesture recognition. Our action recognition framework proved very useful even in this very simple scenario of a skeleton model. Clearly, our action recognition framework is general, and we believe that the performance of our approach can be further improved if better features are judiciously selected. 9.2.2 Group action recognition Here, we primarily focused on the problem in which there is one or two moving persons in a video. In practice, a camera usually captures several people simultaneously. Due to occlusions and low resolution of individuals, it is difficult to recognize a group action. Additionally, it is also challenging to recognize if there is an unusual action in a crowded scene. These are practical issues that arise from video surveillance applications, which are worthwhile to study and resolve. 9.2.3 Extension of current assumptions In Chapter 1 we listed challenges associated with action recognition and made some assumptions in order to make our problem tractable. However, some assumptions are not practical and make our approach difficult to adapt to real-life scenarios. Thus, in the future, action recognition should be studied without constraining assumptions, such as clutter, occlusions, camera motion, zoom-in/zoom-out and camera viewpoint change. All these are practical issues that prevent us from accurately recognizing 99 actions. The datasets we have focused on are limited. In the future, it would be desirable to work on more challenging datasets, such as the VIRAT dataset (Oh et al., 2011). This dataset is designed to be realistic, natural and challenging for action recognition in terms of camera motion, background clutter and diversity in the recorded scenes. There are frequent incidental movers and background activities in videos. This is a very challenging dataset that relaxes most of our assumptions and thus requires more effort to achieve satisfactory action recognition performance. 9.2.4 Action recognition via static scenes Our approach has only studied action recognition by focusing on the moving person. Sometimes, the scene and individual objects in videos can also help understand actions. For example, if we can recognize a swimming pool in a video, it is likely that persons in this video are swimming. Action characteristics are not the only factor that can help us recognize it. However, recognizing scene or objects in a video is still very challenging, perhaps even more challenging than recognizing actions. This approach may be useful in scenarios where we can easily recognize scenes and objects.