Action Recognition Using Log-Covariance Matrices of Silhouette and Optical-Flow

ACTION RECOGNITION USING LOG-COVARIANCE MATRICES OF SILHOUETTE AND OPTICAL-FLOW ACTION RECOGNITION USING LOG-COVARIANCE MATRICES OF SILHOUETTE AND OPTICAL-FLOW ABSTRACT Algorithms for recognizing human actions in a video sequence are needed in applications such as video surveillance and video search and retrieval. Developing algorithms that are not only accurate but also efficient is challenging due to the complexity of the task and the sheer size of video. In this thesis, we develop a general framework for compactly representing, quickly comparing, and accurately recognizing actions using empirical covariance matrices of features. With each pixel we associate a feature vector which provides a localized description of the action. This generates a spatio-temporally dense collection of action feature vectors. We use the empirical covariance matrix of this feature vector collection as a low-dimensional representation of the action. We use two supervised learning methods, the nearest-neighbor classification and sparse-linear approximation classification, for action recognition using labeled training dictionaries of action co- variance matrices. Common to both methods is the novel idea that classification algorithms that have been developed for vectors can be re-purposed for covariance tensors by using a log-nonlinearity to map the convex cone of covariance matrices to the vector space of symmetric matrices. We illustrate the approach on two types of action feature vectors. One is based on silhouette tunnels of moving objects, and the other is based on optical flow. Action feature vectors of the first type describe the shape of the silhouette tunnel. Action feature vectors of the second type describe various motion characteristics such as velocity, gradient, and divergence. We demonstrate state-of-the-art recognition performance for both types of action feature vectors on the Weizmann, KTH, YouTube and the low-resolution ICPR-2010 challenge data sets under modest CPU requirements. We also demonstrate how our approach can be used for sequentially detecting changes in actions in an adaptive, unsupervised manner so as to parse a long video into sub-videos, each containing only a single action class. We use a non-parametric statistical framework to learn the distribution of the nearest-neighbor Riemannian distances between feature covariance matrices of video segments. Then, we use binary hypothesis testing to determine if new video segments include action changes. Our algorithm can detect 98.36% of action boundaries with 0.19% false alarm rate. We conclude by discussing how our framework can be adapted to recognize human interactions, which is usually a more challenging problem due to occlusion between moving individuals. We develop an approach based on dividing human interactions into separate sequences, each containing a single individual, and then combining the estimated action likelihoods for each individual sequence. The excellent performance of log-covariance-matrix representation combined with sparse-linear approximation classification demonstrated here for action recognition should encourage the use of this framework for other recognition problems. 1 Chapter 1 Introduction The proliferation of network cameras in the last decade has led to surveillance video overload. The volume and complexity of generated visual information far exceeds the capacity of human operators to manage, analyze and respond in real-time. It is, therefore, critical to develop efficient and effective automatic methods to analyze video data with the goal of understanding the visual environment. Of the many facets of video analysis, action recognition stands out as particularly important, since it can lead to many practical applications in such areas as video surveillance, video search and retrieval and human-computer interaction. This thesis concentrates on the problem of action recognition, which is defined as follows: given a dictionary of annotated training action videos, recognize the unknown action of a query video. The problem is illustrated in Fig. 1·1. We would like to recognize the action “walking” based on the prior knowledge of several actions. This chapter contains a general introduction to the problem and motivation for this thesis. First, we discuss some practical issues of action recognition. Some interesting applications follow. Finally, we describe our contributions and present outline of the thesis. 1.1 Action recognition challenges Despite significant efforts by the computer vision and image processing communities, recognizing actions in video is still a challenging problem on account of the following 2 Figure 1·1: Illustration of action recognition: input is a query video with an unknown action, output is an action label of this query video. issues: • Scene complexity: many videos are captured under uncontrolled conditions that may involve clutter and occlusions, as shown in Fig. 1·2. Given a video with a number of persons performing various activities, it is hard to track each individual and recognize his/her action. Occlusions, by hiding part of a moving object, may also hinder effectiveness of action recognition, if discriminative information of an action is missing from the video. • Acquisition complexity: in many situations, a camera’s field of view (FOV) is not fixed during video recording. A camera which is panned, tilted or zoomedin/-out can make a truly static background in real life appear to be “moving“ in the video. Although humans can easily distinguish a truly moving foreground from a “moving” background based on experience, computers are not intelligent enough to do so. Therefore, action recognition may be severely affected by nonstationary background in videos. Another acquisition complexity results from 3 the dependence of object appearance on viewpoint. An action captured from different viewpoints may look significantly different, as shown in Fig. 1·3. • Action complexity: there exists intra-class motion variability and inter-class motion ambiguity. No two individuals perform the same action in exactly the same manner. On the other hand, different actions may look similar in some particular poses, such as jumping and skipping. Figure 1·2: Example of a video frame captured under uncontrolled conditions that involves clutter and occlusions. Figure 1·3: The action “walking” looks quite different from different viewpoints. 4 Due to the practical issues discussed above, the focus of this research is on the key sub-problem in which the video footage only involves actions from a single moving object without significant occlusions. Such footage may be obtained by detecting, tracking and isolating object trajectories, although in some scenarios it may be nontrivial. In addition to the three sources of complexity described above, the volume of video data is an issue. Discriminative representations that are amenable to rapid processing (to enable close to real-time operation) and have a low storage cost are needed. 1.2 Applications Action recognition can lead to interesting applications in many areas, such as: • video surveillance, • sports and entertainment, • wildlife monitoring, • human computer interaction. We discuss some of these applications in detail below. 1.2.1 Video surveillance Video surveillance is an important tool to enhance public safety and privacy protection. It has long been deployed in public places such as airports, train stations and city centers. An example of surveillance cameras and monitors is shown in Fig. 1·4. Millions of cameras have been mounted in large cities, such as London, Chicago and New York. Those cameras produce endless video streams, while only a tiny fraction of them can be processed. Since most of the surveillance videos are not critical, one just wants to discover unusual actions from videos. If each video camera embeds an 5 Figure 1·4: Surveillance cameras and monitors. algorithm that can distinguish unusual actions and only transmits the relevant videos, it can save a great deal of bandwidth and human labor. 1.2.2 Sports and entertainment Figure 1·5: Examples of sports and entertainment that can use action recognition. Action recognition may be a useful tool to understand the content of sports and entertainment videos (Fig. 1·5). Today, with the popularity of video websites (YouTube, MegaVideo, etc.), it is quite easy to share sports and entertainment videos with friends and family. Sometimes we are interested in searching videos that are similar to our 6 own videos. Currently, this is based on metadata matching. However, labeling videos to produce metadata is time-consuming and tedious. Actions are usually the most important information in videos, and therefore a better video retrieval technique may be based on action matching. A new video search engine can be developed where input is a query video and output is its matched video that has most similar action to that in the query video. 1.2.3 Wildlife monitoring Figure 1·6: Wildlife monitoring example. Action recognition includes human actions as well as animal actions. Researchers who study wildlife behavior usually cannot stay in animal habitats for a long time. Also, the presence of a human may affect animals’ behavior. Video cameras can replace humans and help better understand wildlife behavior. An intelligent camera can be triggered to record videos when an animal’s movement is detected. Based on action recognition, perhaps different animals can be even distinguished. 1.2.4 Human-computer interaction Action recognition can also be used for human-computer interaction. Microsoft has developed a sensor called Kinect that can provide “controller-free gaming and entertainment experience” (Fig. 1·7). It enables users to control and interact with Xbox 7 Figure 1·7: Kinect sensor and human-computer interaction. 360 without using a physical controller but instead by using gestures and voice. Perhaps in the coming years, a new Windows operating system will have a more natural integrated user interface instead of traditional keyboard and mouse interface. 1.3 Thesis goals and contributions Our goals in the thesis can be summarized as follows: • develop a systematic framework for action recognition, including action representation and classification; • propose an action change detection algorithm; • study the human interaction recognition problem. 1.3.1 Action recognition framework The accuracy and efficiency of an action recognition method critically depends on: • how to model and represent actions; • how to classify actions. In the thesis, we develop a novel method for action representation, based on covariance matrix of a bag of local features. Covariance matrix is a compact representation 8 since it extracts the second order statistics of local features and lies in a much lower dimensional space than local features themselves. The selected features need to include discriminative properties for action classification. We implemented two types of local features, those based on silhouette tunnel and those based on optical flow. Silhouette tunnel describes the shape of an action and optical flow describes the motion dynamics of an action. Both of them include important characteristics for classifying human actions. Action recognition can be considered as a supervised learning problem, in which we can determine the query action class based on the dictionary of labeled action samples. In this thesis, we recognize an action using two classifiers: the nearest neighbor (NN) classifier and the sparse linear approximation (SLA) classifier. The NN classifier has been widely used in many supervised learning problems since it is simple, effective and training free. The SLA was proposed by Wright et. al to recognize human faces (Wright et al., 2009). The classification is based on sparse linear approximation of a query sample with respect to an overcomplete dictionary of training samples (base elements). 1.3.2 Action change detection Our action recognition framework assumes that each video sequence includes only one action class, which is not the case in many practical scenarios. Therefore, we need to parse a long video sequence into sub-sequences, each of which only includes a single action class. In this context, we develop a method that can identify time instants when an action change happens (e.g., from walking to running). In other words, we partition a video into many sub-videos so that each of them contains only one single action. We use the non-parametric statistical framework to learn the distribution of the nearest-neighbor distance between covariance matrices of video segments. Then, we use binary hypothesis testing to examine if a new video segment includes action 9 change. It is an unsupervised learning method that can discover temporal boundaries of actions without knowing action labels. 1.3.3 Human interaction recognition We also study how to recognize human interactions, such as kissing, handshaking, hugging, etc. It is a more challenging problem because: • partial occlusions often happen between moving individuals; • individual actions may not be sufficiently informative to help us fully understand the interaction. In this thesis, given an interaction video, we firstly divide it into individual actions using object tracking. Then, we estimate a confidence measure for each individual that reflects the likelihood of each action class. Finally, we combine the confidence measure to obtain the interaction label. 1.3.4 Summary of main contributions The main contributions of our work are summarized as follows. • We develop a new action recognition framework based on log-covariance matrices of bags of local action features; • We generalize the vector-based NN and SLA Classification algorithms to covariance matrices; • We discover discriminative features for action recognition from object silhouette tunnels and optical flow; • We extend the feature covariance matrix framework to the unsupervised action change detection problem; 10 • We develop a feature covariance based method for human interaction recognition. 1.4 Outline of the thesis The outline of the thesis is as follows: Chapter 2 reviews related work on action recognition. It categorizes past work with respect to action representation model and classification algorithm used. Chapter 3 introduces our proposed action recognition framework, including action representation and classification. Log-covariance matrices of features are used to represent actions. Both NN and SLA classifiers are employed for action classification. In NN classifier, two distance metrics for covariance matrices (affine-invariant Riemannian metric and log-Euclidean metric) are used instead of the Euclidean distance. Chapter 4 discusses two examples of action feature vectors, one based on silhouette tunnels and the other based on optical flow. Chapter 5 discusses practical implementation issues for our action recognition framework, i.e., how to process continuous video with limited memory, how to deal with temporal misalignment between training and query segments, etc. Chapter 6 shows experimental results for our action recognition framework, based on four benchmark datasets: Weizmann, KTH, UT-Tower and YouTube. This chapter also tests the robustness of our method to action variability and viewpoints, followed by computational complexity analysis. Chapter 7 proposes an algorithm to detect action changes in a video, based on a non-parametric framework and binary hypothesis testing. Experimental results are provided to show the effectiveness of our method. Chapter 8 is dedicated to human interaction recognition. The basic idea is to separate interaction into individuals, analyze each individual’s action and finally 11 obtain the interaction label by considering the most likely concurrence of individual actions. Chapter 9 discusses contributions of this thesis, draws conclusions and presents possible directions for future work.