Action Recognition Using Log-Covariance Matrices of Silhouette and Optical-Flow (Related Work)

Chapter 2 Related Work In this chapter we review related work in action recognition. There are basically two important issues in action recognition: action representation and action classification. Accordingly, the existing literature on action recognition, which is vast and rapidly growing, can be loosely grouped by the type of model used to represent actions and the type of classification algorithm used. 2.1 Action representation There are roughly five types of features for action representation that have been studied in the literature: shape-based features, motion-based features, geometric human body features, interest-point features and dynamic models. These are discussed in more detail in the following sections. 2.1.1 Shape-based features Shape-based action representation is widely used in action recognition today. A partial listing of papers using shape models includes (Bobick and Davis, 2001; Yilmaz and Shah, 2005; Gorelick et al., 2007; Chen et al., 2008; Guo et al., 2009; Wang et al., 2007; Ikizler and Duygulu, 2007; Zhang et al., 2009; Ahmad et al., 2010). This representation relies on accurately capturing the silhouette of a moving object at each time instant. A sequence of such silhouettes forms a silhouette tunnel, i.e., a spatiotemporal binary mask of the moving object changing its shape in time. A silhouette 12 13 tunnel is desirable for action recognition as it is invariant to luminance, color, and texture of the moving object as well as the background. Although silhouette tunnels do not capture motion inside objects, the moving silhouette boundary leaves a very distinct signature of occurring activity. An effective method based on silhouette tunnels was developed by Gorelick et al. (Gorelick et al., 2007). At each pixel, the expected length of a random walk to the silhouette tunnel boundary, which can be computed by solving a Poisson equation, is treated as a shape feature of the silhouette tunnel. An action classification based on this approach was shown to be remarkably accurate suggesting that the method is capable of extracting highly-discriminative information. Collins and Gross (Collins et al., 2002) have also used silhouettes to identify human actions, but their method extracts key frames from the query video sequence and matches them with key frames from training videos. The classification is performed by the nearest-neighbor rule based on normalized correlation scores. This method is conceptually simple and easy to implement, but it is based on 2-D silhouettes without considering the dynamics of silhouette evolution. The dynamic nature of video has been also exploited by Bobick and Davis (Bobick and Davis, 2001) who proposed a motion energy image (MEI), that represents where motion has occurred in an image sequence, and motion history image (MHI), that is a scalar field depicting how recently the motion occurred. Together, MEI and MHI act as a two-component version of a temporal template, and are compared with known actions in a database to determine the best action match. In our work, we use silhouette-based features that fall into the category of shapebased features. We extract various spatial and temporal distances that describe the shape of silhouettes and thus implicitly characterize actions. 14 2.1.2 Motion-based features Motion features extract dynamic characteristics of actions, which are some of the most discriminative attributes of actions. A partial listing of papers using motion models includes (Lowe, 2004; Ke et al., 2005; Danafar and Gheissari, 2007; Scovanner et al., 2007; Liu et al., 2008; Fathi and Mori, 2008; Ali and Shah, 2010; Seo and Milanfar, 2011; Wang et al., 2011). Recently, Ali et al. (Ali and Shah, 2010) proposed kinematic features derived from optical flow for action representation. Each kinematic feature gives rise to a spatio-temporal pattern. Then, kinematic modes are computed by performing Principle Component Analysis (PCA) on the spatio-temporal volumes of kinematic features. Multiple Instance Learning (MIL) is employed to recognize actions. The idea of MIL is to represent each action video as a collection or a “bag“ of kinematic modes in which each kinematic mode is referred to as an instance representing that video. Seo and Milanfar (Seo and Milanfar, 2011) used a 3D local steering kernel (3D-LSK) as an action feature that can reveal global space-time geometric information. The idea behind 3D-LSK is based on analyzing the radiometric (pixel value) differences based on estimated space-time gradients, and using this structure information to determine the shape and size of a canonical kernel (descriptor). They also employed Matrix Cosine Similarity (MCS) as a similarity measure between pairs of activity sequences, which generalizes the notion of cosine similarity between two vectors. Wang et al. (Wang et al., 2011) proposed an approach to describe actions by dense trajectories. They sample dense points from each frame and track them based on displacement information from a dense optical flow field. The dense trajectories well characterize actions and a descriptor based on motion boundary histograms is used to encode the trajectory information. In our work, we also use optical-flow features which fall into the category of motion-based features. The optical-flow features extract dynamic motion attributes 15 to represent actions. Some of our features have been successfully used for action recognition in Ali et al.’s work (Ali and Shah, 2010), such as divergence, vorticity, etc. 2.1.3 Geometric human body features Geometric human body model can be built with static and dynamic body parameters. They are mostly used in controlled environments where human body parts, such as legs and arms, are easy to identify. A partial listing of papers using geometric human body models includes (Rohr, 1994; Goncalves et al., 1995; Cunado et al., 2003; Wang et al., 2004; Xie et al., 2011). Rohr et al. (Rohr, 1994) incorporate a 1 DOF pose parameter to aid in model fitting. All the poses in a walking action are indexed by a single number. The work of Goncalves et al. (Goncalves et al., 1995) promotes three-dimensional tracking of the human arm against a uniform background using a two-cone arm model and a single camera. However, acquiring the three-dimensional information from videos is still a very complicated process. Xie et al. (Xie et al., 2011) proposed a pose-based approach for locating and recognizing human actions in video using a deformable body part model. 2.1.4 Interest-point features Interest points, e.g., corners, SIFT features (Lowe, 1999), etc., have also been employed to represent actions. A partial listing of papers using interest-point features includes (Dollar et al., 2005; Smith et al., 2005; Schuldt et al., 2004; Wong and Cipolla, 2007; Niebles et al., 2008; Laptev et al., 2008; Wu et al., 2011; Le et al., 2011; Kovashka and Grauman, 2010). Interest points are sufficiently discriminative and are usually sparse (far fewer interest points than the number of pixels in a video sequence). Niebles (Niebles et al., 2008) and Dollar (Dollar et al., 2005) use 2D Gaussian and 1D Gabor filters to select interest points in a spatio-temporal volume. Laptev 16 et al. (Laptev et al., 2008) utilize the Harris corner detector to locate local salient pixels with significant local variations in both spatial and temporal domains. Wong et al. (Wong and Cipolla, 2007) extract interest points by considering structural information and detecting cuboids in regions that have large probability of containing motion. Wu et al. (Wu et al., 2011) recently proposed a new spatio-temporal context distribution feature of interest points for human action recognition. Each action video is expressed as a set of relative XY T coordinates between pairwise interest points in a local region. Then, Gaussian Mixture Model (GMM) was adopted to model the distribution of context features for each video. Le et al. (Le et al., 2011) developed an extension of the Independent Subspace Analysis algorithm to learn spatio-temporal features of interest points from unlabeled video data. Their method performs well when combined with deep learning techniques such as staking and convolution to learn hierarchical representations. Kovashka et al. (Kovashka and Grauman, 2010) extracted local motion and appearance features, quantized them into visual vocabulary and then formed candidate neighborhoods that are very discriminative for a given action category. Recently, probabilistic latent semantic analysis and latent Dirichlet allocation have been applied to recognize actions. Distributions of spatio-temporal words and intermediate topics are learned corresponding to human action categories in an unsupervised manner. The algorithms are able to localize multiple actions in complex motion sequences. Wang et al. proposed an unsupervised learning framework to model activities and interactions in complicated scenes (Wang et al., 2009) by using hierarchical Bayesian models to connect low-level features, simple “atomic” activities and interactions. Three hierarchical Bayesian models were proposed: the Latent Dirichlet Allocation (LDA) mixture model, the Hierarchical Dirichlet Processes (HDP) mixture model and the Dual Hierarchical Dirichlet Processes (Dual-HDP). Wang and 17 Mori proposed a new “bag-of-words” representation that characterizes each frame as a single word (Wang and Mori, 2009). Additionally, they use the semi-Latent Dirichlet allocation (S-LDA) model and the probabilistic Latent Semantic Indexing (pLSI) for action recognition. 2.1.5 Dynamic models Dynamic models are among the earliest models used for human action recognition. A partial listing of papers using dynamic models includes (Yamato et al., 1992; Starner and Pentland, 1995; Raptis et al., 2010). The general idea is to define each static posture of an action as a state, and describe the dynamics (temporal variations) of the action by using a state-space transition model. An action is modeled as a set of states and connections in the state space using a Dynamic Probabilistic Network (DPN). Hidden Markov Model (HMM), the most commonly used DPN, has the advantages of directly modeling time variations of features of data. The parameters of the dynamic model can be learned from a set of training action videos. To classify an action, the joint probability with the maximum value is selected as the criterion for action recognition. Kale et al. (Kale et al., 2004) used an HMM to model human gait. Each individual is characterized with one HMM and several typical silhouettes are used as hidden states for HMM training. Brand et al. (Brand and Kettnaker, 2000) proposed a Multi-observation-mixture-counter HMM (MOMC-HMM) to factorize the observation space. Oliver et al. further proposed a Coupled HMM (CHMM) (Oliver et al., 2000) to model the temporal and causal correlations among hidden states. Another model, called Dynamically-multi-linked HMM (DML-HMM) was recently developed by Xiang and Gong (Xiang and Gong, 2006). Shi et al. (Shi et al., 2008) proposed a semi-Markov discriminative approach by employing a Viterbi-like column generation algorithm. 18 The models adopted in the literature have their own properties. Shape models are invariant to luminance, color, and texture of the moving object as well as background. The quality of silhouettes greatly affects the success of shape models. Motion models extract the most discriminative attributes of actions, but sometimes background motion can bias the motion models. Geometric human body models can work well in and only in the controlled environment. Interest-point models extract salient features related to moving pixels, but these features are usually sparse and sometimes inadequate to describe an action. Dynamic models can characterize an action very well, but it is complicated and time-consuming to estimate the model parameters. In this thesis, we will focus on shape models and motion models, since they are discriminative for action recognition and they are fairly easy to obtain. 2.2 Action classification In terms of action classification, algorithms from machine learning community have been heavily borrowed. Some action recognition methods are based on the NN classifier (Gorelick et al., 2007; Bobick and Davis, 2001; Dollar et al., 2005; Cunado et al., 2003; Wang et al., 2004; Seo and Milanfar, 2011; Liu et al., 2008; Lowe, 2004), a straightforward method that requires no explicit training. Other methods recognize actions by using support vector machine (SVM) (Ikizler and Duygulu, 2007; Ahmad et al., 2010; Schuldt et al., 2004; Danafar and Gheissari, 2007; Scovanner et al., 2007; Hoai et al., 2011). The objective of SVM is to maximize the separation margin between classes. The approach uses a kernel function to map the training samples to a high-dimensional feature space and finds an optimal separation hyperplane in this feature space. Recently Hoai et al. (Hoai et al., 2011) proposed a method that can jointly perform video segmentation and action recognition using multi-class SVM framework and dynamic programming. Another classification technique used for ac- 19 tion recognition is boosting (Zhang et al., 2009; Smith et al., 2005; Fathi and Mori, 2008; Ke et al., 2005) which improves performance of any classifier by combining a number of weak classifiers into a strong one. A discussion of various classifiers can be found in (Duda et al., 2001). In recent years, classification algorithms based on sparse linear representation, which have connections to the area of compressive sensing, have started receiving significant attention in the computer vision and pattern recognition communities (Wright et al., 2009; Mahoor et al., 2011; Mairal et al., 2008; Elad and Aharon, 2006; Yang et al., 2008; Mei and Ling, 2009; Ying et al., 2010; Concha et al., 2010). Successful applications of sparse representation include face recognition (Wright et al., 2009), facial expression recognition (Ying et al., 2010; Mahoor et al., 2011), object recognition (Mairal et al., 2008), image denoising (Elad and Aharon, 2006), super resolution image processing (Yang et al., 2008) and object tracking (Mei and Ling, 2009). Concha et al. (Concha et al., 2010) developed an approach for action recognition based on compressive sensing principles. In their method, feature vectors extracted from each frame are randomly projected into a lower-dimensional space, and the projected features are used in a classification algorithm based on a Hidden Markov Model (HMM). Although compressive sensing and sparse linear representations share a common mathematical foundation, in the work of Concha et al., compressive sensing is mainly used for feature dimensionality reduction rather than classification based on the characteristics of a sparse representation as in (Wright et al., 2009; Mahoor et al., 2011; Mairal et al., 2008; Ying et al., 2010). In our work, we use the NN classifier and sparse representation classifier for action classification.