Action Recognition Using Log-Covariance Matrices of Silhouette and Optical-Flow (Human Interaction Recognition)

Chapter 8 Human interaction recognition So far we have discussed how to recognize actions and how to detect action changes, based on the assumption that each action is performed by a single person. In practical scenarios, many actions actually involve multiple persons, such as kissing, hugging, handshaking, etc. We call such actions human interactions. In this chapter, we will study how to adapt our framework for recognizing human interactions, which is a challenging problem. This problem has begun to receive attention in the research community in the past decade. In Section 8.1, we review some related work in this area. In Section 8.2, we introduce the framework to solve this problem. Experimental results are shown in Section 8.3. 8.1 Overview of related work The problem of human action interaction recognition has received attention in research community only in the last 10 years or so. (Sato and Aggarwal, 2001; Oliver et al., 2000; Aggarwal and Park, 2004; Park and Aggarwal, 2004; Park and Aggarwal, 2000; Waltisberg et al., 2010). There are two types of human interactions one is usually interested in. The first one is closely related to the relative positions, velocities and trajectories of two persons, for example, one person is following another, two people meet coming from different directions, or one person passes by another person (Sato and Aggarwal, 2001; Oliver et al., 2000). All these interactions are composed of walking, running and standing. The only differences are due to different motion 82 83 trajectories, relative velocities and positions. The second type of human interactions involves more detailed information about an individual’s action, such as handshaking, hugging, pushing, etc (Aggarwal and Park, 2004; Park and Aggarwal, 2004; Park and Aggarwal, 2000; Waltisberg et al., 2010). Trajectories, velocities and positions are only secondary factors here, since each interaction is composed of different individual actions. In this thesis, we only focus on the second type of human interaction. Sato et al. (Sato and Aggarwal, 2001) proposed a method to recognize the first type of human interactions. It consists of two major components: trajectory extraction (human segmentation and tracking) and trajectory classification (interaction classification). The trajectory extraction process outputs human trajectories as the physical positions of humans in the scene over time by recognizing and tracking human images. The interaction classification process extracts some features, such as relative distance and velocity of each person, from the trajectory shape. Then, it selects the most likely behavior type in the training dataset based on feature similarity. Oliver et al. (Oliver et al., 2000) proposed a Bayesian system for modeling the first type of human interactions. They use a Kalman filter to track an individual’s position, velocity and coarse shape, and then model the relationship between individuals by using the coupled Hidden Markov Model (CHMM). In order to recognize the second type of interactions, Park et al. (Park and Aggarwal, 2004) developed a method using a hierarchical Bayesian network (BN). First, body parts are tracked and their poses are estimated at a low level of BN. The overall body pose is estimated at a high level of BN. The pose estimation results are then concatenated to form a sequence, and sequence classification is performed by a dynamic Bayesian network. The recognition of two-person interactions is expressed in terms of semantic verbal descriptions at multiple levels: individual body-part motions at low level, single-person actions at middle level, and two-person interactions at 84 the highest level. Park et al. (Park and Aggarwal, 2000) also proposed a method based on the nearest neighbor classifier applied to a parametric human-interaction model, which describes the interpersonal relationship. Waltisberg et al. (Waltisberg et al., 2010) proposed a hough-voting action recognition system that recognizes each individual’s action separately and then combines the classification results to obtain an action label of the interaction. 8.2 Proposed framework In previous section, we reviewed state-of-art methods for human interaction recognition. In this section, we propose a new framework, which involves individual action recognition followed by a combination of individual action labels. There are basically two approaches for human interaction recognition: the universal approach and divide-and-conquer approach. The universal approach treat interacting persons as a single object with multiple disconnected parts. The advantage of this approach is that it is simple and can directly fit into our previous action recognition framework in Chapter 3 without concerning how many moving persons are involved in the human interaction. However, as we will see, this approach would require more training samples to well represent human interactions. The divide-and-conquer approach, in contrast, divides video spatially to isolate individuals, analyze them separately, and fuse the results. This approach is more complicated since we need object tracking to isolate moving individuals and analyze each action separately. The advantage is that it needs less training samples to well represent human interactions compared with the universal approach. Consider the following example: a human interaction e.g. punching involves two individuals - an attacker who is hitting and a defender who is dodging. If we assume that there are 85 10 action variations for each individual, then the total number of possible realizations of punching is 10 × 10 = 100. In other words, the universal approach needs 100 training samples to well represent punching in this example. In contrast, the divide-and-conquer approach only needs 10 + 10 = 100 samples to well represent the interaction. In general, if there are multiple individuals involved in a human interaction and Mk denotes the number of training samples needed for reliably recognizing the k-th action in the interaction, then the number of examples of human interaction Q needed to reliable recognize the interaction would be N1 = k Mk . Since N1 is at least an order of magnitude larger than Mk , it is much more difficult to obtain sufficient training samples for interaction, compared to an individual’s action. This motivates us to recognize human interaction by dividing it into individuals and classifying each of them separately. Then, we only need Mk training samples to recognize each individual’s action, and thus the sufficient number of training samples for recognizing P human interaction is vien by N2 = k Mk , which is at least an order of magnitude less than N1 . Assume that we have divided a human interaction sequence into individual sequences. Each individual’s action label can be obtained based on our previous approach described in Chapter 3. Then, a human interaction’s label can be obtained based on individuals’ actions. Ideally, if each individual is correctly recognized, the interaction will be correctly recognized as well. However, in practice, an individual may be misclassified, which may make the decision fusion ambiguous. Consider the previous punching example. If the defender’s action is misclassified as handshaking, then the label combination of hitting and handshaking will make no sense. We can, of course, treat such confused decisions as misclassifications. The outcome is that our method will be sensitive to individual errors. In other words, any individual’s misclassification will lead to the misclassification of the interaction. A better alternative 86 is to estimate confidence measures for each individual, that reflects the confidence of each action class to be the best class. Then, in the decision fusion step, we can combine the confidence measures to obtain an interaction label. A hard fusion , i.e., combining labels, only make use of the best action class of each individual. On the contrary, a soft fusion, i.e., combining confidence measures, merges more information (not only the individual’s best-matched class but also the confidence of each action class), and can effectively resolve ambiguities occurring in hard fusion. The overall framework of our approach is as follows, as also shown in Fig. 8·1. 1. Divide a human interaction video into separate individuals by using an object tracking algorithm; 2. Estimate confidence measures for each individual that reflects the confidence of each action class to be the best class. 3. Combine the confidence measures to obtain the interaction label. Figure 8·1: Proposed framework for human interaction recognition. 8.2.1 Individual action recognition We now discuss how to compute the confidence measures given a query individual action. The distance between query video and each action class is apppropriate 87 to indicate the confidence of each class. The smaller the distance, the higher the confidence, and vice versa. Now, the question is how to define the distance between query video and each action class. Given a query video, we can partition it into overlapping segments. Then, for each query segment, we find its nearest neighbor segment among all training segments in action class k and record the NN distance. Finally, after obtaining NN distances of all query segments, we take the average and use this mean distance as the distance between query video and action class k, as shown in Fig. 8·2. More formally, let Ciq be the covariance matrix of ith query segment (there are N query segments in the query video) and let Ck,j be the covariance matrices of training segments in action class k. The mean distance dk between query video and each action class can be expressed as: dk = 1 X min ρ(Ciq , Ck,j ), N i j (8.1) where ρ is the affine-invariant Riemannian metric (3.2) or the log-Euclidean metric (3.4) between covariance matrices. So far we have assumed that all the segments have the same length (number of frames). Actually, segment length may include useful information to discriminate different actions. The fixed segment length can be treated as a special case of variable segment length. Suppose we have obtained a series of appropriate segment lengths for all training classes: L1 , L2 , · · · , LK (K is the number of action classes). Then, segment length of each class is Lk . In order to measure the distance between query video and each training class, we need to partition the query video into K segment sets according to K segment lengths. For the kth set of query segments, we can measure the distance between query video and the kth training class using (8.1), shown in Fig. 8·3. Ciq,Lk is the covariance matrix of N,Lk ith query segment based on segment length Lk , dN is the NN distance between i the ith query segment and its nearest neighbor in training class k, and dLk is the 88 distance between query video and kth training class. The use of variable segment length increases the flexibility of training video partitioning. The success of this method relies on the appropriate selection of segment lengths for each action class. However, it seems difficult to develop a systematic approach to find the optimal variable segment lengths for different classes, and we use a heuristic to manually adjust the segment lengths. The basic rule of our manual adjustment is that if class i and class j are confused in classification, the difference between their corresponding segment lengths should be increased. 8.2.2 Decision fusion Once we obtain the confidence measures of each individual in a human interaction video, the next step is to determine the action label of this interaction by fusing the confidence measures. We need to pay attention to two observations: • Individual action classes and human interaction action classes may not be exactly the same. Take the punching as an example again, the action dodging is not an interaction action. • Some combinations of individual actions are inconsistent with each other. For example, the combination of handshaking and punching is not a valid interaction. The first observation above suggests that we may need to introduce extra action classes for recognizing an individual action, e.g., dodging. The second observation suggests that we should only focus on those combinations of individual actions that make sense when performing the decision fusion. If there are K individual action classes, the possible number of combinations is K 2 , but the actual number of interaction that are consistent is usually much lower than K 2 . 89 The process of decision fusion is shown in Fig. 8·4. For each real human interaction, find its individual actions’ distance measures in the ranking lists. Then, the mean of these distances indicates the distance between the query interaction video and this interaction class. We select the interaction class that is closest to the query video. Our approach to human interaction recognition has unique properties, compared with our single-person recognition method. At the segment level, it measures the distance between a query segment and each training class, rather than determining the label of this segment. At the sequence level, it uses “soft fusion” rather than “hard fusion”(majority rule) to label the query video. The experimental results are presented in the next section. 8.3 Experimental results In order to test the performance of our method, we conducted some experiments on the so-called UT-interaction dataset. This dataset was used at the “High-level Human Interaction Recognition Challenge” in the ICPR 2010 Contest on Semantic Description of Human Activities (SDHA). The UT-Interaction dataset contains videos of continuous executions of 6 classes of human interactions: shaking hands, pointing, hugging, pushing, kicking and punching. Several participants in more than 15 different types of clothing appear in the videos. The videos are taken at the resolution of 720 × 480, 30fps, and the height of a person in the video is about 200 pixels. This dataset is challenging, because pair of individuals performs each action once. In other words, all actions in this dataset are non-repetitive. Therefore, we do not have a sufficient number of samples to represent each action. Additionally, we introduce one more individual action of dodging, which appears in kicking, punching and pushing. Thus, there are 7 individual actions in total. We tested the interaction recognition 90 performance based on both fixed segment length and variable segment length. The fixed segment length we chose was 20. We manually selected the variable segment lengths follows: 10,14,10,16,8,20,10. In our tests, we used feature covariance matrices based on optical flow, and LOOCV. Table 8.1 shows the confusion matrix based on fixed-length segments. We see that punching and pushing, kicking and pushing, hugging and handshaking are difficult to discriminate. Thus, we adjusted the segment lengths so that these actions’ segment lengths are quite different. Table 8.2 shows the confusion matrix based on variable-length segments, which improves the performance by more than 20%. Table 8.3 shows results from Waltisberg et al. (Waltisberg et al., 2010). The performance of our approach is in line with state-of-the-art methods today. hug kick point punch push handshake hug kick point punch push handshake Table 8.1: Interaction confusion matrix obtained by the proposed approach based on fixed length segments (SEQ-CCR = 61.67%). 9 4 0 3 0 0 1 6 0 0 0 0 0 0 5 1 2 1 0 0 1 5 0 0 0 0 1 1 5 2 0 0 3 0 3 7 91 point punch push 10 0 0 10 0 1 2 0 0 0 0 2 kick handshake hug kick point punch push hug handshake Table 8.2: Interaction confusion matrix obtained by the proposed approach based on variable length segments (SEQ-CCR = 85.0%). 0 0 7 2 0 0 0 0 0 6 0 0 0 0 2 0 10 0 0 0 0 0 0 8 hug kick point punch push handshake hug kick point punch push handshake Table 8.3: Interaction action confusion matrix obtained by the method of Waltisberg et al. (Waltisberg et al., 2010) (SEQ-CCR = 88.33%). 7 0 0 0 0 0 2 1 10 0 0 10 0 0 0 2 0 1 0 0 0 10 0 0 0 0 0 0 7 0 0 0 0 0 1 9 92 Figure 8·2: Computation of confidence measures based on fixed segment length. 93 Figure 8·3: Computation of confidence measures based on variable segment length. 94 Figure 8·4: Flow chart illustrating the process of decision fusion in human interaction recognition.