Action Recognition Using Log-Covariance Matrices of Silhouette and Optical-Flow (Action Change Detection)

Chapter 7 Action change detection So far we have studied action recognition based on the assumption that each video only contains a single action class. In practical cases, however, a given a video sequence may contain multiple action classes. For instance, a walking pedestrian may stop to hug a friend, and then start running towards an approaching bus. This example involves three actions, and if we parse the whole video into three sub-videos (single action in each sub-video), we can employ our proposed approach to get the three action labels of the whole video. This temporal parsing can be achieved by identifying time instants when an action changes, i.e., detecting when one action stops and another action begins. Finding these temporal action boundaries is akin to scene cut detection in video, but in the space of actions. The problem is fundamental to action analysis; even if actions can be reliably described (i.e., action representation is established) and even if they can be accurately compared (i.e., action comparison metric is known), it is still unclear to what segment of a video should both be applied. In other words, should the action representation be applied to 5, 10, 20 or more frames? Action change detection establishes temporal boundaries between which no action change takes place and thus action representation is meaningful. This is useful when comparing an observed action with a dictionary of annotated actions, but can also be used for unsupervised action segmentation, i.e., identification of actions as A, B, C, ..., that can be later annotated by a human operator. The detection of abrupt changes is a classical topic that has been widely stud72 73 ied in the past few decades (Basseville and Nikiforov, 1993). It has been applied in many areas, such as quality control, time series signal analysis, fault detection and monitoring, etc. Many algorithms have been developed for abrupt change detection, such as the cumulative sum algorithm (CUSUM), statistical algorithms, Shewharts control charts, etc. In this thesis, we apply a statistical algorithm to detect abrupt action changes. We develop a sequential, adaptive, and unsupervised action change detection algorithm based on our action representation framework, centered around covariance descriptors of local features. The main contribution of this work is the use of a non-parametric statistical framework to learn the distribution of the distance between covariance descriptors and detect action changes as covariance-distance outliers. 7.1 Framework The goal of action change detection is to partition a video into many sub-videos so that each of them contains only one single action. This may be viewed as finding temporal action boundaries inside a video, as shown in Fig. 7·1. To reliably detect action change one must assume that the action persists for some time, i.e., actions don’t keep changing at an arbitrarily fast rate. We make the assumption that actions persist for at least M frames. The main idea here is as follows. Given M frames as the dictionary of an action class, we examine the subsequent segments. If a segment is not similar to any of the segments in the dictionary, we claim that a new action class is detected (Guo et al., 2010a). The key problem here is how to measure the similarity between a segment and the dictionary. A straightforward idea is to find the nearest neighbor of this segment in the dictionary and use the NN distance as the measure of similarity. However, the issue is that an absolute threshold on the NN distance does not capture the extent of its variability 74 A c t b i o o n u c n d h a a r n i e g e s Figure 7·1: Illustration of action change boundaries. within the same action class and would therefore not be a reliable outlier detection method. In order to solve this problem, we learn the statistical characteristics of the NN distance. If the new segment’s NN distance corresponds to a low probability, it indicates that this segment comes from a new action class. The overall algorithm of action change detection is summarized as follows and illustrated in Fig. 7·2: 1. Learn the probability density function (pdf) of the NN distance between all pairs of segments in the dictionary of M training frames; 2. Compute the NN distance between a new segment and the dictionary. If it corresponds to a high probability, examine another segment by repeating this step. Otherwise, declare the detection of an action change, designate the next M frames for training and go back to the first step. 75 Figure 7·2: Action boundary detection approach. 7.1.1 Training phase In the training phase, our goal is to learn the conditional pdf p(dN N |H0 ) of the NN distance between a segment and dictionary, under the null hypothesis H0 that this segment has the same action class as training segments in this dictionary. Let F := {F1 , . . . , Fk } denote the set of k overlapping N-frame-long training segments forming the M-frame dictionary. For each training segment Fi we calculate dN Ni the nearest distance of Fi to the remaining segments in the dictionary. dN Ni := min j=1,...,k,j6=i ρ(Fi , Fj ). (7.1) Repeating this procedure k times provides k dN Ni (7.1) values which can be used to estimate p(dN N |H0 ) using a Parzen window kernel density estimation (KDE): k p(x|H0 ) := 1X φ(x − dN Ni ). k i=1 where φ(x) is a non-negative kernel function that integrates to one. (7.2) 76 7.1.2 Test phase If Ft denotes a test action segment, we firstly compute its NN distance dtest N N to the dictionary: dtest N N := min ρ(Fi , Ft ). i=1,...,k Since p(dN N |H0 ) has been estimated, we can obtain the conditional probability of dtest NN under the null hypothesis: p(dtest N N |H0 ). We can claim a detection of action change test if p(dtest N N |H0 ) is low. If the test segment is very similar to the dictionary, i.e., dN N is very small, clearly there should be no action change. However, small values of test dtest N N are also typically associated with small p(dN N |H0 ) values, which clearly do not correspond to the presence of an action change. This may happen if distances similar to dN N test occurred infrequently during training. In order to solve this problem, we use the cumulative probability function P (dN N |H0 ) instead of p(dN N |H0 ): P (dN N |H0 ) := d ZNN p(x|H0 )dx. (7.3) 0 We use a binary hypothesis test to decide if the test action segment is within or outside the limits of normal variability by comparing P (dtest N N |H0 ) against a confidence level. If it is greater than the confidence level, then we declare an action change. 7.2 Experimental results In order to test the performance of the proposed method, we conducted a series of experiments both on ground-truth synthetic data as well as a time-continuous cameracaptured video. In each video, we applied the following pre-processing steps. First, we performed background subtraction (McHugh et al., 2009) to obtain moving object silhouettes (silhouette tunnels). We applied our action change detection algorithm with M = 30, i.e., we assumed a 30-frame training set. A judicious selection of this 77 parameter is essential for the performance of our algorithm; it must be long enough to capture salient characteristics of an action, but short enough to span a single action only. The precision of detected action boundaries depends on the length of action segments. Clearly, a large segment length increases the uncertainty of action boundary location. In our experiments, we used segments of length N = 8 for action comparison with a 4-frame overlap. We examined different confidence levels (0.600.95) in the hypothesis test. In the ground-truth experiment we used the Weizmann dataset used earlier in 6.1 (Fig. 7·3). In order to measure the performance of our approach, we created 9 singleperson multi-action test video sequences (jumping-jack, jump, sjump, run, side, skip, walk, wave1 and wave2) with exactly known action boundaries, by concatenating all the action videos in the database belonging to the same individual leaving out those action videos which have less than M = 30 frames. There are two types of action Figure 7·3: Action samples of the concatenated video sequence from Weizmann dataset. detection errors: (i) false positive errors which occur when segments that have no action changes are classified as having action changes and (ii) false negative errors which occur when segments that have action changes are classified as having no action changes. The combined total number of action segments across all 9 test video 78 sequences is 597 out of which 61 segments have action changes and 536 do not. The proposed action change detection method produces 1 false negative error (Percent false negative error PF N = 100/61 = 1.64%) and 1 false positive error (Percent false positive error PF P = 100/536 = 0.19%) for a confidence level of 0.9. The values of PF N and PF P change by less than 1% for a range of confidence levels from 0.6–0.95. Fig. 7·4(a) shows the nearest-neighbor distances between each test segment and all segments in the immediately-preceding training set for one individual. Note that the detected change-points always correspond to large spikes in the distance measure, indicating a dissimilarity with respect to the training set. However, the nearestneighbor distances for the detected change-points vary from 2.1 to 3.8, and this range is different for different video sequences. Finding an optimal threshold to detect the action change-points by thresholding the nearest-neighbor distance would be quite challenging. In contrast, it is clear from Fig. 7·4(b) that the cumulative probability of the nearest-neighbor distance is a more robust quantity to base the action change detection on; it adaptively adjusts the scale of nearest-neighbor distances. In the second set of experiments, we tested our algorithm on a video sequence of single person performing 7 different actions (walk, jumping-jack, sjump, wave1, wave2, crouch and run) with natural transitions between consecutive actions, that we captured ourselves (Fig. 7·5). Although there are no ground-truth boundaries available, we subjectively identified the transitions between actions. As shown in Fig. 7·6, our algorithm has detected action change in 6 segments with index numbers: 7, 53, 92, 119, 157 and 196, exactly coinciding with the subjective transitions we had found. 79 Figure 7·4: Nearest-neighbor distances (top) and corresponding cumulative probability values (bottom) for consecutive action segments from ground-truth, multi-action video sequence built from Weizmann Human Action Database for “Daria”. Ground-truth actions are shown schematically above the top plot. The circles mark correctly-identified action boundaries (center of a segment in which action change takes place), while the squares mark erroneous boundaries. 80 Figure 7·5: Action samples of a camera-captured video sequence. 81 Figure 7·6: Nearest-neighbor distances (top) and corresponding cumulative probability values (bottom) for consecutive action segments from time-continuous, camera-captured video. Ground-truth actions are shown schematically above the top plot. The circles mark correctlyidentified action boundaries (center of a segment in which action change takes place).