Action Recognition Using Log-Covariance Matrices of Silhouette and Optical-Flow (Experimental Results)

Chapter 6 Experimental results In previous chapters, we introduced the overall approach for action representation. Specifically, we used two different features based either on a silhouette tunnel or optical flow to represent action dynamics. Then, NN and SLA classifiers were respectively employed for action classification. Thus, there are four possible implementations (2 types of feature vectors × 2 types of classifiers) to recognize actions. We evaluate their performance on four publicly-available datasets: Weizmann (Gorelick et al., 2007), KTH (Schuldt et al., 2004), UT-Tower (Chen et al., 2010) and Youtube action datasets (Liu et al., 2009). Our performance evaluation was based on leave-one-out cross validation (LOOCV), as illustrated in Fig. 6·1. First, we divided each video sequence into N-frame long overlapping action segments. Then, we selected one of the action segments as a query segment and used the remaining segments as the training set (except those segments that came from the same video sequence as the query segment). Finally, we identified action class of the query segment. We repeated the procedure for all query segments in the dataset and calculated the correct classification rate (CCR) as the percentage of query segments that were correctly classified. We call this rate the segment-level CCR, or SEG-CCR. In practice, however, one is usually interested in classification of a complete video sequence instead of one of its segments. Since segments provide time-localized action information, in order to obtain classification for the complete video sequence we employed the majority rule (dominant label wins) to all segments 49 50 in this sequence. This produces a sequence-level CCR, or SEQ-CCR, defined as the percentage of query sequences that are correctly classified. Figure 6·1: Illustration of leave-one-out cross validation. We also tested our method using “leave-part-out” cross validation (LPOCV), a more challenging test sometimes reported in the literature (Fig. 6·2). In LPOCV, we divided the action segments into non-overlapping training set and test set. After selecting a test segment from the test set, we assigned a class label based on the training set. We repeated this procedure for all test segments. The main difference between LPOCV and LOOCV is that the training set in LPOCV is fixed with less training samples than in LOOCV if both are based on the same dataset. Thus, it is expected that LPOCV will attain poorer performance than LOOCV if other settings remain the same. 51 Figure 6·2: Illustration of leave-part-out cross validation. 6.1 Tests on the Weizmann dataset We conducted a series of experiments on the Weizmann Human Action Database available online1 (Gorelick et al., 2007). Although this is not a very challenging dataset, many state-of-the-art approaches report performance on it thus affording an easy comparison. The database contains 90 low-resolution video and silhouette sequences (180×144 pixels) that show 9 different people each performing 10 different actions, such as jumping, walking, running, skipping, etc. Some action examples are shown in Fig. 6·3. We first measure the performance of our action recognition method using silhouette features, and then using optical flow features. 1 http://www.wisdom.weizmann.ac.il/∼vision/SpaceTimeActions.html 52 Figure 6·3: Action examples from Weizmann dataset. 6.1.1 Silhouette features We tested the action recognition performance using NN and SLA classifiers. When we use NN classifier, we obtain SEG-CCR of 97.05% and SEQ-CCR of 100% for segment length N = 8. Table 6.1 shows the action “confusion” matrix based on SEG-CCR. The element in row i and column j of the matrix indicates the percentage of action i segments which were classified as action j. The sum of all elements in every row is 100%. The confusion matrix indicates that while some actions are more confusing, such as “skipping” and “jumping”, others are easier to distinguish, such as “bending” and “jumping-jack”. If we use the SLA classifier, we obtain SEG-CCR of 96.74% and SEQ-CCR of 100% for N = 8. Table 6.2 shows the segment-based action “confusion” matrix. In order to compare the efficacy of NN and SLA classifier, we compute their SEG-CCR and SEQ-CCR under different segment length (N = 8 and N = 20) and different cross validation methods (LOOCV and LPOCV), as shown in Table 6.3. In Table 6.4, we compare the proposed method with some recent action recognition algorithms that are based on LOOCV. The above experimental results indicate that: 53 bend jack jump sjump run side skip walk wave1 wave2 wave2 wave1 walk skip side run sjump jump jack bend Table 6.1: Segment-based action confusion matrix for the proposed method (Silhouette features + NN classifier) on 8-frame segments (SEG-CCR = 97.05%). 98.6 0 0 0 0 0 0 0 0 1.4 0 100 0 0 0 0 0 0 0 0 0 0 96.1 0 0 0 3.9 0 0 0 0 0 0 99.3 0 0 0 0 0 0.7 0 0 0 0 94.0 1.2 2.4 2.4 0 0 0 0 0 0 0 100 0 0 0 0 0 0 1.0 0 10.3 0 86.7 2.1 0 0 0 0 0 0 1.3 0 0 98.7 0 0 0 0 0 0 0 0 0 0 98.0 2.0 0 0 0 0 0 0 0 0 4.9 95.1 bend jack jump sjump run side skip walk wave1 wave2 wave2 wave1 walk skip side run sjump jump jack bend Table 6.2: Segment-based action confusion matrix for the proposed method (optical flow features + SLA classifier) on 8-frame segments (SEG-CCR = 96.74%). 91.9 1.3 0 0.7 0 0 0 0 4.1 2.0 0 99.4 0 0.6 0 0 0 0 0 0 0 95.1 0 0 2.0 2.9 0 0 0 0 0.8 0 96.7 0 2.5 0 0 0 0.7 0 0 0 1.2 91.6 0 1.2 6.2 0 0 0 0 0 0 0 100 0 0 0 0 0 0 1.0 0 4.2 0 92.7 2.1 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0.6 0 0 0 0 99.4 0 0 1.4 0 0 0 0 0 0 1.4 97.2 Table 6.3: Comparison of NN and SLA classifiers based on silhouette features for Weizmann dataset. NN classifier SLA classifier SEG-CCR SEQ-CCR SEG-CCR SEQ-CCR 97.05% 100% 96.74% 100% N=8 LOOCV LPOCV 90.88% 91.11% 91.35% 95.56% 98.68% 100% 99.49% 100% N=20 LOOCV LPOCV 91.82% 95.56% 93.61% 95.56% • the proposed framework for action recognition achieves remarkable performance (up to 97% SEG-CCR and 100% SEQ-CCR); 54 Table 6.4: Comparison of silhouette-based action recognition (N=8) using LOOCV on Weizmann dataset. Method SEG-CCR SEQ-CCR NN classifier 97.05% 100% SR-based classifier 96.74% 100% Gorelick 97.83% - Niebles 90% Ali 95.75% - Seo 96% Xie 95.6% Ikizler 100% • larger N leads to better classification performance; • using LPOCV results in lower CCR than using LOOCV since LPOCV has a smaller training set and accordingly is more challenging; • in most cases, the SLA classifier has similar or better performance compared to the NN classifier; • the majority rule improves the recognition performance. 6.1.2 Optical flow features In contrast to previous tests, we now use optical flow features to represent action dynamics, and either NN or SLA classifier. For the NN classifier, we obtain SEGCCR of 89.74% and SEQ-CCR of 91.11% (N = 8). Table 6.5 shows the action confusion matrix based on SEG-CCR. If we use the SLA classifier, we attain SEGCCR of 92.69% and SEQ-CCR of 94.44%. Table 6.6 shows the action “confusion” matrix based on SEG-CCR. Additionally, we compute their SEG-CCR and SEQ-CCR under different segment length (N = 8 and N = 20) and different cross validation methods (LOOCV and LPOCV), as shown in Table 6.7. 6.1.3 Variation of CCR in LPOCV In LPOCV we partition an action dataset into non-overlapping query set and training set, and then recognize actions in query videos. However, our previous results only show one particular leave-part-out partition, which may not be indicative of the 55 bend jack jump sjump run side skip walk wave1 wave2 wave2 wave1 walk skip side run sjump jump jack bend Table 6.5: Segment-based action confusion matrix for the proposed method (optical flow features + NN classifier) on 8-frame segments (SEG-CCR = 89.74%). 100 0 0 0 0 0 0 0 0 0 0 99.4 0 0.6 0 0 0 0 0 0 0 0 66.7 0 7.1 6.1 20.1 0 0 0 0 5.8 0 93.3 0 0.8 0 0 0 0 0 0 1.3 0 69.6 0 27.9 1.3 0 0 0 0 4.2 0 3.2 83.2 3.2 6.3 0 0 0 0 11.7 0 29.8 0 58.5 0 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 0 98.7 1.3 0 0 0 0 0 0 0 0 1.4 98.6 bend jack jump sjump run side skip walk wave1 wave2 wave2 wave1 walk skip side run sjump jump jack bend Table 6.6: Segment-based action confusion matrix for the proposed method (optical flow features + SLA classifier) on 8-frame segments (SEG-CCR = 92.69%). 98.4 0 0 0 0 0 0 0 1.6 0 0 100 0 0 0 0 0 0 0 0 0 0 71.7 0 1.0 4.0 19.2 3.0 0 1.0 0 0.9 0 99.1 0 0 0 0 0 0.7 0 0 1.3 0 82.3 1.3 8.9 5.0 1.2 0 0 0 2.1 0 0 96.9 0 1.0 0 0 0 0 11.2 0 12.8 0 74.0 2.0 0 0 0 0 0 0 0 0 0 100 0 0 3.3 0 0 0 0 0 0 0.7 94.0 2.0 0 0 0 0 0 0 0 0 0 100 Table 6.7: Comparison of NN and SLA classifiers based on optical flow features for Weizmann dataset. NN classifier SR-based classifier SEG-CCR SEQ-CCR SEG-CCR SEQ-CCR 89.74% 91.11% 92.69% 94.44% N=8 LOOCV LPOCV 79.45% 80.00% 83.20% 88.89% 91.93% 92.22% 94.09% 94.44% N=20 LOOCV LPOCV 81.80% 82.22% 87.35% 88.89% overall classification performance under LPOCV. Thus, we randomly partition the action dataset in 100 trails, and each time we compute SEG-CCRs and SEQ-CCRs 56 under each partition. Finally, we can compute the mean and variance of all SEGCCRs and SEQ-CCRs, to obtain an average classification performance and its spread, as shown in Table 6.8. Table 6.8: Mean and standard deviation (Std) of SEG-CCRs and SEQ-CCRs based on random partitions in LPOCV using NN classifier. Mean Feature SEG-CCR Silhouette 94.95% Optical flow 82.45% Std SEG-CCR 1.84% 4.12% Mean SEQ-CCR 98.56% 84.56% Std SEQ-CCR 2.12% 5.10% We see that the mean of SEG-CCR and SEQ-CCR in Table 6.8 are greater than the corresponding CCRs in Tables 6.3 and 6.7. It implies that the particular leavepart-out partition in Tables 6.3 and 6.7 was more challenging than the 100 partitions on average. Additionally, it is clear that the variances of SEG-CCR and SEQ-CCR are small. Therefore, we see that our approach is robust to different partitions in LPOCV. 6.2 Tests on the KTH dataset We then conducted experiments on the KTH action dataset. This dataset contains six different human actions: handclapping, handwaving, walking, jogging, running and boxing, performed repeatedly by 25 people in 4 different scenarios (outdoor, outdoor with zoomed camera, outdoor with different clothes, and indoor). Action examples are shown in Fig. 6·4. This is a more challenging dataset because: • the camera is no longer static (vibration and zoom-in/zoom-out); • there are large variations in human body shape, view angles, scales and appearance. 57 Due to the difficulties mentioned above, we are not able to obtain a reliable silhouette shape representation of actions. Figure 6·4: Action examples from KTH dataset. Thus, in this section, we test our methods using optical flow features, combined with NN classifier and SLA classifier. For the NN classifier and LOOCV, the SEGCCR is 89.55% and SEQ-CCR is 98.17% (N = 20). Tables 6.9 and 6.10 show action confusion matrices based on SEG-CCR and SEQ-CCR for LOOCV with N = 20. If we use the SLA classifier and LOOCV with N = 20, the SEG-CCR is 90.84% and SEQ-CCR is 98.50%. Likewise, Tables 6.11 and 6.12 show the corresponding action confusion matrices. clap wave walk jog run box box run jog walk wave clap Table 6.9: Segment-level action confusion matrix for the proposed method (optical flow features + NN classifier) on 20-frame segments (SEG-CCR = 89.55%). 90.6 5.9 2.8 0.1 0.2 0.3 4.9 90.3 4.2 0 0.1 0.5 1.9 3.0 94.9 0.1 1.0 0.1 1.1 0.7 0.4 76.6 13.1 8.1 0.3 0.4 0.3 10.3 85.1 3.6 0.6 1.7 0.5 3.1 1.3 93.0 In LPOCV test we followed the training/query set breakup as proposed by Schuldt et al. (Schuldt et al., 2004). When we use the NN classifier, we obtain SEG-CCR of 85.42% and SEQ-CCR of 96.88%. When we use the SLA classifier, we obtain SEG-CCR of 86.04% and SEQ-CCR of 97.40% in this case. 58 clap wave walk jog run box box run jog walk clap wave Table 6.10: Sequence-level action confusion matrix for the proposed method (optical flow features + NN classifier) on 20-frame segments (SEQ-CCR = 98.17%). 100 0 0 0 0 0 0 100 0 0 0 0 0 1 99 0 0 0 0 0 0 97 2 1 0 0 0 6 93 1 0 0 0 0 0 100 clap wave walk jog run box box run jog walk clap wave Table 6.11: Segment-based action confusion matrix for the proposed method (optical flow features + SLA classifier) on 20-frame segments (SEG-CCR = 90.84%). 94.1 2.8 0.8 0.1 0.2 2.0 4.5 90.9 2.7 0.4 0.4 1.2 1.2 1.2 97.4 0 0.1 0.2 0.2 0.5 0.2 81.4 10.0 7.7 0 0.1 0 10.5 86.4 3.0 0.3 0.2 0 4.1 3.1 92.3 clap wave walk jog run box box run jog walk wave clap Table 6.12: Sequence-level action confusion matrix for the proposed method (optical flow features + SLA classifier) on 20-frame segments (SEQ-CCR = 98.50%). 99 1 0 0 0 0 0 100 0 0 0 0 0 0 100 0 0 0 0 0 0 97 2 1 0 0 0 5 95 0 0 0 0 0 0 100 We compared the proposed method with some recent action recognition algorithms. Table 6.13 shows results for LOOCV tests and Table 6.14 shows results for LPOCV tests. Since we know that LPOCV is more challenging than LOOCV, it is unfair to compare method A that is tested under LOOCV with method B that is 59 tested under LPOCV. Although the SEG-CCR for our method is a little worse than that for Ali et al.’s method (Ali and Shah, 2010), our SEQ-CCRs are in line with the best methods today. Table 6.13: Comparison of our optical-flow-based approach (N=20) using LOOCV on KTH dataset. Method SEG-CCR SEQ-CCR NN classifier 89.55% 98.17% SR-based classifier 90.84% 98.50% Kim 95.3% Wu 94.5% Wong 81.0% - Dollar 81.2% - Seo 95.7% Table 6.14: Comparison of the optical-flow-based approach (N=20) using LPOCV on KTH dataset. Method SEG-CCR SEQ-CCR 6.3 NN classifier 85.42% 96.88% SR-based classifier 86.04% 97.40% Ali 87.7% - Laptev 91.8% Le 93.9% Wang 94.2% Kovashka 94.5% Tests on the UT-Tower dataset We also conducted experiments on the UT-Tower action dataset. The UT-tower action dataset comes from the “Aerial View Activity Classification Challenge” in ICPR 2010 Contest on Semantic Description of Human Activities (SDHA). This dataset contains video sequences of a single person performing various actions recorded from the top of the University of Texas at Austin’s main tower. The dataset consists of 108 video sequences of 360 × 240 pixel resolution and frame rate of 10 fps. The contest required classifying video sequences into 9 categories of human actions: {1: pointing, 2: standing, 3: digging, 4: walking, 5: carrying, 6: running, 7: wave1, 8: wave2, 9: jumping}. Each of the 9 actions is performed 2 times by 6 individuals for a total of 12 video sequences per action category. The pointing, standing, digging, and walking video sequences have a concrete background whereas the carrying, running, wave1, wave2, and jumping video sequences have a lawn background. The cameras 60 are stationary but have jitter. The average height of human figures in this dataset is about 20 pixels. In addition to the challenges associated with the low resolution of objects of interest in this dataset, there are additional challenges from shadows and blurry visual cues. Ground truth action labels for all video sequences were provided for training and testing. In addition, in order to alleviate segmentation and tracking issues and make participants focus on the classification problem, ground truth bounding boxes as well as foreground masks for each video sequence were also provided. Action samples are shown in Fig. 6·5. Figure 6·5: Action examples from UT-Tower dataset. 6.3.1 Silhouette features We first tested action recognition performance using the NN classifier and we obtained SEG-CCR of 93.53% and SEQ-CCR of 96.30% (N = 8). Tables 6.15 and 6.16 show the action “confusion” matrices based on SEG-CCR and SEQ-CCR. For the SLA classifier, we obtained SEG-CCR of 96.15% and SEQ-CCQ of 97.22%. Tables 6.17 and 6.18 show the action “confusion“ matrix based on SEG-CCR and SEQ-CCR. 6.3.2 Optical flow features Then, we used optical flow features to represent action dynamics, together with NN and SLA classifiers. For the NN classifier, we obtained SEG-CCR of 82.25% and SEQ-CCR of 86.11% (N = 8). Tables 6.19 and 6.20 show the corresponding action “confusion” matrices based on SEG-CCR and SEQ-CCR. For the SLA classifier, we 61 point stand dig walk carry run wave1 wave2 jump jump wave2 wave1 run carry walk dig stand point Table 6.15: Segment-based action confusion matrix for the proposed method (silhouette features + NN classifier) on 8-frame segments (SEGCCR = 93.53%). 72.3 18.0 9.7 0 0 0 0 0 0 5.8 92.8 1.4 0 0 0 0 0 0 4.5 0.5 94.5 0 0 0 0.5 0 0 0 0 0 97.3 0 2.7 0 0 0 0 0 0 0 97.7 2.3 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 85.6 14.4 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 1.0 0 0 99.0 point stand dig walk carry run wave1 wave2 jump jump wave2 wave1 run carry walk dig stand point Table 6.16: Sequence-based action confusion matrix for the proposed method (silhouette features + NN classifier) on 8-frame segments (SEQCCR = 96.30%). 75.0 16.7 8.3 0 0 0 0 0 0 8.3 91.7 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 point stand dig walk carry run wave1 wave2 jump 88.0 6.0 6.0 4.4 94.2 1.4 2.0 1.5 96.0 1.4 0 0 0 0 0 0 0 0 0 0 0.5 0 0 0 0 0 0 jump wave2 wave1 run carry walk dig stand point Table 6.17: Segment-based action confusion matrix for the proposed method (silhouette features + SLA classifier) on 8-frame segments (SEG-CCR = 96.15%). 0 0 0 0 0 0 0 0 0 0 0 0 0 0.5 0 0 0 0 98.6 0 0 0 0 0 0 99.5 0.5 0 0 0 0 0 100 0 0 0 0 0 0 94.1 5.4 0 0 0 0 7.5 92.5 0 0 0 0 0 0 100 62 point stand dig walk carry run wave1 wave2 jump jump wave2 wave1 run carry walk dig stand point Table 6.18: Sequence-based action confusion matrix for the proposed method (silhouette feature vectors + SLA classifier) on 8-frame segments (SEQ-CCR = 97.22%). 91.7 0 8.3 0 0 0 0 0 0 16.7 83.3 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 obtained SEG-CCR of 81.18% and SEQ-CCR of 85.19%. Tables 6.21 and 6.22 show segment-based and sequence-based action “confusion” matrices. point stand dig walk carry run wave1 wave2 jump jump wave2 wave1 run carry walk dig stand point Table 6.19: Segment-based action confusion matrix for the proposed method (optical flow features + NN classifier) on 8-frame segments (SEG-CCR = 82.25%). 53.0 27.7 14.5 0 0 0 4.8 0 0 45.5 51.5 1.5 0 0 0 1.5 0 0 1.5 1.0 96.5 0 0 0 0.5 0.5 0 0 0 0 93.2 4.1 2.7 0 0 0 0 0 0 0 87.6 12.0 0.5 0 0 0 0 0 0 9.1 90.9 0 0 0 1.1 1.6 0 0 0 0 66.9 30.4 0 0 0 0 0 0 0 17.5 82.5 0 0 0 0 0 0 0 0 0 100 Clearly, the proposed silhouette-based action recognition outperforms its opticalflow-based counterpart by over 10%. This is not surprising when one closely examines two special actions: pointing and standing. These actions, strictly speaking, are not actions as they involve no movement. Thus, optical flow computed in each case is zero (except for noise and errors) leading to failure of optical-flow-based approaches. On the other hand, silhouette-based approaches are less affected since pointing and 63 point stand dig walk carry run wave1 wave2 jump jump wave2 wave1 run carry walk dig stand point Table 6.20: Sequence-based action confusion matrix for the proposed method (optical flow features + NN classifier) on 8-frame segments (SEQ-CCR = 86.11%). 83.3 8.3 8.3 0 0 0 0 0 0 66.7 25.0 8.3 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 75.0 25.0 0 0 0 0 0 0 0 8.3 91.7 0 0 0 0 0 0 0 0 0 100 point stand dig walk carry run wave1 wave2 jump jump wave2 wave1 run carry walk dig stand point Table 6.21: Action confusion matrix for the proposed method (optical flow feature + SR-based classifier) on 8-frame segments (SEG-CCR = 81.18%). 53.0 35.0 8.4 0 0 0 3.6 0 0 36.4 55.7 6.82 0 0 0 1.1 0 0 1.0 1.0 96.0 0 0 0.5 1.5 0 0 0 0 1.4 91.8 1.4 4.1 0 0 1.4 0 0 0 0 95.9 4.2 0.5 0 0 0 0 0 4.6 18.2 77.3 0 0 0 1.1 0.5 1.1 0 0 0 81.0 16.3 0 0 0 0 0 0 0 52.6 47.4 0 0 0 0 0 0 0 0 0 100 point stand dig walk carry run wave1 wave2 jump jump wave2 wave1 run carry walk dig stand point Table 6.22: Action confusion matrix for the proposed method (optical flow feature + SR-based classifier) on 8-frame segments (SEQ-CCR = 85.19%). 66.7 33.3 0 0 0 0 0 0 0 33.3 66.7 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 91.7 8.3 0 0 0 0 0 0 0 58.3 41.7 0 0 0 0 0 0 0 0 0 100 64 standing can still be described by 3-D silhouette shape. Also, note a much higher silhouette-based CCR for wave1. Examining the confusion matrix (not shown here) we realized that around 50% of wave1 videos are misclassified by our optical-flowbased action recognition as wave2. It indicates that the optical-flow features are also not quite discriminative when representing wave1. 6.4 Tests on the YouTube dataset YouTube dataset is a very complex dataset based on YouTube videos (Liu et al., 2009). This dataset contains 11 action classes that are grouped into 25 groups: basketball shooting, biking/cycling, diving, golf swinging, horse-back riding, soccer juggling, swinging, tennis swinging, trampoline jumping, volleyball spiking, and walking with a dog (Fig. 6·6). This dataset is very challenging due to large variations in camera motion, acquisition viewpoint, cluttered background, etc. Since silhouette tunnels are not available for this dataset, we only tested the optical flow features. For the NN classifier, we obtained SEG-CCR of 50.4% and SEQ-CCR of 78.5% (N = 20, LOOCV). Table 6.23 shows the SEQ-CCR comparison of our proposed method with state-of-the-art methods. The performance of our method is in line with state-of-theart methods today. Figure 6·6: Action examples from YouTube action dataset. 65 Table 6.23: Comparison of our proposed approach (N=20) and stateof-art methods based on LOOCV on YouTube dataset. Method SEG-CCR SEQ-CCR 6.5 Proposed 50.4% 78.5% Liu Ikizler 71.2 75.2 Le Wang Noguchi 75.8% 84.2% 80.4% Representation and metric comparison The experiments so far indicate that feature covariance matrices are sufficiently discriminative for action recognition, and the Euclidean distance based on log-covariance matrices is an appropriate metric. However, how would other representations or metrics fair against them? To answer this question, we performed several experiments using silhouette features and the NN classifier on Weizmann dataset. First, rather than using second-order statistics to characterize localized features we have tested first-order statistics, i.e., the mean, under the Euclidean distance metric. As is clear from Table 6.24, recognition performance using mean representation is vastly inferior to that of covariance representation with log-covariance metric (over 50% drop). Secondly, we used a covariance matrix instead of a log-covariance matrix. Again, the performance dropped dramatically compared to the log-covariance representation. Finally, we assumed that feature vectors are drawn from a Gaussian distribution and we estimated this distribution’s mean vector and covariance matrix. Then, we used KL-divergence to measure the distance between two Gaussian distributions. This approach fared much better however it still trailed the log-covariance representation under the Euclidean metric by over 6%. Table 6.24: Recognition performance for various formulations using silhouette features and NN classifier with N=8 on Weizmann dataset. Representation Log-Covariance Mean Covariance Gaussian fit Metric Euclidean Euclidean Euclidean KL-divergence SEG-CCR 97.1 45.8 43.6 91.3 SEQ-CCR 100 48.9 56.7 93.4 66 6.5.1 Analysis of feature importance In Chapter 4, we introduced two examples of local feature vectors: silhouette features and optical flow features, consisting of 13 and 12 features, respectively. It is, therefore, necessary to investigate the contribution of each feature component to the classification performance. In this section, we will analyze the importance of feature components by conducting experiments on the Weizmann dataset based on LOOCV and segment length N = 8 using only subsets of feature components. Silhouette features The silhouette feature vector f(x, y, t) is defined in (4.1). In order to verify usefulness of individual features, we partition f(x, y, t) into three subsets: • (x, y, t): spatio-temporal coordinates, • (dE , dW , dN , dS , dN E , dSW , dSE , dN W ): spatial distances, • (dT + , dT − ): temporal distances. The spatio-temporal coordinates provide localization information for moving objects. Spatial and temporal distances describe local spatial and temporal shape deformations, repectively. Table 6.25 shows the classification performance (SEG-CCR and SEQ-CCR) using subsets of feature components on the Weizmann dataset. The results indicate that the subset (dN E , dSW , dSE , dN W ) is the most significant one. In contrast, the subset (dT + , dT − ) contributes the least to action classification. However, the combination of (x, y, t) and (dT + )/(dT − ) leads to a remarkable classification performance (up to 95.56%). Thus, even if (dT + , dT − ) themselves are not discriminative enough for action recognition, they can still provide auxiliary action characteristics when combined with other features. 67 Table 6.25: Classification performances using subsets of silhouette features on Weizmann dataset using LOOCV (N = 8). Subset of feature components (x, y, t) (dE , dW , dN , dS ) (dN E , dSW , dSE , dN W ) (dT + , dT − ) (x, y, t, dE ) (x, y, t, dW ) (x, y, t, dN ) (x, y, t, dS ) (x, y, t, dE , dW , dN , dS ) (x, y, t, dSE ) (x, y, t, dN W ) (x, y, t, dSW ) (x, y, t, dN E ) (x, y, t, dN E , dSW , dSE , dN W ) (x, y, t, dT +) (x, y, t, dT −) (x, y, t, dT +, dT − ) All features SEG-CCR 69.84% 69.29% 81.83% 33.02% 76.59% 76.83% 82.06% 79.60% 90.71% 80.56% 78.65% 79.68% 81.35% 91.11% 84.92% 85.56% 88.83% 97.05% SEQ-CCR 84.44% 80.00% 89.99% 41.11% 90.00% 91.11% 92.22% 90.00% 96.67% 94.44% 88.89% 92.22% 93.33% 98.89% 95.56% 93.33% 95.56% 100% Optical-flow features The optical-flow feature vector is defined in (4.3). The optical-flow feature components can be partitioned into three groups: • (x, y, t): spatio-temporal coordinates, • (It , u, v, ut, vt ): optical flow and temporal gradients, • (Div, V or, Gten, Sten): optical-flow descriptors derived from fluid dynamics. Table 6.26 shows the classification performance (SEG-CCR and SEQ-CCR) using subsets of optical-flow feature on the Weizmann dataset. From this table we see that (It , u, v, ut, vt ) is the most significant feature subset for action recognition. Specifically, u is the most discriminative feature when combined with the localization subset (x, y, t), that achieves SEQ-CCR of 87.78%. 68 Table 6.26: Classification performance using subsets of optical-flow features on Weizmann dataset using LOOCV (N = 8). Selected features (x, y, t) (It , u, v, ut, vt ) (Div, V or, Gten, Sten) (x, y, t, It) (x, y, t, u) (x, y, t, v) (x, y, t, ut) (x, y, t, vt) (x, y, t, It , u, v, ut, vt ) (x, y, t, Div) (x, y, t, V or) (x, y, t, Gten) (x, y, t, Sten) (x, y, t, Div, V or, Gten, Sten) All 6.5.2 SEG-CCR 71.92% 72.33% 57.14% 73.23% 83.09% 82.43% 80.13% 79.64% 85.63% 79.72% 80.54% 76.35% 77.59% 81.77% 89.74% SEQ-CCR 73.33% 83.33% 78.89% 77.78% 87.78% 84.44% 83.33% 81.11% 88.24% 82.22% 81.78% 77.78% 81.11% 83.33% 91.11% Robustness experiments The proposed framework has performed well when the query action was similar to dictionary actions. In practice, however, the query action may be “distorted”, e.g., person carrying a bag, or may be captured from a different viewpoint. We tested the robustness of our approach to action variability and camera viewpoint on videos originally used by Gorelick (Gorelick et al., 2007) that include 10 walking people in various scenarios (walking with a briefcase, limping, etc.). We tested both silhouette features and optical-flow features using the NN classifier. Experimental results for action variability are shown in Table 6.27. Since there is only one instance of each type of test sequence, SEQ-CCR must be either 100% or 0%. Clearly, all test sequences were correctly labeled even if some segments were misclassified. Also, the optical-flow features perform better overall than the silhouette features (except for “Knees up” at segment level). Experimental results for viewpoint dependence are shown in Table 6.28. The test videos contain the action of walking captured from different angles as shown in 69 Figure 6·7: Action examples of walking variations. Table 6.27: Results of a robustness test to action variability; query actions differ significantly from dictionary actions. Silhouette features Optical-flow features SEG-CCR SEQ-CCR SEG-CCR SEQ-CCR Swing a bag 94.9% 100% 100% 100% Carry a briefcase 100 % 100% 100% 100% Walk with a dog 82.4% 100% 100% 100% Knees up 73.5% 100% 62.7% 100% Limping man 100 % 100% 100% 100% Sleepwalk 100 % 100% 100% 100% Occluded legs 94.9% 100% 100% 100% Normal walk 100 % 100% 100% 100% Occluded by a pole 88.1% 100% 100% 100% Walk in a skirt 100 % 100% 100% 100% Fig. 6·8 (varying from 0◦ to 81◦ with steps of 9◦ with 0◦ being the side view). The action samples in the training dataset (Weizmann dataset) are all captured from the side view. Thus, it is expected that the classification performance will degrade when the camera angle increases. The results indicate that silhouette features are robust for walking up to about 36◦ in viewpoint change and that confusion starts at about 54◦ (walking recognized as other actions). Optical-flow features perform slightly better in this case; good performance up to about 54◦ and misclassification starts around 72◦ . 70 Figure 6·8: Action examples of walking from different viewpoints. Table 6.28: Results of a robustness test to camera viewpoint; query action captured from different angles than those in the dictionary. Silhouette features Optical-flow features SEG-CCR SEQ-CCR SEG-CCR SEQ-CCR Viewpoint 0◦ 100 % 100% 100% 100% Viewpoint 9◦ 100 % 100% 100% 100% Viewpoint 18◦ 100 % 100% 100% 100% Viewpoint 27◦ 92.3% 100% 100% 100% Viewpoint 36◦ 90.8% 100% 100% 100% Viewpoint 45◦ 76.8% 100% 100% 100% Viewpoint 54◦ 38.5% 0% 98.1% 100% Viewpoint 63◦ 20.2% 0% 69.1% 100% Viewpoint 72◦ 13.8% 0% 40.4% 0% Viewpoint 81◦ 5.4 % 0% 30.2% 0% 6.5.3 Computational complexity analysis The proposed approaches are computationally efficient and easy to implement. Our experimental platform is Intel Centrino (CPU: T7500 2.2GHz + Memory: 2GB) with Matlab 7.6. The extraction of 13-dimensional feature vectors from a silhouette tunnel and calculation of covariance matrices take together about 10.1 sec for a 180 × 144pixel, 84-frame silhouette sequence (0.12 sec per frame). The computation of 12dimensional optical-flow feature vectors and their covariance matrices takes about 6 sec (0.07 sec per frame) for the same sequence. Note that silhouette and optical flow estimation are not included. Given a query sequence with 613 query segments and 71 a training set with 605 training segments, the NN classifier requires about 52 sec to classify all query segments (0.08 sec per query segment), while the SLA classifier needs about 44 sec (solving 613 times l1 -norm minimization problem, i.e., 0.07 sec per query segment). The computation time indicates that optical-flow features have lower computational complexity than silhouette-based features, and the NN classifier has very close computational complexity to the SLA classifier. This method is also memory efficient, since the training sets and test sets essentially store 13 × 13 or 12 × 12 covariance matrices, instead of video data.