Topological features for monitoring human activities at distance

Topological Features for Monitoring Human Activities at Distance Javier Lamar Leon1(B),Ra´ul Alonso1, Edel Garcia Reyes1, and Rocio Gonzalez Diaz2 1Patterns Recognition Department, Advanced Technologies Application Center, 7th A # 21406 e/ 214 y 216, Rpto. Siboney, CP 12200 Playa, La Habana, Cuba {jlamar,rbaryolo,egarcia}@cenatav.co.cu 2Applied Math Department, School of Computer Engineering, Campus Reina Mercedes, University of Seville, Seville, Spain [email protected] Abstract. In this paper, a topological approach for monitoring human activities is presented. This approach makes possible to protect the person’s privacy hiding details that are not essential for processing a security alarm. First, a stack of human silhouettes, extracted by background subtraction and thresholding, are glued through their gravity centers, forming a 3D digital binary image I. Secondly, different orders of the simplices are applied on a simplicial complex obtained from I, which capture relations among the parts of the human body when walking. Finally, a topological signature is extracted from the persistence diagrams according to each order. The measure cosine is used to give a similarity value between topological signatures. In this way, the powerful topological tool known as persistent homology is novelty adapted to deal with gender classification, person identification, carrying bag detection and simple action recognition. Four experiments show the strength of the topological feature used; three of they use the CASIA-B database, and the fourth use the KTH database to present the results in the case of simple actions recognition. In the first experiment the named topological signature is evaluated, obtaining 98.8 % (lateral view) of correct classification rates for gender identification. In the second one are shown results for person identification, obtaining an average of 98.5%. In the third one the result obtained is 93.8 % for carrying bag detection. And in the last experiment the results were 97.7 % walking and 97.5 % running, which were the actions took from the KTH database. Keywords: Gait-based recognition ·Topology ·Persistent homology · Gender classification ·Carrying bag detection ·Action recognition 1 Introduction Objects detected by a video surveillance system are usually classified into different categories: human, vehicle, animal, etc. In the case of persons, it is useful another c Springer International Publishing Switzerland 2014 P.L. Mazzeo et al. (Eds.): AMMDS 2014, pp. 40–51, 2014. DOI: 10.1007/978-3-319-13323-2 4 Topological Features for Monitoring Human Activities at Distance 41 level of categorization, which gives clues for the interpretation. After finding the class to which an object belongs, one may try to identify it and interpret its individual behavior in the scene, as well as its interaction with other objects. We consider that nonrigid objects and its actions should be described by the dynamic spatial relations among its different parts. Methods based on geometric features extracted from silhouettes or its contours [1] have been widely used for gait recognition tasks. However, the stability of such features is affected by deformations in the shape of the silhouette. Even for the same individual, little changes on the walking direction, illumination variations and the way the clothes fit to the human body, may cause variability on the geometric features. We conjecture that topological descriptions based on the persistence of homology classes are more invariant to changes and noise in the silhouette shape than classical approaches. This kind of features have been previously used to match nonrigid shapes [2,3], because they are invariant under continuous deformations of the object. Homology is a topological invariant frequently used in practice [4,5]. The ranks of the homology groups, also called Betti numbers, coincide in the first three dimensions with the number of connected components, tunnels and cavities of the object respectively, this as a consequence of the Alexander duality [6]. In particular, the homology could be a robust representation, because the shape of connected components and holes may change under geometric transformations, but their amount will be more stable. Given that is not enough to reach the invariance for the representation, but also needed a set of discriminating features, the approach called homological persistence, which is introduced from now on, will be used in order to elevate the discriminating power of the representation. Ak-simplex σin Rdis the convex hull of a set Sof k+1 affinely independent points, where 0 ≤k≤d. The dimension of σis dim(σ)=|S|−1=k. In (Fig. 1a) from left to right are shown k-simplices of dimensions 0, 1 and 2, which are the only used in this work. For every U⊆Sthe simplex σdefined by Uis said to be a face of σ,ifU=Sthen σis a proper face of σ. A simplex σis a facet or coface of a simplex σwhen σis a face of σ. Let Kbe a collection of simplices, Kis a simplicial complex if it satisfies two properties, namely (i) if σis a face of σand σ∈Kthen σ∈K, and (ii) if Fig. 1. (a) Simplices of dimensions 0, 1 and 2 (left to right), (b) Simplicial complex using 2-simplices (triangles) as maximal dimension simplices. 42 J.L. Leon et al. σ1,σ 2∈Kthen σ1∩σ2is empty or a face of both (Fig. 1b). The dimension of Kis the largest dimension of any of its simplices. It is important to point out that a subset Lof Kis a subcomplex of Kif it is a complex itself, i.e., Lis a subcomplex of Kiff it satisfy property (i), note that Linherit (ii) from K. Let Kbe a simplicial complex with nsimplices, a filter function f:K→R is a function that assigns a real number to each simplex in K.Afilter of Kis an ordering of its simplices [σ1,σ 2, ..., σn], which satisfy that each prefix Ki=[σ1,σ 2, ..., σi] is a subcomplex of the next prefix Ki+1. Persistent homology concerns, given a filter, with how long persist homology classes (connected components, tunnels, cavities, etc.) after they are born; note that looking to a filter as a growing simplicial complex, we may see that homology classes are born and die. The homology classes that persist throughout all the filter i.e., that are born and never die, are the homology classes of K. An algorithm to compute persistent homology takes as input a filter, and gives out a collection of pairs representing the birth and death time of the homology classes. The difference between the birth and death time of a homology class is called its persistence. The persistence diagrams and barcodes are two ways of representing the collection of pairs given up by the algorithm, they are used to study and visualize the persistent homology. In (Fig. 2[7]) is shown an example of barcode representation, where the start of a horizontal bar represents the birth of a homology class and the end its death. An easy handling matlab implementation of an algorithm for computing persistent homology can be found1. Fig. 2. Persistent barcodes. K5betti numbers are H0=1,H 1=1,H 2= 1 and K2 betti numbers are H6=1,H 2=1,H 0=1.Tookfrom[7]. In this paper, a version of the topological features based on homological persistence given in [8,9] is presented, which is valid for gender classification, gait based person identification, carrying bag detection and simple action recognition. As aforementioned, monitoring these human activities makes possible to protect the person’s privacy hiding details, such as faces, that are not essential for 1http://comptop.stanford.edu/programs/plex-2.0.1-windows.zip Topological Features for Monitoring Human Activities at Distance 43 processing a security alarm. The topological features can be used for monitoring human activities even when is possible to use images of high resolution. The rest of the paper is organized as follows. Section 2is devoted to describe the method for obtaining the topological signature. Experimental results are then reported in Sect. 3. We conclude this paper and discuss some future work in Sect. 4. 2 Topological Signature for Activities Monitoring In this section, a topological signature is presented that is used for gender classification, person identification, carrying bag detection and simple action recognition at distance. A traditional approach for constructing a simplicial complex, departs from a points cloud, for which it is necessary to recover topological relations among its points. In our case there are structural relations among the points (pixels), and in fact there are temporal relation too, given that we work with videos. We take advantage of this relations to construct the simplicial complex. 2.1 The Simplicial Complex ∂K(I) First, the foreground (person) is segmented from the background by applying background subtraction and thresholding. The sequence of resulting silhouettes is analyzed to extract one subsequence of representation, which includes at least a gait cycle [10]. The 3D digital binary image I=(Z3,B) (where B⊂Z3is the foreground), is built by stacking silhouettes of a subsequence of representation, aligned by their gravity centers (gc), see (Fig. 3a) and (Fig. 3b). The 3D cubical complex Q(I) associated to Icontains the unit cubes with vertices V={(i, j, k), (i+1,j,k),(i, j +1,k),(i, j, k +1),(i+1,j +1,k),(i+1,j,k +1),(i, j +1, k+1),(i+1,j +1,k +1)}and all its faces (vertices, edges and squares) iff V⊆B. Fig. 3. (a) Silhouettes aligned by their gravity centers. (b) I=(Z3,B) obtained from the silhouettes (GC isthegravitycenterofI). (c) The border simplicial complex ∂K(I). 44 J.L. Leon et al. The squares of Q(I) that are faces of exactly one cube in Q(I) are subdivided in two triangles. The obtained triangles and their faces (vertices and edges) make up the border simplicial complex ∂K(I) (see Fig. 3c). Finally, coordinates of the vertices of ∂K(I)arenormalized to coordinates (x, y, t), where 0 ≤x, y ≤1and tis the number of silhouette of the subsequence of representation. 2.2 Filters for ∂K(I) The topology of ∂K(I) is, in general, very poor. However, in this and the next subsections we present how, using persistence diagrams, it is possible to get a topological signature from ∂K(I) that captures relations among the parts of the human body when walking, and is robust against small input-data perturbations. Up to medical researches [11,12], the natural human gait is defined as a succession of rhythmic and alternate movements of the limbs and the torso. Therefore, the difference in the gait is given by the relative position of the limbs and torso in each moment (structural features). A growing scheme of the simplicial complex ∂K(I) respect to a useful selected filtration can encode the relations among the parts of the human body, which can be therefore very discriminating. When a view direction dis chosen, two filters for ∂K(I) are obtained as follows. All simplices belonging to ∂K(I) are associated with two filter functions f+and f−. For each vertex v∈∂K(I), f+(v) is the distance between vand the plane normal to dpassing through the origin of the reference frame, while f−(v)=−f+(v). Edges and triangles are associated to the smallest value that f+ (resp. f−) assumes on their vertices. Being the simplices of ∂K(I) finite in number, we can determine a minimum value for f+,sayfmin, and a maximum one, fmax. It is now possible to induce two filters on ∂K(I) by ordering its simplices according to increasing values of f+and f−respectively, or according to increasing dimension of the simplices in case of tie, or arbitrarily otherwise. Denote these filters by K[fmin,fmax]=[σ1,...,σ k]andK[−fmax,−fmin]=[σ 1,...,σ k]. 2.3 Persistence Diagrams and Topological Signatures Given a simplicial complex K, a filter function f, and the corresponding filter [σ1,...,σ k]forK,ifσicompletes a p−cycle (pis the dimension of σi) when σiis added to Ki−1=[σ1,...,σ i−1] then a p-homology class γis born at time i, otherwise, a (p−1)-homology class dies at time i. The difference between the birth and death time of a homology class is known as its persistence, which quantifies the significance of a topological attribute. If γnever dies, we set its persistence to infinity. Drawing an horizontal segment [i, j), in a 2D plane, for a p-homology class that is born at time iand dies at time j, we get the p-barcode diagram of the filtration. It represents a p-homology class by a segment whose length is the persistence of that class. In this paper, barcodes are first computed for K[fmin,fmax]and K[−fmax,−fmin]. Then, the barcodes are explored according to a uniform sampling. More precisely, given an integer n>0, n−1 cuts are performed homogeneously in the complex K[fmin,fmax](resp. K[−fmax,−fmin]) as follows: Let’s suppose K[fmin,fmax]={σ0,...,σ m} Topological Features for Monitoring Human Activities at Distance 45 and let’s take Pi=[σ(i−1)m n+1,...,σ im n−1], 1 ≤i≤n, as the partitions given by the cuts. For a fixed i, we compute: (a) Number of homology classes that were born or persist when the simplex σ(i−1)m nis added, and, persist or die when the simplex σim nis added. (b) Number of homology classes that were born in Pi. An analogous process is done for K[−fmax,−fmin]. A vector of 2nentries is then formed containing (a) in entry 2iand (b) in 2i+ 1; this way we obtain for a given dimension p, a vector for each filter of K. The topological signature for a gait subsequence considering a fixed direction of view consists in four 2n-dimensional vectors: (V1,V 2,V 3,V 4) constructed as explained above. Consider that we take into account two dimensions (p=0and p= 1) and two filters (K[fmin,fmax]and K[−fmax,−fmin]). Fig. 4. An example of computation of the first element of a topological signature. For example, consider ∂K(I) given in (Fig. 4) and the direction of view KA. We perform 4 uniform cuts on ∂K(I)[fmin,fmax], see the green lines in the persistence barcode representation. Let’s fix i= 2 and dimension d= 0, according to Fig. 4, the number of homology classes that persist or were born in σ13648, and, persist or die in σ27296 are H0= 6 in dimension 0 and the number of the homology classes that were born in P2are H0= 2 in dimension 0. 2.4 Comparing Topological Signatures The topological signatures for two gait subsequences associated with a fixed view direction, say V={V1,...,V 4}and W={W1,...,W 4}, can be compared according to the following procedure: for every i={1,...,4}compute: Si=Vi·Wi Vi·Wi.(1) which is the cosine of the angle between the vectors Viand Wi. Observe that 0≤Si≤1 since the entries of both vectors are always non-negative. Then, the 46 J.L. Leon et al. total similarity value for two gait subsequences, O1and O2, considering a fixed view direction, is the sum of the 4 similarity measures computed before: S(O1,O 2)=S1+S2+S3+S4.(2) 3 Experimental Results 3.1 Human Gender Classification Human gender classification can be obtained based on face [13], voice [14] or gait [15,16]. Dynamic features when the person walks give the possibility to classify gender at a distance, without any interaction with the subject [10,17,18]. This fact can improve the performance of intelligent surveillance systems and it can reduce the false positive rate during re-identification of an individual on a wide network camera. People not only observe the global motion properties while human walks, but they detect motion patterns of local body parts. For instance, women tent to swing their hips more than their shoulders. On the contrary, men tent to swing their shoulders more than their hips [19]. Moreover, men have in general wider shoulders than women [20]. Experiment 1. The performance of the proposed method is evaluated using the lateral view (90 degrees respect to the camera) in CASIA-B database. This database is composed of 124 subjects, 92 men and 32 women. For each person in the database there are 6 walking sequences, each one provided with background subtraction. In order to avoid bias we selected 25 men and 25 women to perform the experiment. The 50 subjects were divided in 25 disjoint sets, each one containing two subjects (one man and one woman). Only one of these 25 sets was used for testing. The remaining 24 sets were used for training. The correct classification rate (CCR) is the average of the 25 combination of the cross validation. The experimental protocol was made according to [15,16]. In this experiment, a subsequence of representation corresponds to the whole sequence, which has two gait cycles as average. We fixed n= 24 and used 3 view directions. The first one is vertical (i.e. parallel to axis y). The second one forms 45 degrees with axes xand yand 90 degrees with axis t. The third one is parallel to axis t, (see Fig. 5). In each experiment, the results of our method are compared with the methods presented in [15,16]. The aim of this experiment is to evaluate the topological signature for gender classification. Table 1shows the 25-fold-cross-validation of CCR for the whole body using lateral view, as aforementioned. We can see that the topological signature provides better results. 3.2 Gait Recognition Gait recognition is a challenging problem that gives the possibility to identify persons at a distance without any interaction with the subjects, which is very important in real surveillance scenarios [10,18]. Topological Features for Monitoring Human Activities at Distance 47 KB KA KB K A KB KA (a) (c) (b) Fig. 5. View directions used in the experiments. Table 1. Correct classification rates for gait based gender classification. Method Avg computer [15] Avg human observers [15]MCRF [16]Our method CCR 95.97 95.45 98.3 98.88 Experiment 2. We show the performance of the proposed method on the 11 views of the CASIA-B database, which contains 124 subjects. In this experiment the subsequence of representation consists of all the sequence, and the n parameter was set to 24. We have used 4 view directions, they are shown in (Fig. 6)byblacksegments going through the simplicial complex and intercepting the GC point. The first one is parallel to the yaxis and perpendicular to xand t, (Fig. 6a). The second one is parallel to the xaxis and perpendicular to yand t, (Fig. 6b). The third one forms 45 degrees with the xand yaxes and is perpendicular to t, (Fig. 6c). And the last is perpendicular to the previous one, perpendicular to the taxis and as aforementioned goes through the GC point. In all the cases in (Fig. 6)is shown the simplicial complex interleaved between two planes orthogonal to the view direction. The experiment was carried out using 4 video sequences for training, and 2 for testing. The results are compared with the ones in [1]. In Table 2is shown the cross validation average (15 combinations) of correct classification rates. It can be seen in Table 2that the topological approach has a better performance for almost all the view angles, but the algorithm developed in [1] performs better for view angles close to 0 degrees. Therefore, these complementary behaviors conduce us to think in a combination of both approaches in future works. 3.3 Carrying Bag Detection When the goal is to detect a simple behavior, namely a person has left a bag somewhere, it may be important to know that the person is carrying a bag. As carrying some object changes the normal body movements, topological features could be used to differentiate if the person left the bag [21–23]. Experiment 3. In this experiment we used, once again, the CAISA-B database. From the 124 persons, we selected 100 persons carrying bag and 100 walking 48 J.L. Leon et al. (a) (b) (c) (d) Fig. 6. View directions used in the experiments. Table 2. Correct classification rates for gait based recognition. Method 0 18 36 54 72 90 108 126 144 162 180 Avg Wavelet(FD) [1] 100 100 100 93.4 81.1 90.3 90.3 83.3 91.9 92.7 97.6 92.9 Our method 99.3 99.1 98.8 98.3 97.6 98.0 98.3 98.3 98.2 98.2 99.0 98.5 normally. All the sequence was selected as the subsequence of representation, and the nparameter was set to 24. Only lateral view was used in this case. For each person walking normally the 6 sequences provided by the database were used, while for parsons carrying bag the 2 sequences provided by the database were used. The same 4 view directions used for gait based recognition were used in this case. It is important to point out that different kinds of bags carried by the persons, as well as the variation of position where they carry those bags, makes harder the classification, in (Fig. 7) are shown some details. Another hard situation emerges when the body occludes the bag, in this case even humans show difficulties to detect the bag. Fig. 7. CAISA-B carrying bag images. In our experimentation protocol, the 200 subjects to analyze were divided in 100 disjoint sets, each containing a person carrying a bag and a person without bag. One of these sets was used for testing, while the remaining 99 were used for training. The average of the 100 results obtained from the cross validation gives