Activity Recognition System Using Non-intrusive Devices through a Complementary Technique Based on Discrete Methods

Activity Recognition System Using Non-intrusive Devices through a Complementary Technique Based on Discrete Methods Miguel ´ Angel ´ Alvarez de la Concepci´on1, Luis Miguel Soria Morillo1, Luis Gonz´alez Abril2, and Juan Antonio Ortega Ram´ırez1 1Computer Languages and Systems Dept., University of Seville, 41012 Seville, Spain {maalvarez,lsoria,jortega}@us.es 2Applied Economics I Dept., University of Seville, 41018 Seville, Spain [email protected] Abstract. This paper aims to develop a cheap, comfortable and, specially, efficient system which controls the physical activity carried out by the user. For this purpose an extended approach to physical activity recognition is presented, based on the use of discrete variables which employ data from accelerometer sensors. To this end, an innovative selection, discretization and classification technique to make the recognition process in an efficient way and at low energy cost, is presented in this work based on Ameva discretization. Entire process is executed on the smartphone and on a wireless health monitoring system is used when the smartphone is not used taking into account the system energy consumption. Keywords: Contextual Information, Discretization Method, Mobile Environment, Qualitative Systems, Smart-Energy Computing. 1 Introduction In recent years, thanks largely to the increased interest on monitoring certain sectors of population such as elderly people with dementia or people in rehabilitation, activity recognition systems have experienced an increase in both number and quality results. However, most of them are in a high computational cost and hence, it cannot be executed into a general purpose mobile device. Calculation of the physical activity of a user based on data obtained from an accelerometer is a current research topic. Furthermore, many works is going to be analyzed showing some identified limitations that make these systems uncomfortable for users in general. The first difference observed between the systems developed is the type of used sensor. There are systems using specific hardware [1], while others use general purpose hardware [2]. Obviously, the use of generic hardware is a benefit for users, since the cost of devices and versatility of them are points in their favor. Not to mention decreasing the loss and forgetting risk due to they have been integrated on an everyday object like users’ smartphones. Another difference found between the surveyed proposals is the number and position of the sensors. In [3] can be seen that the accelerometer sensor is placed in a glove and a multitude of activities depending on the movement of the hand are recognized. In contrast, other studies use various sensors throughout the body [4], [5] or a wearable wireless sensor node with a static wireless non-intrusive sensory infrastructure [6] to recognize these activities. According to some comparative studies and previous works based on multiple sensors, they are more accurate. Although, works like [2], where a sensor is at users’ pocket or in the hipe, is more comfortable for them. By this way, place them in the monitored person is easier, not to mention that the infrastructure is much lower. Thus, the presented work will is focus on the recognition of physical activities carried out by users throughout their mobile devices. So, it must be paid special attention to energy consumption and computational cost of used methods. Also, a wireless health monitoring system can be used to increment the user acceptance, i.e. the user does not carry the mobile devices all the time in an indoor environment. One step further, some works do not only use data from accelerometers, but use other sources such as microphone, light sensor or voice recognition to determine the context of the user [7]. However, they present problems i.e. when the environment is noisy or the user is alone. There are related works where data for activities recognition are obtained through mobile devices, but these data are sent to a server to process the information [8]. Thus, computational cost is not a handicap and because of this more complex methods are used. In contrast, the efficiency is a crucial issue when processing is carried out in the mobile device [9], [10]. To reduce the cost associated to accelerometer signal analysis, this paper opts for a novel approach based on a discretization method. Thanks to discretization process, classification cost is much lower than working with continuous variables. Because of this, it is possible to eliminate the correlation between variables during the recognition process and on the other hand, to minimize the energy consumption from the process. Working in the domain of discrete variables to perform learning and recognition of activities is a new approach offered by this work. This decision was largely due to the high computational cost required for learning algorithms based on continuous variables used for this purpose over the years. In [11], a labeling process, like a discretization process, is used to obtain a Qualitative Similarity Index (QSI), so it can be said that a transformation of the continuous domain to the discrete domain of values of the variables is beneficial in certain aspects. But, before the self-recognition or learning, it is necessary to carry out a process of Ameva discretization from its algorithm [12]. It has a number of advantages over other well-known discretization algorithms like CAIM discretization algorithm [13], i.e. it is unsupervised and very fast. The most notable of these is the small number of intervals generated which facilitates and reduces the computational cost of the recognition process. It should be noted that many of these studies could be seen in action during the competition EvAAL 2012 [14] in Activity recognition track. EvAAL is an annual international competition that addresses the challenge of evaluation and comparison of Ambient Assisted Living (AAL) systems and platforms, with the final goal to assess the autonomy, independent living and quality of life that AAL systems may grant to their end users. In this track competition, four teams participated in the challenge: CUJ (from the University of Chiba, Japan) [15], CMU (from Carnegie Mellon and Utah Universities, USA) [16], DCU (from Dublin City University, Ireland) [17] and USS (from University of Seville, Spain) [12]. Finally, although CMU had the best accuracy in the results, USS won the competition because its simplicity and interoperability gave good marks in all the evaluated criteria. In order to improve the accuracy problems encountered during the celebration of the EvAAL 2012 competition, some significant improvements in Ameva discretization algorithm are proposed. Also, in addition to detect specific activities, the barometric sensor which is being included in the latest generation of mobile devices is used. Finally, in order to answer the question about what would happen if you decide not to use your mobile device in an indoor environment, as happens in real life, a complementary wireless device is also optionally used. There are other similar EvAAL competitions such as HARL [18], OPPORTUNITY [19], HASC [20] or BSN contest [21]. The paper is organized as follows: first, the activity recognition step is presented in Section 2. Also, the data collection and the set of activities are presented. Section 3 presents the methodology to determine the activity using the Ameva discretization. Section 4 reports the obtained results of applying the methodology. Finally, the paper conclusions with a summary of the most important points are in Section 5. 2 Activity Recognition The final real system consist only of a smartphone and, optionally, a wireless device, configured to detect the competition activities: lie, sit, stand, walk, bend, fall and cycle. 2.1 Data Collection In contrast to the needs of some studies that require a training set to classify a recognized activity correctly, this paper reduces the waiting time for recognition, providing valid information for an activity frequently. To this end, a training set and a recognition set are obtained using 5-secondtime windows of fixed duration which has been determined empirically as optimum length from a performance and an accuracy analysis of the system. The time length of five seconds of these windows has been chosen because for our system is very important to ensure that in each time window there is at least one cycle of activity, where activity cycle is defined as a complete execution of some activity patterns. For example, two steps are a walking activity cycle and one pedal stroke is the activity cycle for cycling. If at least one cycle of activity can not be guaranteed in each time window, it is not possible to determine the activity from accelerometer patterns. This analysis is performed based on the values obtained from the accelerometer, which significantly improve the precision of the body-related activities, and a barometer to detect environment-related activities, such as going upstairs and downstairs. The latter sensor has most often been integrated in recent mobile devices, allow to increase the overall system accuracy detection of activities. So, based on these time windows that contain data for each accelerometer axis and reducing the computational cost of the new solution, signal module has been chosen to work. This eliminates the problem caused by the device rotation [22]. Furthermore, it increases user comfort with the system by removing the restriction to keep the orientation during the learning and recognition process. For each data in a time window size N,ai=(ax i,a y i,a z i), i=1,2,...,N where x,yand zrepresent the three accelerometer axis, the accelerometer module is defined as follow: |ai|=(ax i)2+(ay i)2+(az i)2 Hence, the arithmetic mean, the minimum, the maximum, the median, the standard and the mean deviation, and the signal magnitude area statistics are obtained for each time window. In addition to the above variables, hereafter called temporary variables, a new set of statistics called frequency-domain features from the frequency domain of the problem are generated. Thus, in order to obtain the frequency-domain features, Fast Fourier Transform (FFT) is applied for each time window. For the barometer sensor, two measures are obtained for each time window: at the beginning and at the end, taking into account the difference between them. b=bN−b1 It is important to note that in this case, the absolute value is not takeninto account, contrary to what was done with the values obtained from the accelerometer. 2.2 Set of Activities Far from being a static system, the number and type of activities recognized by the system depends on the user. Thanks to this proposal when users is carrying out activities that have not been learned before can be determined. This is achieved basing on the analysis of probability associated to each pattern while user is performing the activities. Obviously, the number of activities to be detected will impact on the accuracy of the system. Especially if acceleration patterns between activities are very similar. For a large numbers of users could be interesting recognize a few activities, such as walking, sitting and falling. But for another users, activities like driving or biking would be important. However, to carry out a comparative analysis of the accuracy and performance of the discrete recognition method proposed below, 8 activities were taken into account. These activities are immobile, walking, running, jumping, cycling, drive, walking-upstairs and walking-downstairs. Therefore, the learning system allows the user to decide what activities he/she wants the system to recognize. This is highly useful when the determination of certain very specific activities on monitored users is required. 3 Methodology 3.1 Ameva Algorithm Let X={x1,x 2,...,x n}be a data set of an attribute Xof mixed-mode data such that each example xibelongs to only one of the classes of class variable denoted by C={C1,C 2,...,C },≥2 A continuous attribute discretization is a function D:X→Cwhich assigns aclassCi∈Cto each value x∈Xin the domain of property that is being discretized. Let us consider a discretization Dwhich discretizes Xinto kdiscrete intervals: L(k;X;C)={L1,L 2,...,L k} where L1is the interval [d0,d 1]andLjis the interval (dj−1,d j], j=2,3,...,k. Thus, a discretization variable is defined as L(k)=L(k;X;C) which verifies that, for all xi∈X, a unique Ljexists such xi∈Ljthat for i=1,2,...,n and j=1,2,...,k. The discretization variable L(k)ofXand the class variable Care treated from a descriptive point of view. The main aim of the Ameva method [12] is to maximize the dependency relationship between the class labels Cand the continuous-values attribute L(k), and at the same time to minimize the number of discrete intervals k. For this, the following statistic is used: Ameva(k)= χ2(k) k(−1) where χ2(k)=N⎛ ⎝−1+   i=1 k  j=1 n2 ij n·inj·⎞ ⎠ and nij denotes the total number of continuous values belonging to the Ciclass that are within the interval Lj,ni·is the total number of instances belonging to the class Ciand n·jis the total number of instances that belong to the interval Lj,fori=1,2,..., and j=1,2,...,k, fulfilling the following: ni·= k  j=1 nij ,n ·j=   i=1 nij ,N=   i=1 k  j=1 nij The original developed algorithm to obtain the best intervals with the Ameva discretization is based on finding the cutoff points that provide the best coefficient. To do this, the values of the variables are sorted to find the first cut (local maximum). Then, it returns the next cut, and so on, until the Ameva coefficient does not improve. This behavior causes the complexity of the algorithm is quadratic order, O(n2). A graphic with three local maximums can be seen in Figure 1. Fig. 1. An example of Ameva coefficient values with three local maximums The presented improvement in this work allows to find all cuts, allowing the complexity of the algorithm would be of linear order, O(n). Although there is a loss of precision, it is negligible for the field of study of this work, since it allows to obtain good results. Finally, for each statistical Sp∈{S1,S 2,...,S m}, the discretization process is performed, obtaining a matrix of order kp×2, where kpis the number of class intervals and 2 denotes the inf(Lp i)andsup(Lp i) interval limits iof pstatistical. Hence, a three-dimensional matrix containing the statistics and the set of interval limits for each statistic is called Discretization Matrix and it is denoted by W=(wpij ) where p=1,2,...,m,i=1,2,...,k pand j=1,2. Therefore, Discretization Matrix determines the interval at which each data belongs to the different statistical associated values, carrying out a simple and fast discretization process. Class Integration. The aim in the next step of the algorithm is to provide a probability associated with the statistical data for each of the activities based on previously generated intervals. For this purpose, the elements of the training set x∈Xare processed to associate the label of the concrete activity in the training set. In addition, the value of each statistic is calculated based on the time window. For carrying out the previous process, a Class Matrix, V, is defined as a three-dimensional matrix that contains the number of data from the training set associated with a Linterval in a Cactivity for each statistical Sof the system. This matrix is defined as follows: V=(vpij ) where vpij =#{x∈X|inf(Lp i)<x≤sup(Lp i)},andS=Sp,C=Cj, p=1,2,...,m,i=1,2,...,k pand j=1,2,...,. So, each position in the Class Matrix is uniquely associated with a position in the Discretization Matrix determined by its range. At this point, there is not only possible to determine the discretization interval, but the Class Matrix helps to obtain the probability associated with the discretization process performed with the Ameva algorithm. Activity-interval Matrix. The next step is determined a three-dimensional matrix, called Activity-Interval Matrix and denoted by U, which determines the likelihood that a given value xassociated to a Sstatistical corresponds to C activity in a Linterval. This ratio is based on obtaining the goodness of the Ameva discretization and the aim is to determine the most probable activity from the data and the intervals generated for the training set. Each value of Uis defined as follows: upij =vpij vp·j q=1,q=j1−vpiq vp·q −1 where vp·jis the total number of time windows of the training process labeled with the jactivity for the pstatistic, and p=1,2,...,m,i=1,2,...,k pand j=1,2,..., Given these values, Ufor the pstatistic is defined as Up= ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ up00 ... u p0j... u p0 . . ..... . ..... . . upi0... u pij ... u pi . . ..... . ..... . . upkp0...u pkpj...u pkp ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ As can be seen in the definition of U, the likelihood that a data xis associated with the interval Licorresponding to the activity Cj, depends not only on data, but all the elements associated with the interval Lifor the other activities. Thus, each upij matrix position can be seen as a grade of belonging that a given xis identified to Cjactivity, that it is included in the Liinterval of the Spstatistic. Similarly, the elements of Uhave the following properties: –upij =0 ⇐⇒ vpij =0∨vpiq =vp·q,q=j –upij =1 ⇐⇒ vpij =vp·j=vpi· Figure 2 shows the overall process described on this section for carry on data analysis and interval determination. Recovery data from accelerometer sensor End of temporal window? No Build temporal window data set Yes Get time-domain measures Apply Fast Fourier Transform processing Get frequencydomain measures Remove noise applying filters Execute Ameva algorithm over each variable Get intervals for each measure and associated activity (DiscretizationMatrix) number of data from the training set included on each interval (ClassMatrix) Obtain relative probabilities of each activity of belong to each interval of Class-Matrix (Activity-Interval Matrix) Save ActivityInterval Matrix into user profile Fig. 2. Overall process of data analysis and interval determination 3.2 Classification Process Having obtained the discretization intervals and the probabilities of belonging to each interval, the process by which the classification is performed can be described. This classification is based on data from the analysis of time windows. The process is divided into two main steps: the way in which to perform the recognition of physical activity is first described; and the process to determine the frequency at which some particular activity is then presented. Classifying Data. For the classification process, the more likely activity is decided by a majority voting system. As said above, this process parts from the Activity-Interval Matrix and a set of data x∈Xfor the Sset. Therefore, it consists in finding an activity Ci∈Cthat maximizes the likelihood. The above criterion is collected in the following expression, denoted by mpa (most likely activity): mpa(x)=Ck where k=arg(maxjm p=1 upij |x∈(inf(Lp i),sup(Lp i)]). The expression shows that the weight contributed by each statistical to the likely calculation function is the same. This can be done under the assumption that all statistical provide the same information to the system and there is not correlation between them. Thus, the mpa represents the activity whose data, obtained through the processing time window, is more suited to the value set from U. In this way, the proposed algorithm not only determine the mpa, but its associated probability. From this likelihood, certain activities that do not adapt well to sets of generic classification can be identified. It is an indication that user is carrying out new activities for which the system has not been trained previously. Figure 3 shows the overall process described on this section for recognition process from Activity-Interval Matrix calculated in the previous stage. Recovery data from accelerometer sensor End of temporal window? No Build temporal window data set Yes Get time-domain measures Apply Fast Fourier Transform processing Get frequencydomain measures Find an activity such that the sum of each interval associated probabilities (in Activity-Interval Matriz) is maximized Save most-likely activity into user activity log Fig. 3. Overall recognition process from data sensors 4 Method Analysis Once exposed the bases of the developed activities recognition algorithm, an analysis of the new proposal was performed. To do this, the new development was compared with a recognition system widely used based on neural network. In this case, both learning and recognition was performed by continuous methods. The test process was conducted in a Google Nexus One for a group of 10 users. Notably, the activity habits of these users were radically different, since 5 of them were under 30 years while the rest were older than this age. For this purpose, a document was delivered to each user for describing the activity performed, start time and end time. Finally, the learning process consisted on the performing of each activity recognized by the system for a time of 6 minutes. As for the recognition process, users were followed over a period of 72 hours. Moreover, the energy consumption and the processing cost of the system when it is working on a mobile device are considered. In this case, the conclusion reached is that the method based on Ameva reduces the computational cost of the system by about 50% (see Figure 4. The time needed to process a time window by using nueral networks methods is 1.2 seconds, while, for the Amevabased method is 0.6 seconds.