Detecting Social Interactions in Working Environments Through Sensing Technologies

© Springer International Publishing Switzerland 2016 21 H. Lindgren et al. (eds.), Ambient Intelligence- Software and Applications (ISAmI 2016), Advances in Intelligent Systems and Computing 476, DOI: 10.1007/978-3-319-40114-0_3 Detecting Social Interactions in Working Environments Through Sensing Technologies Juan Antonio Álvarez-García, Álvaro Arcos García, Stefano Chessa, Luigi Fortunati and Michele Girolami Abstract The knowledge about social ties among humans is important to optimize several aspects concerning networking in mobile social networks. Generally, ties among people are detected on the base of proximity of people. We discuss here how ties concerning colleagues in an office can be detected by leveraging on a number of sociological markers like co-activity, proximity, speech activity and similarity of locations visited. We present the results from two data gathering campaigns located in Italy and Spain. Keywords Social network analysis · Computational social science · Smart office 1 Introduction* The analysis of social ties among humans is an important area of research due to its usefulness in understanding the dynamics of human relationships, and, more recently, to its potential in applications like online social networks. In recent years, it has proven its importance also in mobile social networks (MSN), where ties are used to detect communities of users that, in turn, are exploited to improve the strategies for the diffusion of information and services in such networks [1]. J.A. Álvarez-García() · S. Chessa · L. Fortunati · M. Girolami Department of Computer Science, University of Pisa, Largo Pontecorvo 3, 56127 Pisa, Italy e-mail: [email protected] S. Chessa · L. Fortunati · M. Girolami Italian National Council of Research (CNR-ISTI), Via G. Moruzzi, 1, 56124 Pisa, Italy e-mail: {stefano.chessa,luigi.fortunati,michele.girolami}@isti.cnr.it J.A. Álvarez-García · Á.A. García Computer Languages and Systems Department, University of Seville, 41012 Seville, Spain e-mail: [email protected] 22 J.A. Álvarez-García et al. Ties can be detected on the base of contextual information, either from users’ agendas, telephone calls registries or e-mails [2], or from sensors [3]. Recent approaches in MSN identify the ties on the base of co-location traces among users [3, 4], mainly for two reasons:(i) co-location in MSN is associated to frequent and lengthy contacts among users, which are essential to sustain the exchange and diffusion of information; and (ii) co-location is easily producible by using shortrange radio interfaces (such as Bluetooth and WiFi in ad-hoc mode) that are widely available on smartphones and other personal devices. MSN use ties to detect communities, and assume that users in the same community share the same interests. This assumption is very useful to reduce the overhead of the information diffusion strategies since it avoids passing information to users that are not interested in it, and it maximizes the probability of passing the information to users that will most likely meet with other interested users. However, the progress of research and experimentation in this field showed that this assumption is usually unmet (for example, two users that take the same metro every day may share no interests at all). Following this observation, a number of recent works proposed to detect ties based on a richer set of contextual information [4, 5, 6, 7]. This work focuses on detection of ties in MSN and it fits this last trend of research. Specifically, we aim at finding ties based on a short-term analysis of information about the user’s actions and interactions detected by the sensors embedded in conventional smartphones. Short-term analysis means that ties are detected on the base of short temporal sequences (a few minutes at most), in order to provide timely information to the community detection algorithms used to support the information diffusion in MSN. Specifically, we limit this preliminary study to ties among colleagues from the same office. The novel aspects of our approach stands in the identification of a number of activities beyond co-location (also referred to as proximity) that are related to a deeper interaction between two users (and thus to a stronger tie between them). Namely, we focus on symmetry of the physical activity of users, i.e. activities that two users are carrying out together, like walking side-by-side or resting and we relate this to the semantic of the place where activities are being carried out (for example the office, a relaxing area, the cafeteria or a meeting room). To this purpose, we set up an experiment in two sites (at CNR located in Pisa and at the Department of Computer Science in Seville) and that involves 8 participants for each location. Participants are equipped with a smartphone running a sensor data logger app. Moreover, they also keep track of their interactions with other colleagues during the working day. This information is used to assess the ground truth of the interactions against which we compare our algorithm for detecting ties. This paper presents preliminary results on the analysis of a selection of sociological markers that we considered meaningful for detecting the existence and the intensity of the social interactions among colleagues. Detecting Social Interactions in Working Environments Through Sensing Technologies 23 2 Related Works The recent advances in the analysis of complex social networks have recently stimulated research efforts from a rather wide range of viewpoints in social studies through sensing devices [8, 9]. There is a large body of proposals related to the integration of social networks and pervasive computing, whose main objective is to analyze social interactions and organizational structures among individuals, starting from a set of sensing solutions. The availability of a set of sensors on mobile devices, such as cameras, GPS, microphones and accelerometers, along with short-range communication interfaces such as Bluetooth and WiFi [4], enables new sensor-based applications oriented towards activity recognition and interactions with other users. In [3], for example, the authors propose a probabilistic model to discover several interaction types from large-scale dyadic data such as proximity of personal smartphones (via Bluetooth co-location traces), phone calls and emails. Using Bluetooth as a proximity sensor to reconstruct social dynamics at large scale has also been investigated in the reality mining initiative [4]. This project studies complex social systems by inferring patterns in daily user activity, relationship among users, meaningful social locations, and organizational structures. The reality mining dataset was also used to identify social groups and to infer frequency or duration of meetings within each group. Another work [10] showed that it is possible to infer different interaction types using a probabilistic model applied on Bluetooth data. Following studies on proxemics [11], researchers have also employed mobile phones [12] or custom designed personal devices, such as sociometric badges [13], to capture in detail users body movements, relative body orientation, interpersonal distance or recurrent speech patterns, as a mean to detect face-to-face interactions. The study of social interactions in academic or work environments proved to be an interesting topic in the research community. These works are mainly motivated by the beneficial effect that social interactions have in collaborative environments, driving the innovation process [14] or generally improving the well-being of employees [15]. 3 Modelling Social Interactions in Mobile Social Networks The network of mobile devices carried by people is a MSN that can be modeled as a temporal graph ܩ௧ൌሺܰǡܧ ௧ሻ where ܰ is the set of devices in the network and ܧ௧ are the ties among devices at time t. An edge ݁௜ǡ௝ אܧ ௧ is the tuple ݁௜ǡ௝ ൌ ሺ݊௜ǡ݊ ௝ǡݓሻ, where w quantifies the intensity of the tie between devices ݊௜ and ݊௝. We define ݁௜ǡ௝ active if and only if w > 0. Graph ܩ௧ is the snapshot of the MSN at time t. Given an observation interval starting at time t0 and ending at time tk the sequence of snapshots ܩሾ௧బǡ௧ೖሿൌሼܩ ௧బڮܩ ௧ೖሽ is the time-varying graph describing the evolution of the ties among people. We compute the weight w among devices with the function Ȗ that evaluates simultaneously a number of sociological 24 J.A. Álvarez-García et al. markers extracted during an observation interval. Some examples of markers are the locations visited by people, the voice activity or the physical activity of people. The combination of them forms the weight of the tie of a dyad, as described with the following general definition: ݓൌߛ ሾ௧బǡ௧ೖሿሺ݊௜,݊௝ሻ (1) We observe that two individuals are involved in a social relationship if some sociological markers appear evident during a period. For example, shaking hands, talking to each other, staying close during a meeting underlie the explicit willingness of interaction between them. Such markers are evaluated by analyzing the output of sensors available on the people’s devices with a given sampling rate Ȝ. Hereafter, we consider the following sociological markers and we assume that they are all analyzed at the same rate. The proximity marker reveals which devices lay within the intimate zone of a person, (for example, the range can be of 10 meters like in Bluetooth). If device ni detects nj (ni ĺ nj) then we can assume that the device’s owners are also close and they might being interacting. We model the proximity with the Boolean function ݂௉ሺ݊௜ǡ݊ ௝ሻ which is 1 if ni or nj are in range of each other, and 0 otherwise. The speech marker is used to detect if people are talking. We are not interested in recognizing the voice of a person or in recognizing the meaning of the speech, rather we limit to detect if people are talking. We model the speech as the function ݂ௌሺ݊௜ǡ݊ ௝ሻ which is 1 if either ni or nj are talking, and 0 otherwise. We recognize the physical activity of two persons with the goal of determining if there exists symmetry, in particular if they are doing the same activity. When combined with other markers, the symmetry of activities is an amplifier of the intimacy of the relationship between a pair. For example, detecting that two people are in proximity, and that they are talking and walking is a stronger combination than detecting only proximity. Among the physical activities (walking, cycling, running or tilting) we only consider two categories: moving or still. We model the activity with the function ݂ ஺ሺ݊௜ǡ݊ ௝ሻ which is 1 is ni or nj are both still or moving, and 0 otherwise The last marker measures the similarity of the locations visited by a pair of individuals. The similarity of the visited locations is given by the similarity of the WiFi access points detected along the time. The more the list of the access points of a pair is similar, the more likely they visited the same locations and hence the stronger is the intimacy of the relationship between them. Given the list Li of access points stored by a device, we compute the similarity of locations with the Adamic-Adar [16] score: ݂௅ሺ݊௜ǡ݊ ௝ሻൌσଵ ୪୭୥ȁ௛௜௧೔ሺ௟ሻା௛௜௧ೕሺ௟ሻȁ ௟א௅೔ת௅೔. The Adamic-Adar metric considers only the locations l detected from both devices, and it returns the inverse of the logarithmic of the number of times ni and nj detected l. The inverse of the logarithmic penalizes very popular locations that might be not so representative of an intimate interaction. Detecting Social Interactions in Working Environments Through Sensing Technologies 25 4 Experimental Settings and Results We analyze the social interactions among people with a number of participants from Italy and Spain. SmartRelationship is the Android data logger application designed to access the device’s sensors and store the collected data. All the data is collected and analyzed at the end of the experiment to detect social ties between different users. As described in Section 3, we consider a number of markers, namely proximity, speech, physical activity and location that are all sampled with a rate Ȝ = 30s. During the working hours (9 A.M. to 6 P.M.), SmartRelationship stores the following information: Proximity: the set of Bluetooth MAC addresses, the name and the category of the devices lying within the range of 0 to 10 meters are stored. The application performs one Bluetooth scan every Ȝ seconds. We then post-process the proximity logs by excluding the MAC addresses of devices not recognized as part of the experiment. Speech: five-second audio clips with the device’s microphone are recorded. The goal is to detect voice or silence, disregarding the meaning and source of the speech. The audio clips are post-processed through a voice activity detection algorithm based on Gaussian Mixed Model, which detects if people are talking or not. This algorithm does not discriminate between noise and voices so it is not a key feature in our system. The solution is naturally prone to false positives, since speech can be detected from external conversations not involving the user. Activity: the physical activity of the device’s owner by exploiting the Google Activity Recognition APIs is gathered. The list of detected activities is filtered by three means of activities: in vehicle, on bicycle and on foot. If the sum of their confidence is over 50%, then the activity detected is moving, otherwise it is still. Location: information about the WiFi access points detected is gathered. In particular, we store the WiFi network name (SSID), the MAC address of the access point (BSSID) as well as the RSSI of the access point. We conducted the experimentation in two locations, namely the ISTI-CNR in Pisa (Italy), and the University of Seville (Spain). In each location we recruited 8 volunteers. The experiment lasted for 5 days, from October 26th to October 30th, 2015. During each day, each participant filled a questionnaire to keep track of all his meaningful social interactions, annotating the people involved and the time window with one-hour granularity. To our purposes, an interaction is meaningful if it lasts at least 2 minutes and it is a face-to-face interaction involving at least proximity and voice. This excludes short and involuntary interactions with other colleagues. 26 We analyze the data s SmartRelationship. N ote devices of the participant could not meet a sampl i following the human rhy t tion period. For each time interval such interval. Figure 1 s h shows the proximity mar k Fig. 1 Sampling rate of the p Fig. 2 Time series of the so c J.A. Álvarez-García et a s et obtained by postp rocessing the logs produced b that we downsample the dataset at Ȝ ’ = 120s b ecause t h s shown very different performances, and slower devic i ng rate of 30 seconds. Also, social interactions evol v t hms and we consider that Ȝ ’ > Ȝ is a reasonable obser v of width Ȝ ’ seconds, we aggregate all the samples wit h h ows how the marke r s are grouped. The blue dotted li n k er (f P ) with rate Ȝ, the orange line shows f P with Ȝ ’ = 120s p roximity marker. c iological markers of two participants. a l. b y h e e s v e v ah in n e . Detecting Social Interactions in Working Environments Through Sensing Technologies 27 We first analyze the time series of the sociological markers described in Section 3 to identify those features underling a social interaction. Figure 2 shows the time series of the proximity, speech, activity and location markers of two participants for 3 days. In the figure, the proximity marker follows an intuitive but important pattern. In fact, contacts among participants happen during the daily working hours (from 9.00 AM to 6 PM). On the other hand, when fP = 0 then participants are not in proximity and hence they are not involved in any social interaction, differently when fP = 1 it means that they are close and they might interact. The speech marker is useful to identify the time periods of absence of verbal interactions (when fS = 0). In fact, if two people are in silence we assume that they are not interacting. The activity and the location markers are used as amplifier of the magnitude of the interactions. In particular, when we do not detect symmetry of physical activity (i.e. when fA = 0), then we penalize the intensity of the tie, conversely when fA = 1 we increase the intensity. Lastly, the location marker reveals the similitude of locations visited along the time. An interesting property from the location marker is that colleagues spending most of the time in their office have a low score of the location marker, because the locations detected become more and more popular day after day. Differently, as soon as two participants move toward a new location, the location score increases. We then combine together the sociological markers shown in Figure 2 with the goal of assigning a weight w to the interactions among participants along the time. To this propose we define the Ȗ function (defined in a general form in (1)) as follows: ߛሾ௧బǡ௧ೖሿሺ݊௜,݊௝ሻൌ݂ ௉ȉሾ݂ ௌȉሺ݂ ஺൅ሺ݂ ௅ሻሿ Figure 3 shows the results of the Ȗ function computed for the time series shown in Figure 2. We compare the Ȗ function against the ground truth obtained by analyzing the questionnaires of the participants. Since questionnaires are not symmetric (the answers from the participants might not be the same), we assign a higher value to the time slots in which we found concordance of answers and lower values when we detect discrepancies on the answers. Fig. 3 The fusion algorithm applied to the sociological markers of two participants. 28 J.A. Álvarez-García et al. Figure 3 shows that the intensity of the interactions between the two participants changes along the time. When the proximity and/or speech markers are not active (fP and fS are 0), then we do not detect interaction and we assign Ȗ = 0. This is the case for Tuesday 27th from 10.30 to 12.00 and sparsely on Wednesday 28th before 12.00). Differently, Ȗ > 0. In particular, Ȗ increases when the two participants are in proximity, they are talking and they are visiting a new location (during lunch time at the canteen). 5 Conclusions The identification of social ties and the assessment of their strength is becoming a problem of great interest due to its application to MSN. Recent works have already identified the limit of considering only co-location to identify ties. This work follows this trend of research and, on the base of a preliminary experimentation on a case of colleagues in an office, we analyze the information provided by markers concerning co-activity such as talking or moving along with location and the more conventional proximity. Although only two subjects are studied, to check the validity of our implementation, we can conclude that similarity in physical activity and fusion of several sources help us to detect real social interactions. Future work will expand this experimentation and will test the method for identification of communities. Acknowledgments This research is partially supported by the projects of the Spanish Ministry of Economy and Competitiveness HERMES (TIN2013-46801-C4-1-r) and Simon (TIC-8052) of the Andalusian Regional Ministry of Economy, Innovation and Science. References 1. Girolami, M., Chessa, S., Caruso, A.: On Service Discovery in Mobile Social Networks: Survey and Perspectives. Computer Networks 88, 51–71 (2015) 2. Vanini, S., et al.: Modeling social interactions in real work environments. In: Proc. 6th International Workshop on Hot Topics in Planet-Scale Measurement. ACM (2015) 3. Do, T.M.T., Gatica-Perez, D.: Human interaction discovery in smartphone proximity networks. Personal and Ubiquitous Computing 17(3), 413–431 (2013) 4. Eagle, N., Pentland, A.: Reality mining: sensing complex social systems. Personal and Ubiquitous Computing 10(4), 255–268 (2006) 5. Matic, A., Osmani, V., Mayora-Ibarra, O.: Analysis of social interactions through mobile phones. Mobile Networks and Applications 17(6), 808–819 (2012) 6. Junker, H., et al.: Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition 41(6), 2010–2024 (2008) 7. Wyatt, D., et al.: Inferring colocation and conversation networks from privacysensitive audio with implications for computational social science. ACM Trans. on Int. Systems and Technology (TIST) 2(1), 7 (2011) Detecting Social Interactions in Working Environments Through Sensing Technologies 29 8. Lane, N.D., et al.: A survey of mobile phone sensing. Communications Magazine, IEEE 48(9), 140–150 (2010) 9. Stopczynski, A., Sapiezynski, P., Pentland, A., Lehmann, S.: Temporal fidelity in dynamic social networks. Eur. Phys. J. B 88, 249 (2015) 10. Do, T.M.T., Gatica-Perez, D: Groupus: smartphone proximity data and human interaction type mining. In: 15th Sym. on Wearable Computers (ISWC) (2011) 11. Hall, E.T.: Proxemics: The study of man’s spatial relations (1962) 12. Banerjee, N., et al.: Virtual compass: relative positioning to sense mobile social interactions. In: Pervasive Computing, pp. 1–21. Springer, Heidelberg (2010) 13. Onnela, J.-P., Waber, B.N., et al.: Using sociometers to quantify social interaction patterns, Scientific Report 4 (2014) 14. Peschl, M.F., Fundneider, T.: Designing and enabling spaces for collaborative knowledge creation and innovation: From managing to enabling innovation as socioepistemological technology. Computers in Human Behavior 37, 346–359 (2014) 15. Dimotakis, N., Scott, B.A., Koopman, J.: An experience sampling investigation of workplace interactions, affective states, and employee well-being. Journal of Organizational Behavior 32(4), 572–588 (2011) 16. Adamic, L., Adar, E.: Friends and neighbors on the web. Social Networks 25(3), 211–230 (2003)