Towards a cloud‑based automated surveillance system using wireless technologies

Towards a cloud‑based automated surveillance system using wireless technologies Javier J. Salmerón‑García1 · Sjoerd van den Dries2 · Fernando Díaz‑del‑Río1 · Arturo Morgado‑Estevez3 · Jose Luis Sevillano‑Ramos1 · M. J. G. van de Molengraft2 addition, this proof-of-concept claims that many interesting opportunities and challenges arise, for example, when mobile watch robots and fixed cameras would act as a team for carrying out complex collaborative surveillance strategies. Keywords Surveillance · World modeling · Wireless communication · Computation offloading · RGBD cameras · Cloud robotics 1 Introduction Nowadays, the development of automated surveillance systems has become more than widespread [7]. There are innumerable environments where their use has become critical, such as cities, households and corporate buildings, not only for security, but also for people location or customer queue analytic [38]. Consequently, there is a need to process, send and analyze the high volume of sensorial media produced by cameras, radars and other sensors. This task involves not only the communication infrastructure but also the necessary multimedia computing framework. More precisely, in a future smart city there would be innumerable devices (small sensors, computers, mobile robots, amongst others) that working together should perfectly create an extremely safe environment. However, several cutting-edge surveillance solutions rely on very expensive sensing technology, high performance dedicated hardware and wired connectivity, and are therefore unaffordable for many companies or scenarios. Hence, the question to be addressed in this paper would be: until what extent can all these heterogeneous devices collaborate to perform a multifaceted task-like surveillance using current low-cost available technologies? Abstract Cloud Computing can bring multiple benefits for Smart Cities. It permits the easy creation of centralized knowledge bases, thus straightforwardly enabling that multiple embedded systems (such as sensor or control devices) can have a collaborative, shared intelligence. In addition to this, thanks to its vast computing power, complex tasks can be done over low-spec devices just by offloading computation to the cloud, with the additional advantage of saving energy. In this work, cloud’s capabilities are exploited to implement and test a cloud-based surveillance system. Using a shared, 3D symbolic world model, different devices have a complete knowledge of all the elements, people and intruders in a certain open area or inside a building. The implementation of a volumetric, 3D, object-ori- ented, cloud-based world model (including semantic information) is novel as far as we know. Very simple devices (orange Pi) can send RGBD streams (using kinect cameras) to the cloud, where all the processing is distributed and done thanks to its inherent scalability. A proof-of-concept experiment is done in this paper in a testing lab with multiple cameras connected to the cloud with 802.11ac wireless technology. Our results show that this kind of surveillance system is possible currently, and that trends indicate that it can be improved at a short term to produce high performance vigilance system using low-speed devices. In *Fernando Díaz-del-Río [email protected] 1 Escuela Técnica Superior de Ingeniería Informática, Universidad de Sevilla, Sevilla, Spain 2 Technische Universiteit Eindhoven, Eindhoven, The Netherlands 3 Escuela Superior de Ingeniería, Universidad de Cádiz, Cádiz, Spain A powerful tool to fulfill this challenge is the incorporation of Cloud Computing, because many of its properties can benefit such surveillance systems. Specifically, the following characteristics fit perfectly: ––On-demand resource provision: The cloud is able to provide both computing and storage resources. This is normally done in the form of the so-called containers. For instance, should a surveillance or “watch” robot require computing power to perform a certain task, then it would only have to spin up as much instances as required to complete it. Most commercial cloud platforms (for example, Amazon EC2, Windows Azure, Google App Engine, etc.) use a pay-as-you-go approach. Nevertheless, this should be much cheaper than all the capital expenditure required for a cluster, not to mention all the maintenance costs. ––Dynamic scalability: This property relates to the ability of the cloud to dynamically adapt to the user needs. A clear example is that of a multiple intrusion. If, suddenly, the number of requests dramatically increase due to this transgression, the service might not be able to process them in a reasonable time. To address this, the cloud could increase the number of computing resources devoted to this task. Moreover, once the threat has ceased and the number of requests decreases, the cloud could reduce the amount of computing resources. This relates to the “utility” concept behind Cloud Computing. In fact, a great interest exists in applying and developing Cloud Computing frameworks to the surveillance area [17, 26]. However, many challenges still remain. For instance, the incorporation of the cloud paradigm forces developers to rethink their software architectures so, for instance, the dynamic scalability property could be exploited (see Sect. 4). Another major concern is that, as pointed out in [26], the cost for deploying a video surveillance system using cloud computing is not necessarily less expensive than implementing the hardware locally. Furthermore, legal and privacy issues must be addressed when using commercial cloud platforms. An interesting approach is the use of a private cloud setting, so that critical surveillance data is always kept in the private cloud [17]. This approach also facilitates the integration of embedded devices that can successfully perform collaborative tasks such as localization, mapping [30], traffic-light status detection [4] or healthcare [10]. As a consequence, concepts such as “Internet of Robots” [13, 37] are now possible. Furthermore, a private cloud can be complemented with commercial cloud platforms when needed to optimize deployment and maintenance costs, an approach commonly known as hybrid cloud [17]. Therefore, using the cloud for a collaborative surveillance system is another interesting application that we address and evaluate in this work. In a great extent, the answer to reduce costs and deploy more easily a vigilance system that integrates a number of embedded devices comes from the use of private/hybrid Cloud Computing and high bandwidth wireless technologies. Moreover, the cloud is the most suitable and costeffective candidate for what is known as computation offloading. The idea is to free an embedded device from heavy computations, so that an external platform would do them and get the results back. This is especially interesting as it allows the devices to perform more complex tasks, overcoming their hardware limitations and their inherent difficult upgrading. Furthermore, offloading CPU-intensive tasks implies less power consumption in the devices and robots. Following this idea, building very simple devices that rely on computation offloading will set a trend in forthcoming years [31]. On the other hand, the use of high bandwidth wireless technologies is especially interesting for two reasons. First, it permits the use of mobile robots or UAVs (Unmanned Aerial Vehicles), bringing innumerable possibilities in surveillance. Second, the new standard 802.11ac promises bandwidths of the order of 1 Gbps [9]. Hence, eventually, current bottlenecks of WiFi connections could be overcome. Therefore, the convenient combination of Cloud Computing, wireless technologies and heterogeneous embedded devices (depending on the budget) could be a perfect candidate for creating a cost-effective surveillance and intruder detection solutions. Besides, there is a problem that needs to be addressed to detect intruders: the correct interpretation and sharing of information of the environment. In this sense, the lack of semantic information provokes problems in this kind of tasks, for instance [5, 11, 22]: ––Some operations such as clearing could be done more efficiently if more information of the static environment was known. ––Without symbolic information, it is very difficult for the robot to make predictions based on the environment. The system presented here is a first prototype of a cloudbased surveillance system using a shared, 3D symbolic world model; simple embedded devices that can offload computation to the cloud; mobile devices including “watch” robots; high-bandwdth wireless connections; and Kinect cameras as the sensing technology. The paper is organized as follows. First, next section presents the system model. In Sect. 3, a brief review of recent advances in the area is presented. Then, in Sect. 4 all the main elements of the prototype (world model, information storage, computation offloading) are described. To prove the efficacy of this first prototype, some preliminary experimental results are included (especially focused on communication and computation performance) in Sect. 5. Finally, future lines of work and conclusions are presented in Sects. 6 and 7. 2 System model In this paper, we propose a framework for giving support to a surveillance system via an Object Oriented World Model that is distributed among the cloud and the rest of devices. We use a 3D volumetric, object-oriented world model, called Environment Descriptor (ED), (developed by the “Control Systems Technology” in TU/e University [5, 11]) that allows the modules in the system to take the context of the environment. Such a fully offloaded cloud-based implementation of a volumetric, 3D, object-oriented world model (including semantic information) has never been carried out as far as we know. Figure 1 shows an application scenario of the proposed Cloud-based World Model platform. Surveillance tasks can be shared by several fixed devices with different hardware configurations and sensing technologies: stereo cameras, RGBD cameras, range finders, laser, amongst others. In addition, each device can be assigned to different tasks. The common case would be that of camera devices installed in different places. Meanwhile, in addition to this, there can be mobile “Watch robots”, which navigate the area or building looking for intruders. Then, as they all share the same world model, if a camera detected an intruder, then all devices would know their exact location. Consequently, “Watch robots” would transform their location coordinates to relate their own posture to that of the intruder, with the aim of pursuing him/her. Figure 2 shows a conceptual diagram of the proposed Cloud-based World Model platform. In our implementation, task organization is flexible enough so that, depending on the hardware specifications, some devices can perform all the computations on their own (Case 2) or offload computations to the cloud (Case 1). Non-sensing devices that form part of the system such as domotic controllers can receive an order and react accordingly, for instance, by locking a door if an intruder were found (Case 3). Hence, using the cloud paradigm as a natural central interface for many devices, an extremely efficient and cooperative surveillance system can be developed. A detailed explanation of the blocks of this Fig. 2 will be described in further sections. It is worth noting that in this kind of applications, concerns such as response time requirements or data safety and privacy may lead to an alternative paradigm to Cloud Computing called Edge Computing [34]. This term refers to processing the data at the edge of the network, that is, closer to the “things”, in the IoT sense, or end devices. As mentioned before, in our experiments we will use a private cloud which could be considered as a kind of edge-computing system. The reason is that this way we avoid longer and more variable delays and can focus on implementation issues such as computation offloading or environment sensing and modeling. However, along the paper we prefer to use the more general model and terminology of Cloud Computing. Furthermore, the extension of our surveillance system to a larger Smart City would benefit from a more general Cloud Computing paradigm, for instance, gathering sensing data from different buildings. 3 Related work There are numerous research works in the area of automated surveillance. In relation with this work, the most relevant can be arranged in those works that use the cloud for integrating sensor information, those that deal with the detection algorithms, those that test several sensor devices, and finally, others that study network technologies. The idea of using the cloud for integrating multiple sensor information, including surveillance information, has also been explored in the literature. Moreover, a new term has been coined to define the application of Cloud Computing (in terms of ubiquity, storage, computation, privacy, scalability, amongst others) to this area: Video Surveillance as a Service [20, 29]. For instance, in [27] the authors deal with the problem of integrating heterogeneous sensor information by creating a database scheme able to integrate different types of sensor. However, most of the available solutions rely on wired networks, such as [14, 15, 21, 36]. Only a few works have developed some aspects of cloud-based surveillance platform using wireless networks. Authors of [38] presented Vigil, where some devices with cameras execute face recognition algorithms and send the results to the cloud. They also introduce the concept of intra-cluster processing, when there is a set of cameras with overlapping visions. Compared with Vigil, the work presented in this paper takes into consideration the use of very simple devices thanks to computation offloading. In [35] a Cloud Gateway for filtering the information from multiple sensors in Wireless Sensor Networks was implemented. This filtering is applied to a basic cloud surveillance system obtaining an 83% accuracy. Most of the cloud-based surveillance systems have the following architecture: multiple cameras stream to the cloud, which is responsible of analyzing the footage in search of events. Only in the case of [14] some pre-pro- cessing is done before sending to the cloud. Compared to these works, this paper offers a more flexible approach, as the devices can decide whether to offload or not depending on their needs. Another important element of Cloud Computing is Quality-of-Service (QoS) management. In this sense, in [3] a service QoS-adaptive configuration framework for optimizing this factor in surveillance systems is proposed. Another study element corresponds to image analysis to detect intruders. Several algorithms have been devised, such as Latent Semantic Analysis (LSA), Kernel-based Fig. 1 Example of application in a building Online Anomaly Detection (KOAD) and Kernel Estimation-based Anomaly Detection (KEAD), Kernel Principal Component Analysis (KPCA) and One-Class Neighbor Machine (OCNM) [2], using machine learning techniques as well. In [25], instead of regular video cameras, omnidirectional cameras are used, which permit the surveillance of a whole room or a bigger open space with only one camera. In [12], the sensor used is that of Microsoft Kinect, with satisfactory results. In this sense, the system presented here also makes use of Kinect Sensors which, at a reasonable price, provide a depth channel, extremely important for object detection and tracking. As it can be seen, most of the surveillance systems available in the literature normally use fixed cameras. In this sense, one of the major assets of this work is the collaboration of different heterogeneous devices (such as mobile “watch” robots) thanks to the use of a shared world model and the adaptation of the cloud to each robot’s needs. Moreover, the use of mobile robots brings some interesting opportunities and challenges for more complex surveillance strategies and systems. The approach of using an edge-computing platform and wireless technologies is becoming increasingly attractive to researchers on surveillance systems [6]. For instance, [16] shows that an edge node can reduce the reaction time between edge and sensors, and can improve network stability as well by uploading only necessary data. Also, [23] improves the optimal bandwidth distribution and allocation in a video surveillance system using up-to-date wireless technologies. In the present paper, we follow this approach with the additional contributions of implementing a 3D, object-oriented, cloud-based world model that includes semantic information, as well as allowing the use of simple embedded devices and mobile “watch” robots through computation offloading. 4 Implementation of the platform In this section, the implementation of the Cloud World Model based Surveillance Platform is detailed. The implementation of the Surveillance Platform consists of the following parts: –– Definition of the elements of World Model (Sect. 4.1) ––Definition and deployment of a World Model Storage System (Sect. 4.2) ––Implementation of the Sensor Integration Module, responsible of processing the image streams (Sect. 4.3). 4.1 Elements of the world model World modeling is the process of creating a model of an environment (with its inherent complexity), which is a key task for intelligent systems. The more accurate the representation of the environment, the more successfully a task will be performed. Hence, as stated in Sect. 1, adding semantic information to each element of the environment implies a dramatic increase in accuracy. In the implementation shown in this paper, the main elements of the world model (simplifying from [5, 11]) are entities and transforms. Entities: An entity is an element of the world. It has the following properties: Fig. 2 Conceptual diagram of the cloud-based surveillance system ––ID: Unique identifier for an entity. No other entity can have the same identifier, even if the entity was removed. ––Type: Specifies the type of the entity. This allows to make classifications or create subsets of entities for different actions. For example, several actions can be done depending if the entity is a person, a furniture or a wall. ––Pose: Establishes the position and orientation of the entity in the world model. These coordinates are absolute. ––Shape or convex hull: Adds volumetric information to the entity. This shape can either be real or virtual. An entity can have both. ––Measurements: Information obtained by sensors that is associated to an entity in the world model. These measurements hold the sensor data and the time-stamp. Transforms: defines the spatial relationship between two different entities in the world model. Multiple transformations for each parent–child pair can be stored for different timestamps which results in a graph of entities and transforms. 4.2 Cloud world model storage As stated in previous sections, a collaboratively built world model offers several advantages. However, there are some bottlenecks that the cloud solution must address. First, if the number of robots increases (and therefore, the number of Database read/write operations), the cloud might not be able to satisfy all queries in a reasonable time. To make matters worse, if the number of entities increases, the cloud resources assigned to the database may run out. Therefore, one of the main objectives of the cloud solution is to adapt computing resources at runtime depending on the needs. To satisfy these requirements, Distributed Database Management Systems (DDBMS) are the perfect candidates for storing all the information, thanks to their massively parallel nature, ease of use and portability. Among all the available options, one of the best DDBMS paradigms is that of Bigtable. Its main advantage is that it uses a sparse, distributed and multidimensional sorted map, using the following data organization [8]: (row:string, column:string, tag:string, time:int64) → stringHence, there would only be one (or two) big tables that can be perfectly scattered among different containers in the cloud. Therefore, if the aforementioned bottlenecks occur, then more containers can be spun up at runtime. On the contrary, if the database is less loaded, then the cloud can scale back, thus saving resources for other tasks. Having in mind this DB model, the next schema was designed to fit with the needs of a surveillance system. The idea is not only to store the current information about entities, but also their history. In this sense, we use the concept of delta, that is, a change in a certain entity (following the so-called event-driven mechanism [17]). To clarify the concept, an example follows (depicted in Fig. 3): suppose two robots A, B and an entity E. Both robots have synchronized their local world models at an instant t, finding that the most recent change was done at t−1 (“Last time” in Fig. 3). Therefore, they know all the information about E (shape, pose, type, etc.). At t+1 , B registers changes in the pose of E, and therefore will update the world model with this new information. If nothing but the pose has changed, re-sending the information about the shape would be a waste of bandwidth. Therefore, B will only send the change (that is, the delta) of the pose. This same reasoning can be applied to A, which is only interested in the most recent changes (deltas) since database’s most recent time-stamp t−1 (to save resources). Thanks to the use of timestamps, a robot can query all the new deltas since a given time t. Taking this into account, there will be a table of deltas in the database with the following columns: pose, type, shape, convex hull and deleted (whether the entity was cleared or not). The information will be stored in the JSON (JavaScript Object Notation) format. The row ID will correspond to the entity’s ID. Among the available implementations of the Bigtable model, Hypertable has been chosen because it is written on C++ and promises higher performance results compared to competitors such as HBase [1]. This is of great importance when working with robots, and in some cases, near real-time conditions. The elements of a Hypertable system that were configured for our system are [19]: range server (for handling the read and write operations in the table), Master node (for creating/deleting tables), ThriftBroker (interface between the database and the clients) and FS Broker (normalized filesystem interface). Figure 4 on the left shows the implementation of the world model using Hypertable (the FS Broker is not shown, as it is being used in all the nodes). On the right, an example of adaptation when a bigger number of robots demands more resources has been depicted. In this case, more instances of Range servers and ThriftBrokers are spin up by the cloud to satisfy this new load. 4.3 Sensor integration modules The sensor integration offloading implemented here is a simplified version of [5, 11]. The workflow is depicted in Fig. 5. All modules are implemented using the Robotics Operating System (ROS) Framework. ROS is an Open Fig. 3 Example of a delta update Fig. 4 Implementation of the cloud World model using hypertable Source Robotics Software Framework (using C++ and Python) that allows the implementation of advanced distributed architectures, very similar to Multi-Agent systems [18]. Thanks to the use of topics, services, peer-to-peer communications and compression techniques, ROS makes the offloading process simple and scalable. Main processes (nodes in ROS convention) correspond to those modules of Fig. 5, while connecting links of these modules have been implemented using ROS topics and services. ––The different operations done in the sensor integration are the following:RGBD Masking: This process will only use the D channel of the stream. It is responsible for detecting entities using feature extraction (producing 3D Point Clouds), association (comparing with a rendered world model) and segmentation (creating new entities from the unassociated data) [5, 11]. Figure 6 shows an example. This module is one of the most computationally demanding. In our case, as we are dealing with surveillance cases, this module will discard those entities whose size is not big enough to be a person. ––Sensor fusion: This process receives the RGBD stream and the aforementioned entities with mask measures. The idea is to use the previously unused RGB channel and extract other properties from the entities. Therefore, the module must buffer the RGBD stream until its related masked measurement arrives. Figure 7, using the mask from Fig. 6, now obtains an RGBD measurement (which, for example, contains color information). ––Perception: This process receives the RGBD measurements (created by the Sensor Fusion module) and performs an analysis to discover the type of the entity. This will be the responsible module for detecting whether an entity is an intruder or not. As our aim is demonstrating the cloud working and feasibility, in the experiments presented in this paper the system only detects whether the entity is wearing or not an uniform. If the person was not wearing it, then it would not be an authorized personnel. On commercial surveillance systems, it is obvious that a knowledge base (with templates) must be used. For the previous example, Fig. 8 analyzes the RGBD measurement and concludes that the entity is an authorized person. The presented platform is aimed to be as much flexible as possible. In this sense, a robot may not have enough processing power to do all the sensor integration on-board but sufficient to do part of this processing. Therefore, the robot should be able to choose which parts of the sensor integration are offloaded. To implement this feature, the system was implemented using modules. For instance, one instance may only have the Perception module (and thus Fig. 5 Workflow of the Sensor Integration model Fig. 6 Example of a mask measurement only perform recognition operations), whereas another node may have all modules. As a consequence, several offloading options can be taken into consideration. 5 Preliminary results These preliminary tests are aimed to show the efficacy and viability of not only having a shared world model, but also of using wireless technologies. The tests are divided into three categories: 1. Real experiment in a testing environment. 2. Queries in the world model. 3. Sensor integration offloading. All the tests were done in a private Cloud with 5 nodes (1 front-end node and 4 computing nodes). Each node has a AMD FX-8320 octa-core CPU (with virtual extensions enabled) and 8 GB of RAM. They are all connected internally using Gigabit Ethernet bandwidth. For the cloud middleware, we decided to make use of Kubernetes + Docker (recently adopted by major cloud vendors, such as Google Container Engine or Amazon EC2 Container Engine) instead of traditional VM-based private clouds (such as Openstack or Eucalyptus). One of the main reasons behind this decision was to avoid the overheads derived from the virtualization of hardware and operating system layers [33]. 5.1 Proof‑of‑concept experiment A few preliminary experiments were devised to check whether the whole system works properly. For this purpose, only a few set of objects were modeled for an indoor environment; mainly rough shapes of the furniture and two people (one intruder and one authorized person). In one of the offices, two camera devices were installed to detect intruders Fig. 7 Example of a RGBD measurement Fig. 8 Example of perception