Enhancing smart environments with mobile robots

Enhancing smart environments with mobile robots Francisco-Angel Moreno, Cipriano Galindo and Javier Gonzalez-Jimenez Machine Perception and Intelligent Robotics Group, System Engineering and Automation Dept., University of M´alaga, Campus de Teatinos, 29071, M´alaga, Spain, {famoreno,cgalindo,javiergonzalez}@uma.es Abstract. Sensor networks are becoming popular nowadays in the development of smart environments. Heavily relying on static sensor and actuators, though, such environments usually lacks of versatility regarding the provided services and interaction capabilities. Here we present a framework for smart environments where a service robot is included within the sensor network acting as a mobile sensor and/or actuator. Our framework integrates off-the-shelf technologies to ensure its adaptability to a variety of sensor technologies and robotic software. Two pilot cases are presented as evaluation of our proposal. Keywords: sensors networks, smart environments, mobile robotics 1 Introduction The interest for sensor networks [7] is growing nowadays, opening new applications in different fields, like Ambient Assisted Living (AAL) [16, 9], health monitoring [15], and smart homes [12]. Specially with the expansion of the socalled Internet of Things (IoT) [17], small devices have appeared in a number of scenarios, e.g. houses, workplaces, and warehouses, to provide assistance to the users’ daily tasks. Typical sensor networks operate controlled by a central computer that is in charge of receiving, processing and logging the information provided by the sensors, hence becoming the element where the intelligence of the system resides. In general, the interaction between the smart system and the user is limited to the exchange of basic information, either by voice or through a visual display as, for example, in the cases of commercial devices like Amazon Echo [10] or Google Home [1]. Moreover, home automation systems can also perform changes on the environment through a variety of static actuators (e.g. switching on/off or dimming the lights, lowering/rising the blinds, etc.). However, less attention has been paid to the inclusion of a mobile robot in the environment, which can act as an additional, mobile, networked element, adopting the role of both a sensor and an actuator. This significantly enhances the interaction capabilities and Work supported by the research projects TEP2012-530 and DPI2014-55826-R, funded by the Andalusia Regional Government and the Spanish Government, respectively, both financed by European Regional Development’s funds (FEDER). 2 Fig. 1. Framework overview. provides the smart system with a variety of new interesting services. Robotic assistance ranges from companion for elder people at home, mitigating their loneliness and monitoring their health parameters, to helper in work environments by providing different services such as agenda management, reminders, pick & carry operations, etc. The integration of the service robot within the system involves endowing the robot with the capabilities of being remotely controlled and providing feedback about the on-going tasks. For this, a communication scheme must be defined to interface the robot. Besides, as it will share its working space with humans, a considerable effort in terms of robustness, flexibility and safety must be done regarding its motion in a possibly dynamic environment. This work describes a framework that conveniently integrates a service robot within a sensor network and presents different scenarios in an office-like environment that proves the suitability and the utility of the system. The presented system uses off-the-shelf technologies for i) building a network based on wireless sensors, ii) implementing the robot control architecture aimed at endowing the mobile robot with the needed abilities, and iii) setting up a middleware that communicates the sensor network and the robotic architecture. By relying on a widely employed message protocol (MQTT) we ensure that the different elements of our framework can be easily replaced with other sensor technologies or robotic architectures. 2 Framework overview This section describes the four pillars of the proposed framework (see Fig. 1), namely: i) the sensor network, ii) the robotic architecture, iii) the communication system and iv) the rule manager. Sensor network. A network of sensors provided by Tunstall [5] has been considered in this work. Tunstall provides solutions to the elderly and handicapped people based on wireless sensors and smart technologies. 3 From the variety of different sensors it provides, we have installed in our lab the following: 4 door contacts, 6 chair occupancy sensors, a power sensor and a presence sensor (PIR) (see Fig. 1). When triggered, these sensors send a wireless signal to the Tunstall manager device, which is in charge of collecting and sending the sensor ID and status to a computer through a standard USB port. The maximum number of sensor devices that can be linked simultaneously to the manager is 35. Robotic Architecture. The mobile robot integrated in the framework is the robot SANCHO [11], built upon a Pionner base and endowed with two laser scanners, a RGB-D camera, and speakers. The robotic architecture considered is an instance of the OpenMORA architecture [2], based on MOOS [14]. Coded in C++, OpenMORA considers a centralized blackboard from which a number of modules that implement the robot capabilities can share information by publishing and subscribing to particular topics through TCP sockets. Some relevant robot capabilities implemented in Sancho are self-localization [8], reactive and global navigation [13], and Text-to-Speech. Note that ROS [4] could also be considered, albeit we prioritized the cross-platform feature of OpenMORA. Communications. We rely on the MQTT (Message Queuing Telemetry Transport) protocol, which is a lightweight messaging protocol based on a publishand-subscribe architecture. It implements a star topology with a central node called broker which broadcasts the incoming messages to the clients according to the topics they are subscribed to. Either encrypted and non-encrypted communications are supported and three levels of quality of service (QoS) are provided as additional features. Moreover, its small footprint and low-bandwidth requirements makes it ideal for wireless networks and devices with limited resources such as embedded systems or smartphones, becoming a popular protocol for Internet of Things (IoT) nowadays. Along the several MQTT brokers available, e.g. HiveMQ, CloudMQTT or Mosquitto, we have opted for the last one, given its compatibility and popularity. Rule management. The rule manager is in charge of taking decisions about the robot operation according to the status of the sensor network. This has been tackled using OpenHAB [3], a Java-based, open source software that integrates different systems, sensors and technologies and provides the capability of creating rules to automatize domotic environments following the Xtend [6] syntax. For instance, a rule can be created to automatically switch on the lights and warm the stove up for dinner. The use of rules and the OpenHAB capability to be MQTT-compliant, make it an ideal element to become a part of our framework so that it can collect messages from all the different devices, analyze the state of the complete system and generate a certain command for the robot according to it. Besides, it provides a web interface that shows the status of all the involved sensors, hence becoming an interesting tool for monitoring an environment. Framework operation. When a sensor is triggered, a radio signal is sent to the Tunstall manager device, connected to a PC running an OpenMORA module 4 Corridor PIR Door sensor Chair sensor Power sensor Room 1 (a) (b) Initial position Re-planning Room 2 1 2 Pilot case #1 Pilot case #2 navigation navigation closed door re-planning found new path navigation to target initial position PIR sensor PIR face recognition & agenda #1 #2 Fig. 2. (a) Environment floorplan and position of the sensors: PIR (orange), door sensor (blue), chair occupancy sensor (green) and power sensor (red). Robot trajectories for the pilot cases are also depicted. (b) Illustration of pilot case #1 (top) and #2 (bottom). that converts the signal to MQTT messages associated to a specific topic. The broker broadcasts the message to the subscribers of the topic, including the rule manager, which operates in a different PC. The rule manager analyzes the status of both the sensor network and the robot, and decides a proper command that is sent to the robot through a MQTT topic. Finally, OpenMORA modules running on the robot’s computer capture such command, and execute it. Note that we have chosen to distributedly run the described subsystems in different computers in order to balance the system’s computational burden. However, it would be possible to run all the elements in the same computer. 3 Pilot cases To evaluate our framework in real conditions, we have designed two pilot cases aiming to prove the potential of the combination of sensor networks and mobile robots to provide ambient intelligence services. Fig. 2(a) sketches the chosen 5 Door sensor Robot OpMORA robot OpenHAB OpMORA logger Tuns. manager PIR sensor time PLAN RULE NAVIGATION NAVIGATION RE-PLAN RULE RULE cancel nav. trigger trigger trigger trigger trigger trigger start navigation start navigation start navigationstart navigation cancel navigation Tunstall trigger process OpenMORA/MQTT conversionOpenHAB rule execution OpenMORA navigation planning Robot operation Fig. 3. Timeline for pilot case #2 . scenario composed of an office-like environment with two connected rooms and a near corridor. Pilot case #1. In this case, when a chair occupancy sensor is triggered, the rule manager sends a navigation command to the robot ordering it to move to the sensor location. Then the robot uses its camera to detect the person on the chair and recognize his/her face. Subsequently, the rule manager retrieves from the user’s agenda all the relevant information and informs about it to him/her by voice. Finally, the robot returns to its initial position and waits until a new sensor is triggered. If the person who triggers the sensor is not recognized by the robot, an alarm is raised informing the lab staff about the situation. A sequence of images illustrating this scenario is presented in Fig. 2(b, top). Pilot case #2. This pilot case demonstrates the capabilities of combining realtime information from the networked sensors and the mobile robot in terms of flexibility and adaptability when performing some tasks. In this case, we have installed a PIR sensor in the corridor outside our lab, which is triggered each time a person walks along it. When activated, the system sends a command to the robot to inspect the corridor by launching a navigation command. This command is processed by the robot architecture by finding the shortest path from the current position to the target and elaborating a planned sequence of intermediate points. Once the robot starts navigating, we manually close one of the doors between the laboratory and the corridor, hence intercepting the planned path of the robot. Thus, the triggered door sensor activates a rule in the system that handles this blocked-trajectory situation by sending a signal to the robot in order to abort the current navigation and to re-calculate an alternative path to the target. Once this has been performed, the robot re-starts its navigation by trying to enter the corridor using an alternative door. Should both doors be closed, the system will command the robot to ask for help to the person closest to the door who, according to the chair occupancy sensors, is actually present. This scenario is illustrated in Fig. 2(b, bottom) while its timeline can be found in Fig. 3. Two videos showing the robot operation in both pilot cases can be found in http://mapir.isa.uma.es/work/smart-environments. 6 4 Conclusion This work has presented a framework that enhances a sensor network with a mobile robot to create a versatile smart environment with increased capabilities in terms of provided services and interaction with the user. By using off-the-shelf technologies and the standard MQTT communications protocol we ensure the adaptability of the framework to different sensor technologies and robotic architectures. In particular, our proposal relies on Tunstall sensors, OpenMORA as robotic software and OpenHAB as rule manager. The capabilities of the complete system have been evaluated through two pilot cases. References 1. Google Home website. https://home.google.com, accessed: 2016-06-01 2. MAPIR website. http://mapir.isa.uma.es, accessed: 2016-06-01 3. OpenHAB website. http://www.openhab.org, accessed: 2016-06-01 4. ROS website. http://www.ros.org, accessed: 2016-06-01 5. Tunstall website. http://www.tunstall.com, accessed: 2016-06-01 6. Xtend website. http://www.eclipse.org/xtend/, accessed: 2016-09-05 7. Akyildiz, I.F., Su, W., Sankarasubramaniam, Y., Cayirci, E.: Wireless sensor networks: a survey. Computer networks 38(4), 393–422 (2002) 8. Blanco, J., Gonz´alez, J., Fern´andez-Madrigal, J.A.: Optimal filtering for nonparametric observation models: applications to localization and slam. The International Journal of Robotics Research (2010) 9. Coradeschi, S., Cesta, A., Cortellessa, G., Coraci, L., Galindo, C., Gonzalez, J., Karlsson, L., Forsberg, A., Frennert, S., Furfari, F., et al.: Giraffplus: a system for monitoring activities and physiological parameters and promoting social interaction for elderly. In: Human-Computer Systems Interaction: Backgrounds and Applications 3, pp. 261–271. Springer (2014) 10. Dempsey, P.: The teardown Amazon Echo digital personal assistant [Teardown Consumer Electronics]. Engineering & Technology 10(2), 88–89 (2015) 11. Gonzalez, J., Galindo, C., Blanco, J., Fernandez-Madrigal, J., Arevalo, V., Moreno, F.: SANCHO, a fair host robot. A description. In: Mechatronics, 2009. ICM 2009. IEEE International Conference on. pp. 1–6. IEEE (2009) 12. Han, D.M., Lim, J.H.: Design and implementation of smart home energy management systems based on zigbee. IEEE Transactions on Consumer Electronics 56(3), 1417–1425 (Aug 2010) 13. Jaimez, M., Blanco, J.L., Gonzalez-Jimenez, J.: Efficient reactive navigation with exact collision determination for 3d robot shapes. International Journal of Advanced Robotic Systems 12 (2015) 14. Newman, P.: MOOS — A mission oriented operating suite (tech. rep. oe2003–07). Department of Ocean Engineering, MIT, Cambridge, MA:, Tech. Rep (2003) 15. Pantelopoulos, A., Bourbakis, N.G.: A survey on wearable sensor-based systems for health monitoring and prognosis. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 40(1), 1–12 (2010) 16. Sun, H., Florio, V.D., Gui, N., Blondia, C.: Promises and challenges of ambient assisted living systems. In: Information Technology: New Generations, 2009. ITNG ’09. Sixth International Conference on. pp. 1201–1207 (April 2009) 17. Xia, F., Yang, L.T., Wang, L., Vinel, A.: Internet of things. International Journal of Communication Systems 25(9), 1101 (2012)