An architecture for the integration of robots and sensors for the care of the elderly in an ambient assisted living environment

robotics Article An Architecture for the Integration of Robots and Sensors for the Care of the Elderly in an Ambient Assisted Living Environment David Loza-Matovelle 1,*,†, Alexandra Verdugo 1, Eduardo Zalama 2and Jaime Gómez-García-Bermejo 2 1Departamento de Ciencias de la Energía y Mecánica, Universidad de las Fuerzas Armadas ESPE, 170501 Sangolqui, Ecuador 2ITAP-DISA, Universidad de Valladolid, 47002 Valladolid, España *Correspondence: [email protected] † Current address: Universidad de las Fuerzas Armadas ESPE, Av. General Rumiñahui S/N, Sector Santa Clara, Sangolquí Valle de los Chillos. Received: 15 July 2019; Accepted: 28 August 2019; Published: 3 September 2019   Abstract: In this paper, the development of an assistance system for the elderly that combines robots with a network of sensors and actuators is described. The architecture was developed with the aim of interaction. With this reason, the system incorporates particular solutions that are adaptable to the needs of the user, such as a pyramid interaction system, a telepresence robot, a biometric bracelet, and others. In the software, the system is composed of two servers: local and web. The local server is in charge of different modules that interact with direct users and their needs. The web server provides different facilities to external users to access the system remotely. A use case is analyzed where the proposed system is validated. Keywords: robot architecture; telerobotics; human-machine interaction 1. Introduction According to [ 1 ], people who reach an advanced age usually need care such as emotional support and well-being, mood, quality of sleep, social activities, cognitive functions, personal hygiene, healthy eating, and regular medical check-ups. An important part of the care of the elderly is the monitoring of their vital signs. Consequently, it is very complicated to monitor a person 24 h a day. Besides, many people with advanced age are prone to fall, and may not have a person who can help them in these cases immediately. In that regard, it could be derived that domotic, telecare systems, among others, facilitate the independent life of the elderly in their homes. Examples of this technology can be found in [ 2 – 4 ]. However, insufficient progress has been made in the area of improving the social and emotional well-being of these persons. In this context, robotics is shown as part of the solution for the care of the elderly in their homes. In particular, social robotics is an effective complement to home automation systems, especially in the case of the elderly, as they have difficulties using technological devices. The integration of robots and advanced interfaces, together with other domotic devices, allows reducing the technological gap of older people to the use of new technologies. These resources can be used together in the framework of adequate planning, oriented to satisfy the needs of the users. In this context, there are different projects, especially from the European Community, where the care of the elderly through robots and IoT is approached. To mention some projects: TERESA (Telepresence Reinforcement-learning Social Agent), is a project of the University of Oxford and consists of a telepresence robot with social characteristics connected to an intelligent environment [ 5 ]. Robotics 2019,8, 76; doi:10.3390/robotics8030076 www.mdpi.com/journal/robotics Robotics 2019,8, 76 2 of 17 Another project recently closed in 2018 is MARIO (Managing active and healthy aging with use of caring service robot), in which social assistance robots are developed in conjunction with the geriatric evaluation whose proposal uses the Kompai platform [ 6 ]. The RADIO project (Robots in assisted living environment, unobtrusive, efficient, reliable and modular solution for independent aging), focusses the development of robots that assist in conjunction with an AAL the needs of the elderly, in turn, tha effectively monitor vital signs [7]. On the other hand, the idea of having intelligent environments and robots in the fields related to health and medical care has been widely considered in the literature. According to [ 8 ], you can perceive certain common problems within such projects, such as unreliability, lack of human interaction, high costs, lack of experience when using, etc. The aforementioned problems reflect that a deficit persists concerning previous studies (some studies have begun to study how healthcare robots are perceived by professionals [ 9 ]), so it is important to analyze the potential of service assistance robots with IoT environments from the perspective of real satisfaction of the user’s needs. The main contribution of this article is an architecture and assistance services for the elderly that integrates home automation and robotic systems-oriented into interaction. The remainder of this document is structured as follows: Section 2deals with IoT structures and robot, in addition to different technologies related to the proposed work; Section 3presents the concept of the system; Sections 4 and 5refer to the hardware and software developed for its implementation; Section 6analyzes the operation of the system; and finally Section 7presents some evaluation and results. 2. Technologies Related to Smart Environments Some trends can be identified within different works related to the focus of this work. They are focused on the areas of health, safety, social interaction, and physical condition. Subsequently, a brief analysis of the technologies developed in these areas is presented. Inside the technologies of assistance to the elderly, those that provide support to health have a priority. These technologies are classified as those that can respond to emergencies, detect falls and monitor physical variables. A variety of technologies are described and compared in [10]. Security indoors can be understood as a great variety of alternatives among which the control of alarms and novelties, devices of doors and beds, etc. can be mentioned. Moreover, the monitoring of activities can help the early detection of cognitive or physical deterioration. Studies and examples of these systems can be found in [11]. Social participation is also a key element. There are two main branches of the technologies oriented to this participation: the robots dedicated to taking care and the robots that facilitate interaction (such as telepresence robots). An example of the first is PARO ([ 12 ]), a baby seal robot with different sensors (touch, light, audio, temperature, and position which uses to feel its location and perceive where users speak) that allow it to interact with the environment and people. As for the robots that facilitate social interaction, we can mention the Giraffe [ 13 ] robot, which incorporates an interface for communication with family members or doctors, and also provides a system that allows monitoring the patient’s activities. In some elderly people, the physical condition is reduced. Technology can help compensate for this deficit through daily activities or provide auxiliary functions in physical rehabilitation (through walkers or therapy elements). In fact, as for the use in daily activities, domestic robots can help in common household tasks (cleaning, cooking, etc.). Some examples of these types of robots are the Roomba [ 14 ], Moley ([ 15 ]), etc. Regarding support robots, we can mention the LEA project ([ 16 ]), which is a robotic walker that provides stability and support through a posture detection system. In all the technologies described above, numerous factors must be taken into account for acceptance by the user. According to certain models and experiences such as [ 17 – 19 ], the main elements to take into consideration are functionality (perception of use), usability (ease of use), Appearance, safety, cost, and reliability. Robotics 2019,8, 76 3 of 17 3. Concept of the Proposed System In this section, the initial idea of the proposed solution with its different characteristics is analyzed. Furthermore, the general structure of the system is described, as well as its main elements, both local and on the web. 3.1. Initial Idea The development of an assistance system that combines autonomous robots with sensors and external actuators forming an intelligent network managed by a planner is proposed. The robots add to the traditional IoT architectures different aspects to be considered: on-site validation through the robot, better interactions with users, more complex operations, flexibility, and recovery, among others. The proposal is integral in that the core of the interaction is the user, taking into account the user’s physical and social needs. We seek to promote the independence and safety of the users through a natural and intuitive interaction, non-invasive in terms of controls. Another aspect to comment on the proposed structure is that resources work in a decentralized manner: robots within the system that can perform their own actions without the need for external resources, but these actions are being suitable, planned, monitored and analyzed. This allows the system to make changes dynamically in the plans or activities, even allowing the user to make certain modifications in the plans, change states within the system, etc. It also incorporates a web management system that allows tasks to both local and external users remotely, in activities such as video calls, vital signs review, among others. Figure 1shows the system concept where the user’s interaction can be seen with different elements of the system. It is based on a heterogeneous network of sensors and actuators which has an IoT structure. The relationship of the robot that can interact with both the environment and the user can be also seen. And finally, it is understood that the services of the network for both direct and indirect users are present within the architecture. Figure 1. Concept of the proposed system. The system has three important characteristics: modularity, adaptability, and integrality. The system is modular because it allows the integration of a wide variety of devices. These modules Robotics 2019,8, 76 4 of 17 can be responsible for health, communications, and interaction, among others. For example, you have a biometric bracelet that is responsible for measuring certain physical variables such as heart rate, temperature among other variables. Regarding the adaptability of the system, it is manifested in the fact that it is not a rigid structure but rather focuses on the individualities of the users. To achieve this, the system periodically evaluates the actions carried out either through metrics (response time, activities carried out successfully, detection of dangerous situations, among others), surveys or external evaluations. This information allows the system to generate customized solutions depending on the tastes and needs of the user. A clear example of the adaptability of the system is that you can have different user profiles (where you can change the number of users, activities, calendars, etc.) and modes of use (surveillance, vacations, among others). Finally, the system proposed intends to have integrality, because it does not focus solely on a specific need of the older person but presents different solutions within the biological, psychic and social aspects. 3.2. General Structure of the System The complete architecture of the system can be seen in Figure 2. The system is composed of two servers: one at the local level, in charge of the needs of the direct user and another with web services, which is used to provide different facilities to external users. Also, in Figure 2it can be seen different elements that interact in the local server through the users, such as sensors, robots, interaction systems, etc. These elements can have their structure and code but share information about their states and certain properties such as location, activities in execution, etc. Figure 2. Proposed architecture. On the other hand, it is important to mention that indirect users can interact with the system through different equipment and platforms (phones, tablets, dedicated applications, etc.) that Robotics 2019,8, 76 5 of 17 allow a more natural and effective communication through the webserver. For this, a set of tools has been developed that allows the communication between web-user (web-interpreter) and intelligent-user-environment (user-interpreter). In its operation, the interpreters take this information from specialized nodes installed within each device that is developed according to a set of standard protocols (MQTT, COAP, etc). The information shared with the system can be varied and ranges from global variables to energy levels or failures. In Section 7we work with an example of the capabilities of the system. It concerns when a resident in their daily routine will have different needs. The modules of the system and their different execution are presented in the use case. 4. Sensors and Actuators Network The sensor and actuator network implemented in the system consists of a set of devices that can be connected to the local server through a protocol. Through this network, the sending and receiving of information are managed, where the data can be transferred from multiple nodes through a gateway. The devices are implemented for the sensing and control of lights, blinds, doors, and temperature. The information captured by the sensors is sent to the acquisition card and through the wireless network; they also send data to the local server. This central server manages the information and, based on the received data, sends it to the devices that require it, actuators or services within the system. For communication between devices, the MQTT “Message Queuing Telemetry Transport” [ 20 ] protocol is used, which allows sending and receiving information through a central intermediary. Once the information is sent to the cloud, it can be viewed by the users who have access to it, whether by a computer, phone, smartwatch or other intelligent devices that have access to the system. 4.1. Network Design To work with the proposed network, the design of an IoT architecture that allows the inclusion of robots and different devices need to be designed. There are IoT solution providers. However, the purpose of the research is to propose an own architecture oriented to the sensor network, the robot, and the system’s own devices. An IoT architecture is composed of the following elements: devices (sensors and actuators), communication (protocols), cloud services and layers (administration, security, application). The scheme shown in Figure 3is proposed, where different components that conform to the system are shown: • Devices: This section shows the different sensors and actuators that are part of the IoT network proposed for the AAL that includes: control of lights, blinds, door opening, and temperature. • Communication: For communication between devices it is important to define the network and protocol to be used, in this case, the implementation of the MQTT protocol that works with the WiFi wireless network under port 1883 is proposed. • Cloud services: The services provided by the cloud and that will be used for the implementation will be oriented to the management of the information acquired by the sensors through storage and database. • Applications: Are those that allow direct contact with the end-user. That is why among the applications of visualization NodeRed is used, which will allow the control and monitoring of the devices, as well as a chatbot created in Dialog Flow for the interaction of people with a virtual avatar projected on the holographic pyramid. The robot can connect to the Internet-of-things, either in passive or active mode. In passive mode, the robot is not connected to the Internet but can be identified uniquely through an RFID tag. Other things connected to the Internet with RFID reading capabilities can identify it and publish information related to the robot, for example, robot location information. In an active mode, the robot is connected to the Internet, which allows the traffic of sending information in real-time through Internet services. Robotics 2019,8, 76 6 of 17 Figure 3. IoT Scheme. 4.2. Comunication Once the devices are defined, the next step is to define the means for communication, the first step is the selection of the network, which must be wireless since a wiring system would be anti-aesthetic and impractical for implementation. WiFi is the selected network for acceptance in households. Concerning protocol, as previously mentioned, there are different protocols for IoT applications. However, the selected protocol has been MQTT due to the advantages it presents for its implementation. MQTT consists of a messaging protocol based on a system of publications and subscriptions, in which the devices that are part of the architecture send and receive data through a central server or broker that is responsible for managing the information. 5. Hardware As mentioned in the introduction, there is a diversity of needs of the characters, who have their own particularities. Due to this fact, some special elements with specific functions have been designed, constructed and implemented. Figure 4shows three of these elements: a biometric bracelet, an interactive pyramid, and a telepresence robot. This is due to a network of sensors and actuators distributed by the environment explained in the previous section. All of them have valuable information about the environment that after receiving and processing it, according to each device, it is sent to the local server through the ROS-interpreter. These elements are described below. Robotics 2019,8, 76 7 of 17 Figure 4. System hardware ( a ) Prototype of biometric bracelet; ( b ) Interaction pyramid and ( c ) Telepresence robot. 5.1. Interactive System-Holographic Pyramid The holographic pyramid proposed in [ 21 ] is used as a channel of interaction with the elderly person. Its operation is based on the effect of Pepper’s ghost technique and an animated avatar is projected. It is an interactive system, easy to replicate, and with which a natural interaction can be achieved. It is basically composed of three subsystems: the holographic retroprojection subsystem that uses the pyramid as a means of projection, the subsystem of gestures that is responsible for the different expressions and visemes of the avatar and finally the interaction subsystem that includes the modules in a vision set, chatbot and others. The pyramid has been integrated into the proposed architecture and provides a natural interaction through voice commands. The holographic pyramid receives information obtained from the interaction between the avatar and the user, which is connected to the system through ROS (“Robot Operating System”) [ 22 ]. This avatar is in charge of making the direct contact of the system with the user and through which a direct link can be established between the two. The information is received by the pyramid through the camera and the microphones that the device has, to be processed and converted into appropriate data that the interpreter can process. 5.2. Telepresence Robot The telepresence robot developed in [ 23 ] has dimensions of 500 × 500 × 1500 mm and a weight of 35 Kg. The robot has a mobile platform of differential configuration, a system of height regulation and a mechanism of two degrees of freedom for the screen, to improve the user’s visual field when making video calls. It also has an infrared sensor system to avoid possible collisions with its surroundings. Robotics 2019,8, 76 8 of 17 The system can self-locate within the environment, avoid obstacles and be teleoperated in a simple way. Also, its features include face recognition [ 24 ] that allows the robot to focus on the user during the interaction. It has a system based on IP addresses that allows knowing its approximate location(as long as it is connected to the webserver). In summary, the telepresence robot is built in a differential platform that integrates other parts like a microphone, screen, and camera. Also, It can be located in its surroundings, teleoperate remotely through the internet in safe mode, avoid obstacles and make calls through its interface. Its integration into the system is simple because of its software is developed in ROS. The telepresence robot is connected to the system through the ROS-interpreter to which information obtained by its sensors, such as obstacle detection, request for video calls, facial recognition, among others, is sent. The information received from the environment or the user is converted into adequate information to be sent to ROS-interpreter. Also, the robot through the ROS-interpreter sends the necessary information so the actuators can start operating and performing the actions for which it was implemented as moving forward, back and side, as well as rotating in both directions and allowing adjustment of its screen on the vertical axis. 5.3. Biometric Bracelet The biometric bracelet consists of a device for monitoring and interpretation of vital signs of elderly people, which is integrated into the system through Openhab [ 25 ] according to [ 26 ]. The device can monitoring and interpreting vital signs such as body temperature and heart rate. It has a fall detection system and automatically emits emergency alerts and, if necessary, sends an alert by pressing a button located on the top of it. It transmits information gathered wireless using the MQTT communication protocol. For the detection of falls, the behavior of the angular acceleration in the x, y, z axes of a gyroscope and accelerometer in relation to the reference system of a standing person is studied. This device was created with an ergonomic structure made by additive manufacturing methods. Besides, the energy autonomy in continuous operation is about 15 h. The used electronic circuits are in the smallest possible size obtaining an embedded and non-invasive system which uses free software for programming. The biometric bracelet is connected to the system using ROS-interpreter to which is sent the information obtained through their sensors. Upon receiving the data, this is processed through the internal logic of the interpreters and converted into adequate information for the user’s interpretation. 6. Software As shown in Figure 3, the system has two servers: one at the local level (communication with the elderly) that controls all the in-situ elements and an online system (communication with relatives, doctors, etc.) that manages the service point. Below is a description of the different elements of the system. 6.1. Local Server The local server (Figure 5) is a set of modules that can perform the following activities: receive requests from the environment, plan different activities, manage system resources and make requests to web services, among others. The functions of the local server can be decomposed into three groups: social interaction (user status, reward system), organization (ROS scheduler and interpreter) and performance (performance monitor and knowledge base). The elements that make up the local server are ROS-interpreter-, planner, knowledgebase, performance monitor, user status and reward system. These elements are described below. Robotics 2019,8, 76 9 of 17 Figure 5. Structure of the Local Server. 6.1.1. ROS-Interpreter ROS-Interpreter allows the communication of the different devices connected locally with the server in situ. The system works by the connection between nodes both in the computers and in the local server, these nodes specify several elements within which we can specify three types: head, internal services, and external services. The head contains the main characteristics of the equipment as well as the overall identification within the system. The second element is the internal service, which has functions and operations that the device performs internally for its correct operation and in which the architecture has no interference or control. Finally, external services are actions that the device can perform and report to the architecture to contribute a planner’s objective. An example of how the system interprets the different elements can be seen in Figure 6, where all the elements of the telepresence robot are described by the three groups, previously analyzed as head, internal and external services. For example, in the case of “head”, the dimensions and the weight of the robot are described, it is also interesting to note that each group can associate particular functions, in this case, the telepresence robot can call the following functions: go to a point, call or reload, etc. Each function has a message type associated with its structure in ROS inside the robot for that matter. Currently, elements are supported in MQTT and ROS In the future, it is proposed to include other types of protocols (such as COAP) and other types of identification (such as RFID). The main idea of having an interpreter will be to group by common types to build own nodes that perform these functions. The ROS-interpreter connects to each of the elements registered in the system and turn, connects within the architecture with the performance monitor and the scheduler. The program will update the elements through its tags and the planner structures the different actions to be carried out. The system has subroutines that allow the installation of the different devices quickly and easily. Robotics 2019,8, 76 16 of 17 6. Recupero, R.D.; Gangemi, A.; Nolfi, S.; Nuzzolese, A.; Presutti, V.; Raciti, M.; Messervey, T.; Casey, D.; Dupourque, I.; Pegman, G.; et al. MARIO: Managing Active and Healthy Aging with Use of Caring servIce Robots; EU Project Networking at ESWC: Portoroz, Slovenia, 2015. 7. Antonopoulos, C.; Keramidas, G.; Voros, N.; Huebner, M.; Schwiegelshohn, F.; Goehringer, D.; Dagioglou, M.; Stavrinos, G.; Konstantopoulos, S.; Karkaletsis, V. Robots in Assisted Living Environments as an Unobtrusive, Efficient, Reliable and Modular Solution for Independent Ageing: The RADIO Experience. In Proceedings of the 14th International Symposium on Applied Reconfigurable Computing (ARC 2018), Santorini Island, Greece, 2–4 May 2018; pp. 712–723. [CrossRef] 8. Jaschinski, C.; Ben Allouch, S. Ambient Assisted Living: Benefits and Barriers from a User-Centered Perspective. In Proceedings of the Fourth International Conference on Ambient Computing, Applications, Services and Technologies, Rome, Italy, 24–28 August 2014; pp. 56–64. 9. Van Kemenade, M.A.M.; Hoorn, J.F.; Konijn, E.A. Do You Care for Robots That Care? Exploring the Opinions of Vocational Care Students on the Use of Healthcare Robots. Robotics 2019,8, 22. [CrossRef] 10. Shi, W. A Survey on Assistive Technologies for Elderly and Disabled People. J. Mechatron. 2015 ,3, 121–125. [CrossRef] 11. Ohta, S.; Nakamoto, H.; Shinagawa, Y.; Tanikawa, T. A Health Monitoring System for Elderly People Living Alone. J. Telemed. Telecare 2002,8, 151–156. [CrossRef] [PubMed] 12. Inoue, K.; Wada, K.; Uehara, R. How Effective Is Robot Therapy?: PARO and People with Dementia. In Proceedings of the 5th European Conference of the International Federation for Medical and Biological Engineering, Budapest, Hungary, 14–18 September 2011. 13. Coradeschi, S.; Cesta, A.; Cortellessa, G.; Coraci, L.; Galindo, C.; Gonzalez, J.; Karlsson, L.; Forsberg, A.; Frennert, S.; Furfari, F.; et al. GiraffPlus: A System for Monitoring Activities and Physiological Parameters and Promoting Social Interaction for Elderly. In Human-Computer Systems Interaction: Backgrounds and Applications 3; Hippe, Z.S., Kulikowski, J.L., Mroczek, T., Wtorek, J., Eds.; Springer International Publishing: Cham, Switzerland, 2014; pp. 261–271. [CrossRef] 14. iRobot. Cambie la Forma de Limpiar Para Siempre. 2018. Available online: www.irobot.es/robotsdomesticos/aspiracion (accessed on 30 August 2019) 15. Moley—Robotic Kitchen. Future is Served Moley Robotics. 2018. Available online: www.moley.com (accessed on 30 August 2019) 16. Robot Care Systems. Lea Care. 2018. Available online: www.robotcaresystems.com/lea-care/ (accessed on 30 August 2019) 17. Fink, J.; Bauwens, V.; Mubin, O.; Kaplan, F.; Dillenbourg, P. People’s Perception of Domestic Service Robots: Same Household, Same Opinion? Social Robotics; Mutlu, B., Bartneck, C., Ham, J., Evers, V., Kanda, T., Eds.; Springer: Berlin/Heidelberg, Germany, 2011; pp. 204–213. 18. Young, J.E.; Hawkins, R.; Sharlin, E.; Igarashi, T. Toward Acceptable Domestic Robots: Applying Insights from Social Psychology. Int. J. Soc. Robot. 2008,1, 95. [CrossRef] 19. Torta, E.; Werner, F.; Johnson, D.O.; Juola, J.F.; Cuijpers, R.H.; Bazzani, M.; Oberzaucher, J.; Lemberger, J.; Lewy, H.; Bregman, J. Evaluation of a Small Socially-Assistive Humanoid Robot in Intelligent Homes for the Care of the Elderly. J. Intell. Robot. Syst. 2014,76, 57–71. [CrossRef] 20. MQTT. MQTT org. 2018. Available online: http://mqtt.org/ (accessed on 30 August 2019) 21. Loza, D.; Marcos, S.; Zalama, E.; García-Bermejo, J.G. Interaction System Based on an Avatar Projected on a Pyramidal Display. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, 1–5 October 2018; pp. 3943–3948. [CrossRef] 22. ROS. 2018. Available online: http://www.ros.org/ (accessed on 30 August 2019) 23. Medina, S.; Córdova, F. Diseño y Construcción del Prototipo de Robot Móvil para Telepresencia Controlado a Través de Internet; Technical Report; Universidad de las Fuerzas Armadas—ESPE: Sangolquí, ON, Ecuador, 2018. 24. Viola, P.; Jones, M.J. Robust Real-Time Face Detection. Int. J. Comput. Vision 2004,57, 137–154. [CrossRef] 25. openHAB. openHAB Empowering the Smart Home. 2018. Available online: https://www.openhab.org/ (accessed on 30 August 2019). Robotics 2019,8, 76 17 of 17 26. Aguilar, R.; Ortí, C. Diseño e Implementación de un Sistema robóTico de Monitoreo e Interpretación de Signos Vitales de Personas de Edad Avanzadas Basado en Openhab; Technical Report; Universidad de las Fuerzas Armadas—ESPE: Sangolquí, ON, Ecuador, 2017. 27. Kaelbling, L.P.; Littman, M.L.; Cassandra, A.R. Planning and Acting in Partially Observable Stochastic Domains. Artif. Intell. 1998,101, 99–134. [CrossRef] c 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).