Document [original]

Human Robotic Avatars

in Outpatient Care –

Combining Professional

Expertise and Autonomous

Functions Using Immersive

Telepresence and Shared

Control Methods

Friese, Christopher 1

Klebbe, Robert 1

Vorwerg-Gall, Susan 1

1 Charité – Universitätsmedizin Berlin, corporate member of Freie

Universität Berlin and Humboldt-Universität zu Berlin, Department of

Geriatrics and Medical Gerontology, Geriatrics Research Group

ORCID:

Friese, Christopher (Corresponding author): 0000-0003-1564-0492

Klebbe, Robert (Contributing author): 0000-0001-7182-4232

Vorwerg-Gall, Susan (Contributing author): 0000-0002-7227-2799

2Introduction and problem description

The demographic change in Germany presents society with the major

challenge of providing dignified and needs-based care for the increas-

ing number of care recipients with a limited number of nursing profes-

sionals. At the end of 2021, around 5 million people in Germany needed

care as defined by the German Long-Term Care Insurance Act (SGB XI).

Of all care recipients 83% (4.17 million) were cared for at home and of

these, 25% (1.05 million) were cared for by outpatient care services. In

comparison, the total number of care recipients at the end of 2019 was

4.13 million, corresponding to an overall increase of 21%. Related to

the increasing number of care recipients, a central problem in the care

sector is the comparatively low growth rate in the number of nursing

professionals. During the aforementioned period, the number of care

recipients in outpatient care increased by 6.5%, while the number of

nursing professionals grew by only 5.1% (Statistisches Bundesamt,

2022). This is exacerbated by the fact that around 500,000 nursing

professionals will retire in the next 10 years (Deutscher Pflegerat, 2021)

as well as by poor working conditions, which strongly impact recruit-

ment and turnover (Bleses, Busse and Friemer, 2020). These working

conditions include an increasing workload due to the accumulation

of work assignments, an increasing delegation of medical tasks, and

a considerable amount of bureaucratic documentation. As a result of

the high levels of mental and physical stress, nursing professionals are

exposed to an increased risk of musculoskeletal and cardiovascular

diseases, sleep disturbances, and other mental health disorders, such

as burnout (Hasselhorn, 2005; Rothgang and Müller, 2018; Schmucker,

2020; Bundesagentur für Arbeit, 2021).

Against this background, the use of robotics in care is

one possible solution to meeting these challenges. Robotic systems

can support nursing professionals in everyday care activities or take

over entire tasks autonomously. Despite considerable technological

progress in the field of autonomous assistive robotics, the current

state of research does not suggest that such systems will be able to

make a significant contribution to care in the near future (Gisinger,

2018; Hülsken-Giesler and Daxberger, 2018). Therefore, a teleoperation

approach that combines the autonomous capabilities of robotic sys-

tems with human expertise could be a promising solution to overcome

the current challenges of autonomous robotics. In outpatient care

in particular, the use of teleoperated robotic systems could support

care recipients and relieve the physical and mental strain on nursing

professionals.

2State of the art in science and technology

Robotics research in the field of nursing and healthcare dates back

to the 1980s. As a result of rapid technological advances, the areas in

which robotic applications can be used have expanded enormously

(Becker, 2013; Bilyea et al., 2017; Kehl, 2018). Potential fields of appli-

cation include diagnostics, surgery, therapy (Birks et al., 2016; Laut,

Porfiri and Raghavan, 2016; Liu et al., 2017; Haenssle et al., 2018) pro-

cess automation (e.g. medication and material management, cleaning)

and the support of older people in the context of daily activities (e.g.

eating, hygiene, mobility, housekeeping, social participation) (Becker,

2013; Kachouie et al., 2014; Kehl, 2018). Currently, robots are mainly

used in medical facilities. With regard to the care sector, it must be

noted that most robotic solutions are still in the development stage

and only a few are already commercially available (Bedaf, Gelderblom

and de Witte, 2015; Bardaro, Antonini and Motta, 2021).

Assistance systems to support older people and people

in need of care are primarily aimed at promoting independence and

support in basic activities of daily living. The spectrum of robotic solu-

tions in this field ranges from simple systems with specialized assis-

tance functions to autonomous, multifunctional assistance systems.

While specialized systems with a clearly defined area of application and

relatively low complexity are already commercially available, multifunc-

tional home assistance systems are still largely in the development and

prototype stage (Royakkers and van Est, 2015). Examples of multifunc-

tional assistance robots already available are “Care-O-bot 4” and “Lio”

(Langegger and Gnass, 2023). When using robots in care, reliability and

functional safety are of outstanding importance (Jacobs and Virk, 2014).

However, autonomous systems face the challenge that their operation-

al environment, the private home of a person needing care, is highly

unstructured and therefore difficult to fully automate. In contrast, pure

teleoperation is not expedient due to the high cognitive demands placed

on the teleoperator. User studies focusing on care using traditional

teleoperation also show that a non-intuitive user interface is one of the

main barriers to the use of teleoperated systems in practice. Complex

user interfaces as well as difficulties in understanding the limits of

robot articulation, end effector rotation, or camera perspectives lead to

imprecise manipulation and significantly increase the time to complete

the task (Li et al., 2017; Koceska et al., 2019). Consequently, approaches

for combining autonomous robotics and human-controlled teleoper-

ation (human-in-the-loop) are promising solutions for combining the

strengths of automation, such as precision and high availability, with the

strengths of humans, such as situational awareness and adaptability.

2One approach is to use methods from the field of shared

control to support the teleoperator during different aspects of a task.

However, another unsolved problem is dealing with time delays in the

area of shared control under haptic feedback. Studies show that it is

difficult for a teleoperator to separate the effects of time delay from

the mechanical characteristics of the system to be controlled (Fu et al.,

2019). To enable the beneficial use of shared control procedures, fur-

ther development of existing teleoperation user interfaces is essential.

Objectives

In the “Teleskoop” project, Devanthro GmbH, FZI Research Center for

Information Technology, and Geriatrics Research Group of Charité -

Universitätsmedizin Berlin collaborate to further develop the robotic

telepresence system “Robody” in a user-centered approach and to

evaluate the system iteratively in the context of outpatient care. The

focus is on relieving the burden on nursing professionals and helping

them to organize their working hours more efficiently by enabling cer-

tain everyday care tasks to be performed via shared control supported

teleoperation and thus without physical presence. At the same time,

the interpersonal relationship between the person in need of care and

the nursing professionals should be preserved.

Methodology

The “Teleskoop” project pursues a user-centered research and devel-

opment approach in which the target groups are continuously involved

to determine requirements and needs for use in the outpatient care

setting as well as to evaluate the user-friendly implementation of these

requirements in the further technical development of the Robody. Eth-

ical, legal, and social implications (ELSI) are also continuously consid-

ered and reflected. The planned realization chain of the project can be

divided into three phases.

Figure 1: Phases in the “Teleskoop” research and development project

Phase 1 Requirements analysis and system

architecture

Phase 2

Adaptation of Robody to application

scenarios

Shared control to support the

teleoperator

Phase 3

Iterative evaluation in outpatient

care contexts

ELSI of the

system to be

developed

The first phase focused on identifying the individual needs of po-

tential users in everyday care, specific application scenarios, and the

associated requirements for the further development of the system

to support both people in need of care and outpatient nursing pro-

fessionals. Based on a qualitative research approach, semi-structured

interviews were conducted with the target groups of nursing profes-

sionals, people in need of care, and caregiving relatives (n=10). The

interviews consisted of two sections. First, the general care situation

of those in need of care and the tasks to be performed by formal and

informal caregivers were determined. The second part consisted of

the presentation of a short video about the Robody, as a basis for the

participants to give feedback on the system and express their ideas

on possible application scenarios. On the one hand, the results of the

requirements analysis were used to develop personas to concret-

ize target groups and application scenarios. On the other hand, they

served as the foundation for a further development and implementa-

tion concept for the advancement of Robody. Moreover, the ELSI-SAT

screening and assessment tool developed for research teams was used

in phase 1. Using the questionnaire provided in the tool, ethical, legal,

and social implications (ELSI) were identified concerning human-tech-

nology interaction for “Teleskoop”. Furthermore, recommendations for

action were derived supplementing the measures already planned to

address the identified risks.

In the current second phase, the hardware of Robody, the telepres-

ence interface, and the associated shared control system are being

advanced following the development and implementation concept

created in phase 1. After the integration, simulative and practical func-

tional tests are performed.

2In the third phase of the project, the further developed system will be

tested and evaluated in outpatient care. Both accompanying investiga-

tions of partial developments and the final evaluation of the research

demonstrator with the target groups are planned. In addition to

evaluating the technical feasibility, the tests will focus on the accep-

tance and user-friendliness of the teleoperated robotic system. In this

context, the development of a target group specific training concept

for the successful use of the Robody is a further focus of the project.

Conceptual innovation

The Robody to be researched and further developed in the “Teleskoop”

project is a system with an immersive teleoperation interface and

methods of shared control technology for the application area of out-

patient care. The use of a Robody is intended to enable nursing profes-

sionals to perform everyday outpatient care tasks without having to be

physically present, thus allowing them to devote the majority of their

working time to core nursing tasks.

Through immersive telepresence using visual, auditory, and haptic

connection to a Robody, the nurse is put in the position of the ro-

bot and can control it intuitively and cooperatively with the whole

body. The teleoperator interface is designed for precise control of the

Robody in real-time. A commercially available virtual reality system

is used to create an immersive control environment for the operator.

This includes a 180-degree 3D first-person view with stereovision

Figure 2: Controlling the Robody

2based on a live video stream from Robody’s cameras. Furthermore, the

operator hears the environment around the robot and can commu-

nicate with his own voice via Robody’s loudspeaker. The robot’s head

also displays the operator’s face and facial expressions in real-time.

The Robody is controlled in sync with the operator’s movements, as

the operator’s head and arm movements are transmitted directly to

Robody’s head and arms. The joysticks on the controllers can also be

used to intuitively maneuver the mobile platform.

The stereoscopic vision, bidirectional audio con-

nection, and full-body movement mapping in combination with the

operator’s apprehension enable precise movement in confined and

unstructured spaces on the one hand and natural interaction between

the operator and the person in need of care on the other. Familiar in-

terpersonal interaction patterns can thus be transferred to the robot.

This represents a significant development compared to traditional

teleoperation approaches, in which the robot is controlled using a

joystick and various camera images on a screen. Safety during robot

operation is also increased, as previous problems with the perception

of mechanical limits are reduced. As Robody never acts autonomously

but is always controlled by an operator, this human-in-the-loop ap-

proach also ensures that decisions, especially in complex or sensitive

situations, are always made under human guidance.

Figure 3: Holding hands via Robody

2In addition, methods of shared control are being tested within the

project, which can combine the strengths of humans in the detection

of complex and unstructured situations with the strengths of auto-

mation in the precise execution of movement sequences. The shared

control algorithms support the operator during the execution of vari-

ous tasks and should therefore increase the efficiency of the nurse and

at the same time increase safety for the person receiving care. Shared

control is also intended to enable the execution of tasks close to the

body (e.g. handing over a glass of water, holding a hand) by provid-

ing haptic feedback to the operator to indicate how much force the

Robody is currently applying. By integrating shared control, tasks are

to be performed that currently neither autonomous systems nor fully

teleoperated systems are able to handle.

Challenges and limitations

Publicly funded research and development projects usually serve to

investigate and evaluate socially, economically, and technically risky

developments (Fellbaum, 1982). The “Teleskoop” project is funded as

part of the BMBF’s START-interaktiv funding program, in which high-

risk pre-competitive research on interactive technologies for health and

quality of life is conducted (Bundesministerium für Bildung und For-

schung, 2021). In connection with the associated priorities and frame-

work conditions, there are various challenges and limitations for the

scientific research process, which are briefly described below.

When conducting field studies, particularly in the care

sector, there are major challenges in terms of access to the target

groups and the duration of the evaluation, as the entire research and

development process is limited to three years in most funding pro-

grams. For the user-centered research process, which requires continu-

ous involvement of the target groups, this also means that the research

demonstrator can only be evaluated as a proof of concept with relatively

small samples, typically in explorative pilot studies. Consequently, the

generalizability of the results is limited.

Another challenge in this context is researching the

effects on users and those affected as well as validating the research

results. Given the early stage of development of the innovative solution,

research questions relating to the effects of diffusion into existing social

structures and conditions can hardly be predicted.

Despite the limitations in terms of generalizability and

predictability, the results represent an essential starting point for further

evaluation of such innovative solutions, also for related fields of research.

2Prospect

Although Robodies are designed as support systems for care, the

technology can be extended to other user groups and use cases. This

includes, for example, people with disabilities or patient care during a

pandemic. In addition, Robodies can be equipped with further sensors,

for example, to measure vital parameters, which provide information

for remote diagnosis and monitoring. In the long term, Robodies could

also be used outside the medical care sector, which could lead to a

further increase in acceptance, a reduction in costs, and the general

penetration of robotics into everyday life.

2References

Bardaro, G., Antonini, A. and Motta, E. (2021) ‘Robots for Elderly Care

in the Home: A Landscape Analysis and Co-Design Toolkit’, Interna-

tional Journal of Social Robotics [Preprint]. Available at: https://doi.

org/10.1007/s12369-021-00816-3.

Becker, H. (ed.) (2013) Robotik in Betreuung und Gesundheitsver-

sorgung. Zürich: vdf Hochschulverl (TA-SWISS, 58).

Bedaf, S., Gelderblom, G.J. and de Witte, L. (2015) ‘Overview and Cat-

egorization of Robots Supporting Independent Living of Elderly People:

What Activities Do They Support and How Far Have They Developed’,

Assistive Technology, 27(2), pp. 88–100. Available at: https://doi.org/10.1

080/10400435.2014.978916.

Bilyea, A. et al. (2017) ‘Robotic assistants in personal care: A scoping

review’, Medical Engineering & Physics, 49, pp. 1–6. Available at:

https://doi.org/10.1016/j.medengphy.2017.06.038.

Birks, M. et al. (2016) ‘Robotic Seals as Therapeutic Tools in an Aged

Care Facility: A Qualitative Study’, Journal of aging research, 2016, pp.

8569602–8569602. Available at: https://doi.org/10.1155/2016/8569602.

Bleses, P., Busse, B. and Friemer, A. (eds) (2020) Digitalisierung der

Arbeit in der Langzeitpflege als Veränderungsprojekt. Berlin, Heidelberg:

Springer Berlin Heidelberg. Available at: https://doi.org/10.1007/978-3-

662-60874-6.

Bundesagentur für Arbeit (2021) Berichte: Blickpunkt Arbeitsmarkt –

Arbeitsmarktsituation im Pflegebereich. Nürnberg, p. 22. Available at:

https://statistik.arbeitsagentur.de/DE/Statischer-Content/Statistiken/

Themen-im-Fokus/Berufe/Generische-Publikationen/Altenpflege.

pdf?__blob=publicationFile (Accessed: 20 May 2021).

Bundesministerium für Bildung und Forschung (2021) Bekanntma-

chung - BMBF, Bundesministerium für Bildung und Forschung - BMBF.

Available at: https://www.bmbf.de/bmbf/shareddocs/bekanntmachun-

gen/de/2021/04/3576_bekanntmachung.html (Accessed: 19 December

2023).

Deutscher Pflegerat (2021) Deutscher Pflegerat, 500.000 professionell

Pflegende werden in den nächsten 12 Jahren in Rente gehen. Available

at: https://deutscher-pflegerat.de (Accessed: 17 November 2023).

Fellbaum, K.-R. (ed.) (1982) Telekommunikation von A - Z. Berlin:

VDE-Verl (Gesellschaft zur Förderung der Unterhaltungselektronik.

Handbuch, 4).

Fu, W. et al. (2019) ‘Framework for Human Haptic Perception With

Delayed Force Feedback’, IEEE Transactions on Human-Machine

Systems, 49(2), pp. 171–182. Available at: https://doi.org/10.1109/

THMS.2018.2885401.

Gisinger, C. (2018) ‘Pflegeroboter aus Sicht der Geriatrie’, in O. Bendel

(ed.) Pflegeroboter. Wiesbaden: Springer Fachmedien Wiesbaden, pp.

113–124. Available at: https://doi.org/10.1007/978-3-658-22698-5_6.

2Haenssle, H.A. et al. (2018) ‘Man against machine: diagnostic perfor-

mance of a deep learning convolutional neural network for dermoscopic

melanoma recognition in comparison to 58 dermatologists’, Annals of

Oncology, 29(8), pp. 1836–1842. Available at: https://doi.org/10.1093/

annonc/mdy166.

Hasselhorn, H.-M. (ed.) (2005) Berufsausstieg bei Pflegepersonal:

Arbeitsbedingungen und beabsichtigter Berufsausstieg bei Pflegeper-

sonal in Deutschland und Europa. Bremerhaven: Wirtschaftsverl. NW,

Verl. für Neue Wiss (Schriftenreihe der Bundesanstalt für Arbeitsschutz

und Arbeitsmedizin Ü, Übersetzung, 15).

Hülsken-Giesler, M. and Daxberger, S. (2018) ‘Robotik in der Pflege

aus pflegewissenschaftlicher Perspektive’, in O. Bendel (ed.) Pflegerobo-

ter. Wiesbaden: Springer Fachmedien Wiesbaden, pp. 125–139. Available

at: https://doi.org/10.1007/978-3-658-22698-5_7.

Jacobs, T. and Virk, G.S. (2014) ‘ISO 13482 - The new safety standard

for personal care robots’, in ISR/Robotik 2014; 41st International Sym-

posium on Robotics, pp. 1–6.

Kachouie, R. et al. (2014) ‘Socially Assistive Robots in Elderly Care: A

Mixed-Method Systematic Literature Review’, International Journal of

Human-Computer Interaction, 30(5), pp. 369–393. Available at:

https://doi.org/10.1080/10447318.2013.873278.

Kehl, C. (2018) Robotik und assistive Neurotechnologien in der Pflege -

gesellschaftliche Herausforderungen. TAB Arbeitsbericht Nr. 177, p. 240.

Koceska, N. et al. (2019) ‘A Telemedicine Robot System for Assisted and

Independent Living’, Sensors, 19(4), p. 834. Available at:

https://doi.org/10.3390/s19040834.

Langegger, D. and Gnass, I. (2023) ‘Pflegeunterstützende Maßnahmen

durch robotische Systeme’, HeilberufeScience [Preprint]. Available at:

https://doi.org/10.1007/s16024-023-00399-1.

Laut, J., Porfiri, M. and Raghavan, P. (2016) ‘The Present and Future

of Robotic Technology in Rehabilitation’, Current physical medicine

and rehabilitation reports, 4(4), pp. 312–319. Available at: https://doi.

org/10.1007/s40141-016-0139-0.

Li, Z. et al. (2017) ‘Development of a tele-nursing mobile manipulator

for remote care-giving in quarantine areas’, in 2017 IEEE Interna-

tional Conference on Robotics and Automation (ICRA). Singapore,

Singapore: IEEE, pp. 3581–3586. Available at: https://doi.org/10.1109/

ICRA.2017.7989411.

Liu, H.-H. et al. (2017) ‘Robotic surgical systems in maxillofacial

surgery: a review’, International journal of oral science, 9(2), pp. 63–73.

Available at: https://doi.org/10.1038/ijos.2017.24.

Rothgang, H. and Müller, R. (2018) Pflegereport 2018. Edited by Barm-

er. Berlin (Schriftenreihe zur Gesundheitsanalyse).

2Royakkers, L. and van Est, R. (2015) ‘A Literature Review on New

Robotics: Automation from Love to War’, International Journal of Social

Robotics, 7(5), pp. 549–570. Available at: https://doi.org/10.1007/

s12369-015-0295-x.

Schmucker, R. (2020) ‘Arbeitsbedingungen in Pflegeberufen: Ergeb-

nisse einer Sonderauswertung der Beschäftigtenbefragung zum

DGB-Index Gute Arbeit’, in K. Jacobs et al. (eds) Pflege-Report 2019.

Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 49–60. Available at:

https://doi.org/10.1007/978-3-662-58935-9_3.

Statistisches Bundesamt (2022) Pflegestatistik – Pflege im Rahmen

der Pflegeversicherung – Deutschlandergebnisse – 2021, Pflegestatistik –

Pflege im Rahmen der Pflegeversicherung – Deutschlandergebnisse –

2021. Available at: https://www.destatis.de/DE/Themen/Gesellschaft-

Umwelt/Gesundheit/Pflege/Publikationen/Downloads-Pflege/pflege-

deutschlandergebnisse-5224001219005.html (Accessed: 17. November

2023)