De Backere, Femke

De Pestel, Stijn

De Turck, Filip

Elprama, SA

Jacobs, A

Jewell, C

Kilpi, K

Mahieu, Christof

Nelis, Jelle

Ongenae, Femke

Simoens, Pieter

English

Ghent University Academic Bibliography

Internet of Robotic Things:Context-Aware and Personalized Interventions of Assistive Social RobotsPieter Simoens, Christof Mahieu, Femke Ongenae, FemkeDe Backere, Stijn De Pestel, Jelle Nelis, Filip De TurckDept. of Information Technology - IBCNGhent University - iMindsGhent, BelgiumEmail: psimoens@intec.ugent.beShirley A. Elprama, Katriina Kilpi,Charlotte Jewell, An JacobsStudies on Media, Information, Telecommunication (SMIT)Vrije Universiteit Brussel - iMindsBrussels, BelgiumEmail: selprama@vub.ac.beAbstract—Assistive service and companion robots are versa-tile and dexterous actuators that operate in our daily livingenvironment. These robots are able to manipulate physicalobjects, to displace themselves and to engage in conversations.Human behavior is dynamic and oftentimes unpredictable,therefore it is crucial for such robotic systems to be assistedby a cloud-backend which: i) analyzes data from sensorand wearables ii) determines which robotic tasks need tobe executed and iii) provides the necessary support for theexecution of these tasks in our daily living environment. Inthis paper, we present our Internet-of-Robotic-Things systemarchitecture design for a case study on personal interactionsby a companion robot to alleviate behavioral disturbances ofpeople with dementia.Keywords-IoT; robot; interventionI. INTRODUCTIONThe swarm of sensors and wearables in our living en-vironment produces a vast amount of data. From this datavaluable knowledge can be extracted about the context inwhich people find themselves and about their activities [1].As of today, this knowledge is mainly applied in intelligentmobile and cloud-based software applications. The reversepath from the cyber to the physical world remains far lessexploited. Contemporary scenarios are limited to automationalgorithms driving thermostats, light armatures as well asservice robots such as vacuum cleaners and lawn mowers.Compared to these single-purpose actuators, assistive so-cial robots are much more dexterous and versatile. The mainfunction of such robots is either to support independentliving or to act as a companion [2], a domain growingin importance in view of the ageing population. Whensuch robots are integrated as additional actuator in smartenvironments (home, office, factory), we create an Internet-of-Robotic-Things (IoRT). The smart IoRT environmentanalyzes data from sensors and wearables and then infersan actuation strategy to reach the desired environment state.This strategy is dependent on the context, as well as onthe functional capabilities and availability of all robotic andnon-robotic actuators. For example, when an older personmust be notified, the smart home can deliver the messageindirectly via the smartphone of visiting family or caregiverswhen the companion robot (which has a tablet) is unavailablebecause it is currently charging.The integration in the Internet-of-Things is not onlyimportant to determine which tasks must be executed bythe robot, it is also crucial for the assistive social robotto execute the assigned tasks. The information providedby on-board sensors is not sufficient for a robot to fullygrasp the complex and dynamic environment it operatesin. Instead, the robot needs to fuse on-board sensor datawith information provided by the manifold external sensors,smartphones and wearables.In view of this tight relation between robots and theirenvironment, as well as the significant computational chal-lenges to interpret the context information, support from acloud back-end in the IoRT is crucial. In this paper, wepresent our system architecture for an IoRT cloud back-endplatform, applied to a case study we are currently workingon regarding technological care support of older adults innursing homes (NH).II. CASE STUDY: PERSONALIZED ROBOTINTERVENTIONS FOR PERSONS WITH DEMENTIAAlmost all persons with an advanced form of dementiaexhibit behavioral disturbances (BD) such as mood anddisorders, wandering, physical and verbal aggression. TheseBD impose an enormous emotional toll on the patient, familyand increase the burden and stress of caregivers. Empiricalevidence suggests that manifestations of BD can be pre-vented by care strategies that preserve the dignity of personswith dementia (PWD) and promote their comfort and qualityof life. For example, activities organized by the NH care staffhave the largest effect on reducing the number of BD if theyare adapted to the individual’s needs, emotions, interestsand functional ability and carefully spaced throughout theday [3]. The labor intensity of such person-centric care,combined with downward pressure on health care budgetsworldwide, forms a clear need for technological solutionsassisting the NH caregiver staff.A. Technology conceptWe are currently building the IoRT service platformvisualized in Figure 1. This platform drives personalizedinterventions of the humanoid care robot Zora1 in thedementia ward of nursing homes.Figure 1. Behavioral disturbances are detected by combining sensor andwearable data with context information. The platform alerts a caregiver orcreates a companion robot intervention task. Tasks are scheduled over apool of robots, taking into account their current position and battery level.The aim is to have Zora walking from one resident toanother, each time engaging in a personalized interaction,e.g. playing a favorite song or asking questions aboutmemorable events in the lifetime of the PWD. As PWDrevisit the past more and more, the research hypothesis iswhether the robot will be able to elicit positive memoriesassociated with positive feelings, which in turn could preventor alleviate BD manifestations.Currently, Zora is used for recreational and therapeuticactivities. These are essentially predefined sequences ofinstructions that are manually activated by the care staff.Our service platform instead embeds Zora in a more struc-tural way in the care process, so as to have one or morerobots operating 24/7 without involvement of the care staff.BD manifestations are detected by analyzing sensor andwearable information. These detections are combined withpersonal information about the residents provided by the carecoordinator, as well as organizational context information(menu, floor map, activity and meal timetables) to generatea per-resident intervention strategy. In acute situations, theactuation strategy is to alert the staf or to send a robotto temporarily distract the PWD. In addition, proactiveinterventions are scheduled collaboratively over multiplerobots throughout the day/night (24/7), based on historicalknowledge extracted from the sensor and wearable data,1www.zorarobotics.beand taking into account the limitations of individual robots,e.g. on battery lifetime or walking speed.B. Use case scenariosWandering It’s 23:50 and the NH residents are in theirrooms. Marie, a PWD, woke up, left her room, and iscurrently wandering in the hallways. She is known to wanderaround at times and bother other residents by entering theirrooms. As soon as the presence sensors in the NH registerthat Marie left her room, the Zora robot is instructed toapproach her. In the mean time, the nurse on night shiftreceives a message that Marie left her room but that Zorais on her way to Marie. The robot finds Marie using acombination of tracking sensors and signals emitted by achip in her bracelet. Zora says to Marie that it is too earlyto be up and suggests to go back to her room. Marie goesback to her room and the system sends a message to thenurse that Marie is back in her room.Screaming Paul, a PWD, is waiting in the common areaof the NH after dinner, while all residents are brought totheir rooms one by one by a team of caregivers. Someresidents are able to go to their room by themselves, but themajority - like Paul - needs help. Paul raises his voice andcalls for the caregivers, but they are busy taking care of otherresidents and go to him immediately. The calling has beendetected by microphones and Zora is walking towards Paul.At the same time, the caregivers on duty receive a messageon their smartphone that Paul is calling but that Zora is onher way. Since it is known that Paul likes American music,Zora plays a song of Sinatra. Paul enjoys the song and isdistracted, so he stops calling for the caregivers and startssinging along with the robot. Then, the caregivers receivea message on their smartphone that Paul has calmed downand stopped screaming by Zora playing music for him.III. SYSTEM ARCHITECTUREFigure 2 visualizes the different modules of our cloudplatform. Each module is further discussed below.A. Sensor and wearable data input layerThis layer provides the necessary input data to accuratelydetect and predict BD manifestations, and to assist thehumanoid robots in their personal interventions. We haveinstalled GrovePi+ gateways at various places in two NHparticipating in this project. The installed sensors capturedata relevant for the two use cases described, as well asadditional context information so as to identify possiblepatterns and triggers of BD manifestations. To hide thedifferent protocols and interfaces of the installed sensors,we leverage on our DYAMAND middleware [4].Wandering behavior is an umbrella term for variousin-room and out-of-room movement patterns, and can becaptured using indoor localization techniques [5]. BecausePWD are removing body-worn sensors and wearables, weFigure 2. Technical architecture of our IoRT service platform, comprising five modules.are experimenting with positioning sensors placed on walls2,complemented with Bluetooth Low Energy tags attachedto wheelchairs and rollators. Screaming can be detectedby measuring the sound level in various parts of the NH.Additional sensors provide context information on roompresence, temperature and humidity.B. Data analytics (DA)This layer is responsible for translating raw sensor datainto meaningful context information. It is comprised of twomodules, one for the detection of BD manifestations and oneto track the robots, NH residents and caregivers.In the first module, state-of-the-art probabilistic patternrecognition algorithms for BD detection [6] are combinedwith reasoning algorithms on an ontology-based contextmodel. The ontology is an extension of the Ambient-awareContinuous Care Ontology [7], with specific knowledgeneeded in the BD domain, such as the observations madeby the sensors and their interrelationships, the spatial layoutof the NH, the time of day, the activity schedule, and the pro-files of the elderly, e.g., which BDs they exhibit, preferencesand needs, severity of dementia. To implement this context-& profile-aware algorithm, the MASSIF platform [8] is usedfor semantic annotation of and reasoning on IoT data.The second DA module is the environmental mapping.This component enriches the floor plan of the building withinformation that is particularly relevant for the robotic pathplanning. Zones such as a bathroom are inaccessible forthe robot for privacy reasons. Other regions should onlybe accessed under very specific conditions. For example,robots underneath a table might surprise older adults, yetthis region can be used in case the robot must rapidly freethe way for an approaching older adult. All this informationis encoded as annotations to the floor plan. The floor planis also updated based on dynamic information, e.g. doorsbeing closed, residents moving around, etc.C. Intervention Task Generation (ITG)This component determines the appropriate IoRT actua-tion strategy, based on the context, type and priority of the2provided by another project partner, see www.xetal.netBD detected. The algorithms are also implemented with theMASSIF platform using the same ontology as the DA com-ponent. Possible actuations are alerting caregivers or robotictasks. For example, when the algorithm detects with a lowprobability that somebody is wandering in the hallway, butthe identity of the PWD is difficult to assess, the interventioncould consist of sending the robot to the scene in order togather additional information through its camera and throughconversation. On the other hand, when aggressive behaviorof a PWD is detected, a better intervention is to immediatelyalert the staff. The type of intervention also depends on thePWD and on the context. Some PWD might respond betterto particular types of interactions (conversation, playingmusic), and the sound level of the robot should be adaptedto the degree of PWD hearing impairment as well as to thetime of the day. Variation over time should also be takeninto account, to ensure long-term interest of the PWD inrobot interaction. Lastly, the platform’s intervention strategyalso comprises pro-active robot tasks, e.g. the robot shouldvisit each PWD two times a day, or it should announce thenews item of interest to the PWD each morning.D. Robot Pool Management (RPM)The RPM module is responsible for scheduling the per-sonal intervention tasks over the set of robots deployed inthe NH, taking into account priorities and robot availability.The scheduler receives interventions tasks from the ITG,as well as other types of tasks, such as to return to a chargingstation, or to take an evasive maneuver when encounteringa resident. The actual execution of the tasks is delegatedto the Task Execution Control Module. Decoupling the taskplanning from the task execution is necessary because robotsare frequently taken out of the system, e.g. because they areused for therapy, have lost Wi-Fi coverage or are locked upby a closed door. In these cases, the tasks assigned to thatrobot will be redistributed. The scheduler is supported bytwo planning services.Charging needs to be planned carefully to ensure 24/7operation without manual interventions from the NH staff.Two robots cannot be connected to the same chargingstation at the same time, and a robot should have sufficientremaining battery capacity to move from its current positionto the selected charging station. It is also important to havea sufficient number of robots available with an acceptablebattery level at all points in time in order to respondadequately to acute BD manifestations.Path planning identifies the most appropriate trajectory fora robot to move from point A to point B. It takes inputfrom the environmental map, avoiding areas as needed. Ap-proaching a person needs to take additional considerationsinto account, such as the estimated position of a wanderingolder adult by the time the robot arrives, and the angle ofapproach to avoid scaring someone. This might require acontinuous refinement of the path based on input from thesensor and wearable abstraction layer.E. Task Execution Control (TEC)This module governs the execution of tasks by the robotthat was selected by the scheduler. Because of the unreliabil-ity of the wireless connection and the limited on-board robotresources, tasks are converted in event sequences. For exam-ple, if the robot is instructed to approach person X, we willnot load the entire path onto the robot. Instead, we identifya number of waypoints on the path for the robot to reportback to the platform and receive the next path segment. If therobot crosses an older adult, it will step aside, temporarilysuspending the task execution. Once arrived in the roomof the target of the intervention task, the platform willactivate and configure the correct robotic software module.Internally, the module contains one controller per attachedrobot. This controller can be seen as the robotic counterpartof the sensor abstraction layer, as it hides the proprietarydata structures and commands to steer a particular robot.The controller also acts as a connection daemon, throwingan unavailability exception to the scheduler if the connectionto the robot is lost, or if the robot is temporarily taken outof service, e.g. because it has been picked up by a therapistto assist in rehabilitation exercises or has been locked in aroom by someone inadvertently closing a door.IV. RELATED WORKCloud robotics is an active research topic in the domain ofambient assisted living. To tackle the inherent heterogeneityof the end-user population, their housing arrangements, theirindividual situations and requirements, in [9], a cognitiverobot approach is presented that is centered around theconcept of robotic ecologies, consisting of sensors, actuatorsand robots. The main focus is on automatically detectingchanges in the user’s habits and evolving care needs. Inthe Robot-Era project, cloud services for user localisation,care reminding and environmental monitoring have beendeveloped [10], [11]. Our work builds on these results, fo-cusing on behavioral disturbances of persons with dementia.To this end, we will take an even more personalized andcontext-aware intervention approach: the strategy to achievea specific goal are adapted to the specific situation and to theindividual’s preferences. We will also include a multi-robotplanning aspect in the intervention strategy, with chargingcycles being taken into account to ensure a 24/7 operation.V. CONCLUSIONThe Internet of Robotic Things is a new paradigm thatcombines the computational power of the cloud and thecontext information provided by sensors and wearables tointelligently define robotic tasks and support the executionthereof. We are currently working on the realization of ause case on personalized robot interventions to alleviatebehavioral disturbances of PWD.ACKNOWLEDGMENTThe work in this article is partially realized in the WON-DER project, co-funded by iMinds and Agentschap Innov-eren & Ondernemen. Project partners are iMinds, ZoraBots,Xetal, WZC De Vijvers, WZC Weverbos.REFERENCES[1] E. A. Lee et al., “The swarm at the edge of the cloud.” IEEEDesign & Test, vol. 31, no. 3, pp. 8–20, 2014.[2] J. Broekens et al., “Assistive social robots in elderly care: areview,” Gerontechnology, vol. 8, no. 2, pp. 94–103, 2009.[3] A. K. Desai et al., “Behavioral disturbance in dementia,”Current psychiatry reports, vol. 14, no. 4, pp. 298–309, 2012.[4] J. Nelis et al., “Dyamand: dynamic, adaptive management ofnetworks and devices,” in Local Computer Networks (LCN),2012 IEEE 37th Conference on. IEEE, 2012, pp. 192–195.[5] N. K. Vuong et al., “mhealth sensors, techniques, and ap-plications for managing wandering behavior of people withdementia: A review,” in Mobile Health. Springer, 2015.[6] F. Palumbo et al., “Multisensor data fusion for activity recog-nition based on reservoir computing,” Evaluating AAL Sys-tems Through Competitive Benchmarking, Communicationsin Computer and Information Science series, vol. 386, 2013.[7] F. Ongenae et al., “An ontology co-design method for the co-creation of a continuous care ontology,” Applied Ontology,vol. 9, no. 1, pp. 27–64, 2014.[8] P. Bonte et al., “The massif platform: a modular & semanticplatform for the development of flexible iot services,” Knowl-edge and Information Systems, 2016.[9] M. Dragone et al., “A cognitive robotic ecology approachto self-configuring and evolving aal systems,” EngineeringApplications of Artificial Intelligence, vol. 45, 2015.[10] M. Bonaccorsi et al., “Design of cloud robotic services forsenior citizens to improve independent living and personalhealth management,” in Ambient Assisted Living. Springer,2015, pp. 465–475.[11] ——, “A cloud robotics solution to improve social assistiverobots for active and healthy aging,” International Journal ofSocial Robotics, pp. 1–16, 2016.

Internet of robotic things : context-aware and personalized interventions of assistive social robots

https://biblio.ugent.be/publication/8159134/file/8159153.pdf

Internet of robotic things : context-aware and personalized interventions of assistive social robots

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography