Home Mobility Hazards Detected via Object Recognition in Augmented Reality

We present an Environmental Analysis and Safety Advisor system capable of identify the environmental barriers and hazards found in the homes of elderly people. This augmented reality tool runs on a portable computing device and can be used by informal and formal caregivers without specific knowledge of Accessible Design, to evaluate the safeness of an elderly home environment, ensuring that potential fall hazards are detected and corrected. The system recognizes specific indoor elements of the house (e.g. arm-chair, bed, chair), and then computes and analyses their mutual distances in the environment so that a warning of hazard is emitted in case of need (e.g. loose cable, not enough space to pass a wheelchair). In this context, we implemented object recognition at the category level of miniature versions of real sized furniture and the determination of the distance between neighboring objects, signaling if it is below a certain threshold value. Environmental Analysis tool can then recognize furniture and measure the distance between two furniture elements enabling the system to pop up an alert sign if the space left does not guarantee good accessibility.


INTRODUCTION
Current demographic statistics present clear trends in the increase of migration to urban settings.This tendency along with the aging of society, notably in Europe and in Japan, result in more elderly people living alone.This demographic tendency creates highpressure, not only on public and private care and social security systems, but also on society as a whole, making current approaches to care and service deliv-ery unsustainable.
The authors have been engaged in the research project Organizational Life Assistant (OLA), an Active Assisted Living funded project, which aims to offer an answer to the above mentioned societal challenges, by providing an innovative virtual presence that supports instrumental activities related to daily living needs of older adults, allowing them to be more independent, self-assured and to have a healthier, safer and organized life.Simultaneously OLA helps caregivers by supporting them with innovative Information and Communication Technologies (ICTs), offering high-quality assistance.
The aim of the project is to mediate and facilitate interaction (communication and collaboration) between senior citizens and their formal and informal caregivers and other services, using technological means such as standard computers, mobile devices (tablets) and home automation.These ICT devices are based on an innovative multimodal human-computer interaction model, embracing various physical, health and cognitive characteristics of older adults and is specifically oriented to increase the level of independence of the elderly, by supporting the possibility of caretakers to assist elderly remotely and by improving the accessibility to ICTs.
The methodology used adopts a phase-based approach for solution development comprising the following stages: i) user requirements and system design; ii) backend system development; iii) frontend system development and integration by means of the OLA app providing the end-user experience; iv) pilot application and user studies.The OLA system comprises four modules: Personalized Well-being Advisor system; Collaborative Care Organizer system; Every-day Instrumental Activities Memory Support system; Environment Analysis and Safety Advisor system.The Environment Analysis and Safety Advisor (OLA EA-SA) system and its implementation is the focus of this paper.This system aims at helping elderly by identifying and alerting to the architectonic hazards in their houses.According to Foster (1998) difficulties in the access to buildings correspond to the major cause of limitation for individuals with disabilities.Falls are the most common accidents that occur in older people and they can be quite disabling.According to several authors, this fear leads to a deterioration in the quality of life of the elderly (Campbell, Robertson, Gardner, Norton, & Tilyard, 1997;Feder, Cryer, Donovan, & Carter, 2000).Some common architectural elements in buildings and homes, such as stairs, are considered as physical barriers for the older population.Thus, many of the falls in built spaces occur on stairs, and their use is considered to be one of the most challenging tasks for people over 65 years old (Tiedemann, Sherrington, & Lord, 2007).
We present the mock-up of an Augmented Reality Application that can identify various types of indoor architectural barriers and hazards, showing corresponding alerts, and presenting suggestions to correct the detected anomalies.The core of our proposed system is object recognition at the category level of never-before-seen objects, that might exist inside elderly homes, such as pieces of furniture.We have used miniatures of real sized objects, that are deployed in a space that represents a certain indoor area.The system can recognize the objects and computing the distance between neighbouring ones, therefore identifying candidates to an architectural barrier for the elderly.
This paper is divided into four main sections.In section 2, Ambient Assistive Living concepts and solutions are presented and discussed.The third section addresses how universal design principles and architectural barriers coexist.Section 4 describes the methodology of development and deployment of the OLA EA-SA.We end the paper with a discussion and conclusion section.

AMBIENT ASSISTIVE LIVING SOLUTIONS
New ICT solutions have been developed in the last years in order to embed intelligent objects in the environment to support people to live independently and with more autonomy.Some of the main endusers of these technologies are older people and people with some degree of permanent or temporary disabilities.Such technologies are designed to help older people prolonging their active and independent life, while supporting communication and integration with society, and with formal and informal caregivers, in particular.
Several Ambient Assistive Living (AAL) research and innovation projects have developed innovative ICTs' solutions, combining the inputs from various disciplines of research in order to guarantee a full support to older people.Projects like AAL PaeLife, AAL S4S and QREN AAL4ALL created new AAL solutions to facilitate the daily activities of older people.The PaeLife project (Personal Assistant to Enhance the Social Life of Seniors) developed the AALFred application, a personal virtual life assistant that helps and guides older users to perform daily activities at home (Teixeira et al., 2013).The S4S project (Smartphones for Seniors) provided adapted technology to the older users in mobile environments, by customizing the user interface of the main functions of Windows Phone smartphones.Several applications were developed that ranged from Medication Intake Assistance and Reminder, focusing on the health of the elderly, to the Living Home Center and XisQuê apps, focused on the social inclusion of the elderly (Dias et al., 2015).The QREN AAL4ALL project (Ambient Assistive Living for All) developed an ecosystem of standardized and interoperable AAL products and services in Portugal, associated to a business model and validated through large scale trial, that involved over 400 elders across the country.In Clockwork project (Pereira & Nunes, 2018) the consortium implemented external synchronizers that will help users monitoring their daily living conditions and improve them.

UNIVERSAL DESIGN AND ARCHITECTURAL BARRIERS
In the 1960s, Selwyn Goldsmith (1963) started to discuss and provide guidance to architects for designs that promoted an inclusive society for people with disabilities.In 1961 the "Accessible and Usable Buildings and Facilities" (ICC A117.1) (American National Standard Institute, 1961) was approved as the first standard for accessibility in the US, although the concept was only introduced in legal standards in the US in 1973 by the Rehabilitation Act section 504 [1].Legislation and regulation about Design for All, Universal Design and Accessible Design has been also introduced in Europe since the 60s.Concepts, recommendations and standards on these topics, have been defined at European and national legislative and executive institutions, at standardized bodies, as well as product and service providers.At a national level, most European states have introduced legislation or created recommendations about Universal Design that contemplate most areas of day-to-day life.
In the scope of the research carried out for this paper, only legislation and recommendations regarding building design were analyzed.These documents address several scales of designing -from the street level to the buildings indoors -and several building elements -e.g.ramps, stairs, elevators, bathrooms, corridors.In addition to the situations addressed by the regulations and norms, other hazards and barriers can exist that affect the general mobility of people and introduce in their lives the risk of stumble or fall.Most of these hazards and barriers can be noticed and avoided by people with full visual acuity and full range of movements but represent a risk for the ones loosing visual acuity and range of mobility which are problems related to old age and temporary or permanent disabilities.
Various studies (Sattin, Rodriguez, DeVito, & Wingo, 1998;Stevens, Holman, Bennett, & de Klerk, 2001) have demonstrated that the identification and correction of architectonic hazards reduce the number of falls registered on the elderly population.The hazards identified by the authors which are not addressed directly in the norms and regulations are: carpets/floor coverings torn, bent or in poor condition; uneven or in poor condition flooring; rugs that slip; unstable furniture; chairs without armrests or with low backs; cabling across passage zones; toilet located outside the house; step edges hard to see; poor lighting; protruding elements, amongst others.

OLA -ENVIRONMENTAL ANALYSIS AND SAFETY ADVISOR
The OLA EA-SA system focuses on assisting older people to transform and/or maintain an accessible and risk-free environment at home.A software tool provides information about the existing elements of the indoor space, such as detecting furniture that could interfere with an accessible path or alerting about steps, rugs and objects lying on the floor or any other phenomenon that might represent a hazard situation for users.The tool suggests changes for the indoor space, such as removing/changing the position of the furniture according to Universal Design principles.For this, computer vision algorithms for real-time object recognition and scene reconstruction were developed (i.e., 3D object and space reconstruction by using an RGB-D sensor -Microsoft Kinect One).The reconstructed scene is automatically analysed and exploited in an augmented reality setting, where visual elements are registered to provide advice on safety measures.Such elements support the elderly by providing hints/advices on the most appropriate action to be taken, or on suggesting changes on environmental elements that interfere with accessible indoor paths and can create hazard situations.

Personas and scenarios
As a first step in the development phase we started by defining personas and then defining usage scenarios based on the personas and their expected tasks.These scenarios were then used to design mock-ups of the OLA platform.Personas were informed by the study developed in the first stage of the OLA project (Casquilho, 2018).
In this paper, we only include a brief reference to the scenario related to the OLA EA-SA system, from which we derived the corresponding features.
Scenario 1 -With the help of her son Pedro, Teresa decided to use the OLA Environment Analysis and Safety Advisor feature to improve her mobility in the house, and consequently to avoid any injuries that might worsen her health condition.The tablet equipped with the scan device was temporarily made available to Teresa and delivered to her home.Her son helped her to perform a complete environmental scan of her house using the scanning wizard provided by the OLA app.They also took a picture of the house floor plan Teresa had in her archives, and uploaded it using the tablet.One hour later, OLA displayed a notification announcing that the environ-mental analysis results were available.Teresa and her son decided to explore the results using augmented reality.On the bottom left corner of the tablet screen, the floor plan that was previously uploaded was presented with various environmental barriers marked on it.As they moved around the house holding the tablet, they noticed a red dot in the living room between the sofa and an armchair (Fig. 1).
As Teresa tapped the red dot, the space between the sofa and the armchair was highlighted with a mark showing the width of the space between the two elements.Simultaneously a popup appeared on the screen with the message "Mobility Barrier: There isn't enough space between the two pieces of furniture" which was also read out loud by the OLA app (Fig. 2).As she tapped the mark on the screen, a new popup appeared on the screen with the message "Recommendation 1: Move the Armchair away from the Sofa; Recommendation 2: Remove one of the elements" which was also read out loud by OLA (Fig. 3 and Fig. 4).Teresa and her son then closed all the popups and continued to examine the remaining environmental barriers that were identified in her house.By providing an overview of the environmental barriers existing in her home, OLA helped Teresa and her son to correct them and avoid potential hazards.

Mobility Hazards
We opted to follow a size-based approach to our Architectural Barriers analysis.To support this analysis and to pinpoint the norms that are considered by our tools, we based our work on Portuguese, English, Swedish and Polish "Design for All" rules and regulations.In addition to the environmental barriers and hazards, which can be detected by our computer vision-based Environment Analysis system (e.g.distances between furniture and furniture and wall or cables loose on the floor), some other types were considered to more easily be detected through direct user feedback (e.g.slippery steps and steps with edges difficult to differentiate).In these cases, a question will be prompted when the respective architectural element is detected (e.g stairs, steps, platforms).

Technology development
The OLA EA-SA system pipeline, encompasses several logical steps: i) Object Recognition at the Category level so that the system can recognize different pieces of furniture and indoor building elements such as walls, ii) identification of the bounding box of each piece of furniture or indoor building element iii) determination of distances between detected objects, based on the bounding boxes iv) definition and implementation of action rules when the encountered distance values are not according to the regulations, through the suggestion of changes in the environment (Fig. 5).These could be simply performed by moving apart neighbouring pieces of furniture that have regulation offending distances.The developed features follow the requirements derived from scenario 1, namely, by checking distances between different pieces of furniture, automatically identified and classified in the indoor space.This section deals with the task of Object Recognition at the Category level, predicting the category of never-before-seen objects, using as input, a-priori knowledge of a set of objects (i.e. a dataset) (Proença, Gaspar, & Dias, 2015).
We have built a special-purpose dataset for this research, which used the Microsoft Kinect Sensor to acquire both RGB and Depth data, of a set of object instances belonging to various categories (or classes).The RGB channel stores de visual appearance of one object (e.g.colour, texture, or silhouette), whereas the depth channel has 3D information of the object, like shape and size, which are handy for our recognition purposes.Using our in-house data capture setup (Pascoal et al., 2015), a dataset comprising eight object categories (i.e.arm chair, bed, chair, medium height furniture, sofa, table, tall pieces of furniture), with a total of 76 object instances, was constructed.Due to the limitations in terms of size of the turntable rig used for data capture (Fig. 6. ), we used miniatures of pieces of furniture that typically exist in most houses.To extrapolate the use of our dataset to real sized furniture, a scaling factor was applied to the data captured by the Kinect One sensor.The data from the dataset was also segmented in a post-processing stage, to enable its use in our recognition framework.
Our basic technique is described in more detail in (Proença et al., 2015) was applied to the mentioned dataset.At testing time, given a query frame (i.e. a point cloud captured from the RGB-D sensor) from a never-seen before object, we extract DSIFT, SPIN and SHOT (Proença et al., 2015) descriptors from the query frame, following the same technique of the training phase.Then the detected object is classified to a category.The result of the category recognition is presented by a bounding box with a label of the category in the RGB channel of the Kinect sensor (Fig. 7).Apart from the object category recognition, the system also determines the distance between the bounding boxes of neighboring objects.There is no limitation in the number of objects present on the scene.The determination of the distance is crucial when it comes to identifying architectural barriers inside the house, for a scenario, comprising live data acquisition of real size objects, as presented in the mock-up examples (Fig. 7).

DISCUSSION, CONCLUSIONS AND FUTURE WORK
Falls are a commonly found problem within elderly and their consequences are timely and difficult to overcome, involving costly stays in the hospital, time and dedication of informal and formal caregivers and most of all, lost in the quality of life for the elderly.
This paper presented and discussed an Environmental Analysis and Safety Advisor system devel-oped in the scope of AAL OLA, an ambient assisted living research project funded by the AAL Joint Programme (AAL JP) and National Authorities in Portugal, Sweden and Hungary.This system was developed through a methodology comprising the following stages: i) definition of personas; ii) definition of scenarios based on the personas and their expected tasks; iii) definition of the user requirement, features and system design; iv) backend system development; v) frontend system development.
The OLA EA-SA system uses in-house developed computer vision algorithms for real-time object recognition and scene understanding, based on data collected from RGB-D sensor, and adopts universal design principles and augmented reality as visual interface with the end-user.The OLA EA-SA system, after automatically recognizing indoor architectural elements present in a house (e.g.furniture, walls, carpets, cables), detects potential hazards and advises the caregiver (end-user) on what action to take to turn such home in a safer place for the elderly.
AAL OLA in general, and the Environmental Analysis and Safety Advisor System in particular aim at responding to the growing number of elderly living alone and to the need of providing these citizens with tools that enable more autonomy in their daily living.In this context, such system is a contribution to help the elderly and their informal and formal caregivers, to analyze and decide which actions to take in order to ensure that elderly's homes do not have indoor architectonic barriers that may be potential dangerous, notably, provoking fall hazards.AAL OLA included this system in the global OLA set of technologies aiming at an integrated solution for improving the elderly quality of life.Several improvements are currently being implemented in the system that for now considers a simplified version of our proposed Augmented Reality Application.It uses miniature versions of selected furniture and determines distances between them, signaling when those distances are below a certain threshold value.Therefore, future improvements of our implementation, should consider the recognition of full-sized furniture, and the detec-tion of more cases of indoor architectural barriers, like for example the distance between furniture and walls (including the detection of the latter).Future work include the test of our system with user groups, will be done on a first stage on a controlled environment with few objects, by running the OLA App by caregivers and testing if minimum distances between furniture are respected, to ease the mobility of the seniors.
Figure 1 Mock-up of the OLA EA-SA module showing an alert sign near two pieces of furniture after detecting a hazard.
Figure 2 Mock-up of the OLA EA-SA module showing that, when pressed, shows the found hazard situation.
Figure 3 Mock-up of the OLA EA-SA module showing user taping on the screen.
Figure 4 Mock-up of the OLA EA-SA module showing user selecting the hazard and a recommended action for the short distance hazard.
Figure 5 OLA EA-SA system pipeline