Gabor Barcodes for Medical Image Retrieval

In recent years, advances in medical imaging have led to the emergence of massive databases, containing images from a diverse range of modalities. This has significantly heightened the need for automated annotation of the images on one side, and fast and memory-efficient content-based image retrieval systems on the other side. Binary descriptors have recently gained more attention as a potential vehicle to achieve these goals. One of the recently introduced binary descriptors for tagging of medical images are Radon barcodes (RBCs) that are driven from Radon transform via local thresholding. Gabor transform is also a powerful transform to extract texture-based information. Gabor features have exhibited robustness against rotation, scale, and also photometric disturbances, such as illumination changes and image noise in many applications. This paper introduces Gabor Barcodes (GBCs), as a novel framework for the image annotation. To find the most discriminative GBC for a given query image, the effects of employing Gabor filters with different parameters, i.e., different sets of scales and orientations, are investigated, resulting in different barcode lengths and retrieval performances. The proposed method has been evaluated on the IRMA dataset with 193 classes comprising of 12,677 x-ray images for indexing, and 1,733 x-rays images for testing. A total error score as low as $351$ ($\approx 80\%$ accuracy for the first hit) was achieved.Comment: To appear in proceedings of The 2016 IEEE International Conference on Image Processing (ICIP 2016), Sep 25-28, 2016, Phoenix, Arizona, US

An Intelligent Ambulatory Fall Risk Assessment Method Based on the Detection of Compensatory Balance Reactions and Environmental Factors

Falls in older adults are a critical public health problem worldwide and impact one in three older adults at least once each year. In addition to physical consequences (e.g., hip fracture) falls can lead to negative psychological outcomes, such as depression. Fall risk assessment (FRA) is the initial step for fall prevention programs and interventions. In particular, clinicians aim to understand what factors put older adults at high risk of falling to inform the selection and timing of fall prevention interventions (e.g., strengthening programs). These risk factors are generally categorized as intrinsic or biological (e.g., gait and balance disorders) and extrinsic or environmental (e.g., slippery surfaces). While supervised FRAs, including performance-based (e.g., Timed up and Go) and instrumented methods (e.g., motion capture systems), capable of quantifying intrinsic risks have advanced significantly, falls still remain a major priority in geriatric medicine and public health. This can be due to the Hawthorne effect, the heterogeneous nature of older adults' health, lifestyle, and behaviors, and the complex, multifactorial etiology of falls. To address the limitation of supervised FRAs, a growing body of literature has focused on wearable sensor-based methods for free-living (or ambulatory) FRA. These studies, reviewed in Chapter 2, investigated the relationships between free-living digital biomarkers (FLDBs) extracted from wearable sensors data (generally, inertial data) and the frequency of prospective/retrospective falls in older adults. However, many FLDBs exhibited inconsistent fall predictive powers across studies, indicating they may not be stable in distinguishing fall-prone individuals. Moreover, the relationships between falls and free-living dynamic postural control measures, such as step width and the frequency of naturally-occurring compensatory balance reactions (CBRs), have yet to be investigated in depth. Considering controlled studies reported balance impairment as one of the strongest risk factors for falls, the investigation of balance-related FLDBs may lead to more stable risk assessments and provide new insights into fall prevention in older adults. Although gait-related FLDBs extracted from inertial data can be impacted by both intrinsic and environmental factors, their respective impacts have not been differentiated by the majority of free-living FRA methods. This may lead to the ambiguous interpretation of the subsequent FLDBs, and less precise intervention strategies to prevent falls. A context-aware free-living FRA would elucidate the interplay between intrinsic and environmental risk factors and clarifies their respective impacts on fall predictive powers of FLDBs. This may subsequently enable clinicians to target more specific intervention strategies including environmental modification (e.g., eliminating tripping hazards) and/or rehabilitation interventions (e.g., training to negotiate stairs/transitions). This doctoral thesis aims to address the aforementioned research gaps by proposing multiple machine learning frameworks and incorporating an egocentric camera along with wearable inertial measurement units (IMUs). Chapter 3 discusses the development of random forest models to differentiate between normal gait episodes and multidirectinoal CBRs (e.g, slip-like, trip-like, sidestep) elicited by a perturbation treadmill in controlled conditions in healthy young adults, where the CBR detection model achieved the overall accuracy of ~96%. This chapter established the infrastructure for Chapter 4, where a validation study was performed to detect older adults' CBRs under free-living conditions. Random forest models were trained on independent/unseen datasets curated from multiple sources, including perturbation treadmill CBRs. By investigating 11 fallers' and older non-fallers' free-living criterion standard data, 8 naturally-occurring CBRs, i.e., 7 trips (self-reported using a wrist-mounted voice-recorder) and 1 hit/bump (verified using egocentric vision data) were localized in the corresponding trunk-mounted IMU data. A subset of models differentiated between naturally-occurring CBRs and free-living activities with high sensitivity (100%) and specificity (~99%) suggesting that accurate detection of naturally-occurring CBRs is feasible. Moreover, to address the limitations of IMUs in terms of the estimation of step width in free-living conditions, Chapter 5 presents a novel markerless deep learning-based model to obtain gait patterns by localizing feet in the egocentric vision data captured by a waist-mounted camera. With the aim of improving the interpretability of gait-related FLDBs and investigating the impact of environment on older adults' gait, Chapter 6 proposes a vision-based framework to automatically detect the most common level walking surfaces. Using a belt-mounted camera and IMUs worn by fallers and non-fallers (mean age 73.6 yrs), a unique dataset was acquired (a subset of Multimodal Ambulatory Gait and Fall Risk Assessment in the Wild (MAGFRA-W) dataset). A series of ConvNets were developed: EgoPlaceNet categorizes frames into indoor and outdoor; and EgoTerrainNet (with outdoor and indoor versions) detects the enclosed terrain type in patches. EgoPlaceNet detected outdoor and indoor scenes in MAGFRA-W with 97.36% and 95.59% (leave-one-subject-out) accuracies, respectively. EgoTerrainNet-Indoor and -Outdoor achieved high detection accuracies for pavement (87.63%), foliage (91.24%), gravel (95.12%), and high-friction materials (95.02%), which indicate the models' high generalizabiliy. Overall, promising results encourage the integration of wearable cameras and machine learning approaches to complement IMU-based free-living FRAs, towards stable context-aware FLDBs for fall prevention in older adults. Implications for further research to examine the relationships between naturally-occurring CBRs and fall risk, and clinical applications are discussed

Machine Learning-based Detection of Compensatory Balance Responses and Environmental Fall Risks Using Wearable Sensors

Falls are the leading cause of fatal and non-fatal injuries among seniors worldwide, with serious and costly consequences. Compensatory balance responses (CBRs) are reactions to recover stability following a loss of balance, potentially resulting in a fall if sufficient recovery mechanisms are not activated. While performance of CBRs are demonstrated risk factors for falls in seniors, the frequency, type, and underlying cause of these incidents occurring in everyday life have not been well investigated. This study was spawned from the lack of research on development of fall risk assessment methods that can be used for continuous and long-term mobility monitoring of the geri- atric population, during activities of daily living, and in their dwellings. Wearable sensor systems (WSS) offer a promising approach for continuous real-time detection of gait and balance behavior to assess the risk of falling during activities of daily living. To detect CBRs, we record movement signals (e.g. acceleration) and activity patterns of four muscles involving in maintaining balance using wearable inertial measurement units (IMUs) and surface electromyography (sEMG) sensors. To develop more robust detection methods, we investigate machine learning approaches (e.g., support vector machines, neural networks) and successfully detect lateral CBRs, during normal gait with accuracies of 92.4% and 98.1% using sEMG and IMU signals, respectively. Moreover, to detect environmental fall-related hazards that are associated with CBRs, and affect balance control behavior of seniors, we employ an egocentric mobile vision system mounted on participants chest. Two algorithms (e.g. Gabor Barcodes and Convolutional Neural Networks) are developed. Our vision-based method detects 17 different classes of environmental risk factors (e.g., stairs, ramps, curbs) with 88.5% accuracy. To the best of the authors knowledge, this study is the first to develop and evaluate an automated vision-based method for fall hazard detection

Radon-Gabor Barcodes for Medical Image Retrieval

In recent years, with the explosion of digital images on the Web, content-based retrieval has emerged as a significant research area. Shapes, textures, edges and segments may play a key role in describing the content of an image. Radon and Gabor transforms are both powerful techniques that have been widely studied to extract shape-texture-based information. The combined Radon-Gabor features may be more robust against scale/rotation variations, presence of noise, and illumination changes. The objective of this paper is to harness the potentials of both Gabor and Radon transforms in order to introduce expressive binary features, called barcodes, for image annotation/tagging tasks. We propose two different techniques: Gabor-of-Radon-Image Barcodes (GRIBCs), and Guided-Radon-of-Gabor Barcodes (GRGBCs). For validation, we employ the IRMA x-ray dataset with 193 classes, containing 12,677 training images and 1,733 test images. A total error score as low as 322 and 330 were achieved for GRGBCs and GRIBCs, respectively. This corresponds to $\approx 81\%$ retrieval accuracy for the first hit.Comment: To appear in proceedings of the 23rd International Conference on Pattern Recognition (ICPR 2016), Cancun, Mexico, December 201

First-person Vision-based Assessment of Fall Risks in The Wild, Towards Fall Prevention in Older Adults

Falls in older adults is one of the most important public health problems world-wide. In our previous works, we showed that first-personvision (FPV) data acquired by chest- and waist-mounted camerashave the potential to be utilized to (A) develop novel markerlessdeep models to estimate spatiotemporal gait parameters over time(e.g., step width) by localizing feet in 2D coordinate system of RGBframes (using optical flow and RGB streams) and (B) automaticallyidentify environmental hazards (e.g., curbs, stairs, different terrains)that may lead to falling. In this paper, a summary of our recent FPV-based approaches for fall risk assessment in the wild are being discussed. These approaches aimed to eventually inform clinical decisions on the most appropriate prevention interventions to reducefall incidence in older populations

First-person Vision-based Assessment of Fall Risks in The Wild, Towards Fall Prevention in Older Adults

Falls in older adults is one of the most important public health problems world-wide. In our previous works, we showed that first-personvision (FPV) data acquired by chest- and waist-mounted camerashave the potential to be utilized to (A) develop novel markerlessdeep models to estimate spatiotemporal gait parameters over time(e.g., step width) by localizing feet in 2D coordinate system of RGBframes (using optical flow and RGB streams) and (B) automaticallyidentify environmental hazards (e.g., curbs, stairs, different terrains)that may lead to falling. In this paper, a summary of our recent FPV-based approaches for fall risk assessment in the wild are being discussed. These approaches aimed to eventually inform clinical decisions on the most appropriate prevention interventions to reducefall incidence in older populations

Fall risk assessment in the wild: A critical examination of wearable sensor use in free-living conditions

Background Despite advances in laboratory-based supervised fall risk assessment methods (FRAs), falls still remain a major public health problem. This can be due to the alteration of behavior in laboratory due to the awareness of being observed (i.e., Hawthorne effect), the multifactorial complex etiology of falls, and our limited understanding of human behaviour in natural environments, or in the’ wild’. To address these imitations, a growing body of literature has focused on free-living wearable-sensor-based FRAs. The objective of this narrative literature review is to discuss papers investigating natural data collected by wearable sensors for a duration of at least 24 h to identify fall-prone older adults. Methods Databases (Scopus, PubMed and Google Scholar) were searched for studies based on a rigorous search strategy. Results Twenty-four journal papers were selected, in which inertial sensors were the only wearable system employed for FRA in the wild. Gait was the most-investigated activity; but sitting, standing, lying, transitions and gait events, such as turns and missteps, were also explored. A multitude of free-living fall predictors (FLFPs), e.g., the quantity of daily steps, were extracted from activity bouts and events. FLFPs were further categorized into discrete domains (e.g., pace, complexity) defined by conceptual or data-driven models. Heterogeneity was found within the reviewed studies, which includes variance in: terminology (e.g., quantity vs macro), hyperparameters to define/estimate FLFPs, models and domains, and data processing approaches (e.g., the cut-off thresholds to define an ambulatory bout). These inconsistencies led to different results for similar FLFPs, limiting the ability to interpret and compare the evidence. Conclusion Free-living FRA is a promising avenue for fall prevention. Achieving a harmonized model is necessary to systematically address the inconsistencies in the field and identify FLFPs with the highest predictive values for falls to eventually address intervention programs and fall prevention

Automated Detection of Older Adults’ Naturally-Occurring Compensatory Balance Reactions: Translation From Laboratory to Free-Living Conditions

Objective: Older adults’ falls are a critical public health problem. The majority of free-living fall risk assessment methods have investigated fall predictive power of step-related digital biomarkers extracted from wearable inertial measurement unit (IMU) data. Alternatively, the examination of characteristics and frequency of naturally-occurring compensatory balance reactions (CBRs) may provide valuable information on older adults’ propensity for falls. To address this, models to automatically detect naturally-occurring CBRs are needed. However, compared to steps, CBRs are rare events. Therefore, prolonged collection of criterion standard data (along with IMU data) is required to validate model’s performance in free-living conditions. Methods: By investigating 11 fallers’ and older non-fallers’ free-living criterion standard data, 8 naturally-occurring CBRs, i.e., 7 trips (self-reported using a wrist-mounted voice-recorder) and 1 hit/bump (verified using egocentric vision data) were localized in the corresponding trunk-mounted IMU data. Random forest models were trained on independent/unseen datasets curated from multiple sources, including in-lab data captured using a perturbation treadmill. Subsequently, the models’ translation/generalization to older adults’ out-of-lab data were assessed. Results: A subset of models differentiated between naturally-occurring CBRs and free-living activities with high sensitivity (100%) and specificity (≥99%). Conclusions: The findings suggest that accurate detection of naturally-occurring CBRs is feasible. Clinical/Translational Impact- As a multi-institutional validation study to detect older adults’ naturally-occurring CBRs, suitability for larger-scale free-living studies to investigate falls etiology, and/or assess the effectiveness of perturbation training programs is discussed

Egocentric vision-based detection of surfaces: towards context-aware free-living digital biomarkers for gait and fall risk assessment

Background: Falls in older adults are a critical public health problem. As a means to assess fall risks, free-living digital biomarkers (FLDBs), including spatiotemporal gait measures, drawn from wearable inertial measurement unit (IMU) data have been investigated to identify those at high risk. Although gait-related FLDBs can be impacted by intrinsic (e.g., gait impairment) and/or environmental (e.g., walking surfaces) factors, their respective impacts have not been differentiated by the majority of free-living fall risk assessment methods. This may lead to the ambiguous interpretation of the subsequent FLDBs, and therefore, less precise intervention strategies to prevent falls. Methods: With the aim of improving the interpretability of gait-related FLDBs and investigating the impact of environment on older adults’ gait, a vision-based framework was proposed to automatically detect the most common level walking surfaces. Using a belt-mounted camera and IMUs worn by fallers and non-fallers (mean age 73.6 yrs), a unique dataset (i.e., Multimodal Ambulatory Gait and Fall Risk Assessment in the Wild (MAGFRA-W)) was acquired. The frames and image patches attributed to nine participants’ gait were annotated: (a) outdoor terrains: pavement (asphalt, cement, outdoor bricks/tiles), gravel, grass/foliage, soil, snow/slush; and (b) indoor terrains: high-friction materials (e.g., carpet, laminated floor), wood, and tiles. A series of ConvNets were developed: EgoPlaceNet categorizes frames into indoor and outdoor; and EgoTerrainNet (with outdoor and indoor versions) detects the enclosed terrain type in patches. To improve the framework’s generalizability, an independent training dataset with 9,424 samples was curated from different databases including GTOS and MINC-2500, and used for pretrained models’ (e.g., MobileNetV2) fine-tuning. Results: EgoPlaceNet detected outdoor and indoor scenes in MAGFRA-W with 97.36% and 95.59% (leave-one-subject-out) accuracies, respectively. EgoTerrainNet-Indoor and -Outdoor achieved high detection accuracies for pavement (87.63%), foliage (91.24%), gravel (95.12%), and high-friction materials (95.02%), which indicate the models’ high generalizabiliy. Conclusions: Encouraging results suggest that the integration of wearable cameras and deep learning approaches can provide objective contextual information in an automated manner, towards context-aware FLDBs for gait and fall risk assessment in the wild