ENVIRONMENT PERCEPTION FOR MICROMOBILITY APPLICATIONS

The objective of this project is to see how deep learning technologies, specifically image recognition features and RNNs, can help to make the life of everybody living in the city safer, and help preventing some of the accidents that occur every day, by implementing a functional system capable of detecting the type of lane a Personal Mobility Device (PMD) is circulating in the city of Barcelona. It is also necessary that among the five type of lanes to predict, a higher importance is given to the identification of wether a PMD is circulating or not through the sidewalk, where there is a higher risk of accidents involving pedrestians, and velocity should be reduced more. Besides this, another objective of the thesis will be the implementation, training and evaluation of various deep Recurrent Neural Networks, and posteriorly the comparison with simple convolutional neural networks and viability analysis of these newer technologies applied to PMD

Consistent Video Saliency Using Local Gradient Flow Optimization and Global Refinement

We present a novel spatiotemporal saliency detection method to estimate salient regions in videos based on the gradient flow field and energy optimization. The proposed gradient flow field incorporates two distinctive features: 1) intra-frame boundary information and 2) inter-frame motion information together for indicating the salient regions. Based on the effective utilization of both intra-frame and inter-frame information in the gradient flow field, our algorithm is robust enough to estimate the object and background in complex scenes with various motion patterns and appearances. Then, we introduce local as well as global contrast saliency measures using the foreground and background information estimated from the gradient flow field. These enhanced contrast saliency cues uniformly highlight an entire object. We further propose a new energy function to encourage the spatiotemporal consistency of the output saliency maps, which is seldom explored in previous video saliency methods. The experimental results show that the proposed algorithm outperforms state-of-the-art video saliency detection methods

Better Images : Understanding and Measuring Subjective Image-Quality

The objective in this thesis was to examine the psychological process of image-quality estimation, specifically focusing on people who are naïve in this respect and on how they estimate high-quality images. Quality estimation in this context tends to be a preference task, and to be subjective. The aim in this thesis is to enhance understanding of viewing behaviour and estimation rules in the subjective assessment of image-quality. On a more general level, the intention is to shed light on estimation processes in preference tasks. An Interpretation-Based Quality (IBQ) method was therefore developed to investigate the rules used by naïve participants in their quality estimations. It combines qualitative and quantitative approaches, and complements standard methods of image-quality measurement. The findings indicate that the content of the image influences perceptions of its quality: it influences how the interaction between the content and the changing image features is interpreted (Study 1). The IBQ method was also used to create three subjective quality dimensions: naturalness of colour, darkness and sharpness (Study 2). These dimensions were used to describe the performance of camera components. The IBQ also revealed individual differences in estimation rules: the participants differed as to whether they included interpretation of the changes perceived in an image in their estimations or whether they just commented on them (Study 4). Viewing behaviour was measured to enable examination of the task properties as well as the individual differences. Viewing behaviour was compared in two tasks that are commonly used in studies on image-quality estimation: the estimation of difference and the estimation of difference in quality (Study 3). The results showed that viewing behaviour differed even in two magnitude-estimation tasks with identical material. When they were estimating quality the participants concentrated mainly on the semantically important areas of the image, whereas in the difference-estimation task they also examined wider areas. Further examination of quality-estimation task revealed individual differences in the viewing behaviour and in the importance these viewing behaviour groups attached to the interpretation of changes in their estimations (Study 4). It seems that people engaged in a subjective preference-estimation task use different estimation rules, which is also reflected in their viewing behaviour. The findings reported in this thesis indicate that: 1) people are able to describe the basis of their quality estimations even without training when they are allowed to use their own vocabulary; 2) the IBQ method has the potential to reveal the rules used in quality estimation; 3) changes in instructions influence the way people search for information from the images; and 4) there are individual differences in terms of rules and viewing behaviour in quality-estimation tasks.Tämä väitöskirja käsittelee subjektiivista kuvanlaadun arviointiprosessia, etenkin keskittyen kuvanlaadun arvioinnin suhteen kouluttamattomien ihmisten korkea laatuisten kuvien arviointiin. Kuvanlaadulla tarkoitetaan tässä kuvan prosessointiin liittyviä tekijöitä. Tavoitteena on lisätä ymmärrystä kuvanlaadun arviointiprosessista ja sen mittaamisesta. Kuvanlaadun arviointiprosessissa on yleisesti keskitytty saamaan yksi arvio laadusta tai yksi arvio jollain etukäteen määritellyllä skaalalla. Tällöin emme tiedä mihin kouluttamaton arvioitsija olisi kiinnittänyt huomionsa ja millä perusteilla hän olisi kuvaa arvioinut. Tätä selvittämään kehitimme menetelmän, jolla voimme tarkastella ihmisten arvioissaan käyttämiä perusteita. Ihmiset kuvailivat perusteita vapaasti ja kun he saivat käyttää omaa sanastoaan, he olivat myös johdonmukaisia arvioissaan. Tätä menetelmää käytettiin myös selvittämään subjektiivisia kuvanlaatu-ulottuvuuksia, joita olivat värien luonnollisuus, tummuus ja tarkkuus. Toinen osa väitöskirjaa käsittelee kuvanlaadun arviointitehtävää prosessina. Selvitimme miten pieni muutos koehenkilöille annetussa ohjeistuksessa muuttaa heidän tapaansa katsella kuvaa heidän tehdessä siihen liittyviä arvioita. Tehtävänä oli kahdessa kuvassa näkyvien erojen arviointi joko erojen suuruuden tai kuvanlaadun erojen mukaan. Kuvanlaatua arvioitaessa huomio kiinnittyi enemmän kohtiin, jotka olivat semanttisesti merkityksellisiä, kun eroja arvioitaessa laajempi alue otettiin huomioon. Tarkastelimme myös kuvanlaadunarviointeihin liittyviä yksilöiden välisiä eroja. Koehenkilöt pystyttiin jakamaan kolmeen ryhmään heidän katselutapojensa perusteella. Nämä katselutaparyhmät erosivat toisistaan myös siinä kuinka paljon he käyttivät arvioinneissaan perusteina vaikutelmia, jotka syntyivät kuvanlaadun muutosten pohjalta. Toiset keskittyivät arvioimaan kuvanlaatua siihen liittyvien attribuuttien mukaan, kun toiset käyttivät perusteina näiden attribuuttien kuvan viestiin synnyttämiä vaikutelmia. Korkean kuvanlaadun arvioinnissa on usein kyseessä mieltymyksiin perustuva laadun arviointi. Tällöin on tärkeää antaa ihmisten käyttää omia käsitteitään, sekä ottaa huomioon että pienimmätkin tekijät, kuten sanavalinnat kysymyksissä ja ihmisten väliset eroavuudet, vaikuttavat arviointeihin. Tämä väitöskirja antaa eväitä tarkastella arviointiprosessia

Smart and Secure Augmented Reality for Assisted Living

Augmented reality (AR) is one of the biggest technology trends which enables people to see the real-life surrounding environment with a layer of virtual information overlaid on it. Assistive devices use this match of information to help people better understand the environment and consequently be more efficient. Specially, AR has been extremely useful in the area of Ambient Assisted Living (AAL). AR-based AAL solutions are designed to support people in maintaining their autonomy and compensate for slight physical and mental restrictions by instructing them on everyday tasks. The discovery of visual attention for assistive aims is a big challenge since in dynamic cluttered environments objects are constantly overlapped and partial object occlusion is also frequent. Current solutions use egocentric object recognition techniques. However, the lack of accuracy affects the system's ability to predict users’ needs and consequently provide them with the proper support. Another issue is the manner that sensitive data is treated. This highly private information is crucial for improving the quality of healthcare services. However, current blockchain approaches are used only as a permission management system, while the data is still stored locally. As a result, there is a potential risk of security breaches. Privacy risk in the blockchain domain is also a concern. As major investigation tackles privacy issues based on off-chain approaches, there is a lack of effective solutions for providing on-chain data privacy. Finally, the Blockchain size has been shown to be a limiting factor even for chains that store simple transactional data, much less the massive blocks that would be required for storing medical imaging studies. To tackle the aforementioned major issues, this research proposes a framework to provide a smarter and more secure AR-based solution for AAL. Firstly, a combination of head-worn eye-trackers cameras with egocentric video is designed to improve the accuracy of visual attention object recognition in free-living settings. A heuristic function is designed to generate a probability estimation of visual attention over objects within an egocentric video. Secondly, a novel methodology for the storage of large sensitive AR-based AAL data is introduced in a decentralized fashion. By leveraging the power of the IPFS (InterPlanetary File System) protocol to tackle the lack of storage issue in the Blockchain. Meanwhile, a blockchain solution on the Secret Network blockchain is developed to tackle the existent lack of privacy on smart contracts, which provides data privacy at both transactional and computational levels. In addition, is included a new off-chain solution encapsulates a governing body for permission management purposes to solve the problem of the lost or eventual theft of private keys. Based on the research findings, that visual attention-object detection approach is applicable to cluttered environments which presents a transcend performance compared to the current methods. This study also produced an egocentric indoor dataset annotated with human fixation during natural exploration in a cluttered environment. Comparing to previous works, this dataset is more realistic because it was recorded in real settings with variations in terms of objects overlapping regions and object sizes. With respect to the novel decentralized storage methodology, results indicate that sensitive data can be stored and queried efficiently using the Secret Network blockchain. The proposed approach achieves both computational and transactional privacy with significantly less cost. Additionally, this approach mitigates the risk of permanent loss of access to the patient on-chain data records. The proposed framework can be applied as an assistive technology in a wide range of sectors that requires AR-based solution with high-precision visual-attention object detection, efficient data access, high-integrity data storage and full data privacy and security

Are All Pixels Equally Important? Towards Multi-Level Salient Object Detection

When we look at our environment, we primarily pay attention to visually distinctive objects. We refer to these objects as visually important or salient. Our visual system dedicates most of its processing resources to analyzing these salient objects. An analogous resource allocation can be performed in computer vision, where a salient object detector identifies objects of interest as a pre-processing step. In the literature, salient object detection is considered as a foreground-background segmentation problem. This approach assumes that there is no variation in object importance. Only the most salient object(s) are detected as foreground. In this thesis, we challenge this conventional methodology of salient-object detection and introduce multi-level object saliency. In other words, all pixels are not equally important. The well-known salient-object ground-truth datasets contain images with single objects and thus are not suited to evaluate the varying importance of objects. In contrast, many natural images have multiple objects. The saliency levels of these objects depend on two key factors. First, the duration of eye fixation is longer for visually and semantically informative image regions. Therefore, a difference in fixation duration should reflect a variation in object importance. Second, visual perception is subjective; hence the saliency of an object should be measured by averaging the perception of a group of people. In other words, objective saliency can be considered as the collective human attention. In order to better represent natural images and to measure the saliency levels of objects, we thus collect new images containing multiple objects and create a Comprehensive Object Saliency (COS) dataset. We provide ground truth multi-level salient object maps via eye-tracking and crowd-sourcing experiments. We then propose three salient-object detectors. Our first technique is based on multi-scale linear filtering and can detect salient objects of various sizes. The second method uses a bilateral-filtering approach and is capable of producing uniform object saliency values. Our third method employs image segmentation and machine learning and is robust against image noise and texture. This segmentation-based method performs the best on the existing datasets compared to our other methods and the state-of-the-art methods. The state-of-the-art salient-object detectors are not designed to assess the relative importance of objects and to provide multi-level saliency values. We thus introduce an Object-Awareness Model (OAM) that estimates the saliency levels of objects by using their position and size information. We then modify and extend our segmentation-based salient-object detector with the OAM and propose a Comprehensive Salient Object Detection (CSD) method that is capable of performing multi-level salient-object detection. We show that the CSD method significantly outperforms the state-of-the-art methods on the COS dataset. We use our salient-object detectors as a pre-processing step in three applications. First, we show that multi-level salient-object detection provides more relevant semantic image tags compared to conventional salient-object detection. Second, we employ our salient-object detector to detect salient objects in videos in real time. Third, we use multi-level object-saliency values in context-aware image compression and obtain perceptually better compression compared to standard JPEG with the same file size

Simulator Sickness in Fahrsimulationsumgebungen - drei Studien zu Human Factors

Die wachsende Popularität von Fahrsimulationen in Forschung und Praxis rückt auch die Interaktion von Mensch und Maschine in den Fokus (Rizzo, Sheffield, Stierman & Dawson, 2003). Zentral ist dabei die Untersuchung potentieller negativer Nebeneffekte wie Simulator Sickness (Biernacki & Dziuda, 2014; Brucks & Watters, 2009), was Symptome von Übelkeit, Okulomotorik oder Desorientierung umfasst. Die Untersuchung psychologischer Korrelate von Simulator Sickness ist unterrepräsentiert (Milleville-Pennel & Charron, 2015). Die vorliegende Dissertation beschäftigt sich aus diesem Grund mit der Forschungsfrage, welche Human Factors mit dem Erleben von Simulator Sickness in Fahrsimulations-umgebungen verbunden sind. Um dieser Fragestellung nachzugehen, wurden drei Untersuchungen durchgeführt. Die erste Untersuchung widmete sich der Beziehung zwischen aktuell erlebten physischen sowie psychischen Beschwerden und der Ausprägung von Simulator Sickness nach einer Fahrsimulationsexposition. Statistische Analysen ergaben, dass physische Beschwerden kein signifikanter Prädiktor für das Erleben von Simulator Sickness waren, psychische Beschwerden hingegen schon. Die zweite Untersuchung widmete sich der Beziehung zwischen visueller Aufmerksamkeitsleistung und dem Erleben von Simulator Sickness. Es konnten keine signifikante Beziehung zwischen visueller Aufmerksamkeitsleistung und den Skalen des Simulator Sickness Questionnaires aufgezeigt werden. Die dritte Untersuchung widmete sich zwei Fragestellungen: Zum einen sollte herausgefunden werden, welche Fahrertypen anhand verschiedener Human Factors ermittelt werden können, zum anderen sollte untersucht werden, ob sich die Fahrertypen (im Sinne von Merkmalskombinationen verschiedener Human Factors) in ihrem Erleben von Simulator Sickness unterscheiden. Es konnten vier Fahrertypen identifiziert werden (ängstlich, leichtsinnig, vorsichtig und aggressiv), welche sich hinsichtlich des Erlebens von Simulator Sickness nicht signifikant unterschieden. Die durchgeführten Untersuchungen sind limitiert durch die Nutzung jeweils einer Simulationsaufgabe. Künftige Forschung sollte Schwierigkeitsgrade der Aufgaben variieren und experimentelle Designs nutzen. Die Untersuchungen unterstreichen allerdings die Relevanz der Erforschung der Beziehung von Human Factors und Simulator Sickness, welche noch zahlreiche Forschungslücken aufweist.Driving simulations grow in importance in research and practice as well as the interaction of humans and machines (Rizzo, Sheffield, Stierman & Dawson, 2003), especially concerning investigations on potential negative side effects like simulator sickness (Biernacki & Dziuda, 2014; Brucks & Watters, 2009). The phenomena consists of symptoms like nausea, oculomotor, and disorientation. Thus, the investigation of psychological correlates of simulator sickness is scarce (Milleville-Pennel & Charron, 2015). This dissertation deals with the research question which human factors are related with the experience of simulator sickness in driving simulation environments. For this purpose, three studies were conducted. The first study dealt with the relationship physical and psychological complaints and the experience of simulator sickness after the exposition to a driving simulation. Statistical analyses showed that physical complaints are not a significant predictor for simulator sickness, whereas psychological complaints were find to be a significant predictor. The second study addressed the relationship between visual attention and simulator sickness. The results did not show a significant correlation between these variables. The third study focused on driver types that were identified based on combinations of individual characteristics. Furthermore, this study aimed to examine if the driver types differ concerning their experience of simulator sickness. Four driver types were identified (anxious, careless, cautious, and aggressive) that did not differ significantly in the experience of simulator sickness. The studies are limited due to the use of only one simulation task, respectively. Further research should vary the task difficulty and should use experimental designs. Nonetheless, the studies stressed out the importance of examinations on the relationship of human factors and simulator sickness

Visual saliency in image quality assessment

Advances in image quality assessment have shown the benefits of modelling functional components of the human visual system in image quality metrics. Visual saliency, a crucial aspect of the human visual system, is increasingly investigated recently. Current applications of visual saliency in image quality metrics are limited by our knowledge on the relation between visual saliency and quality perception. Issues regarding how to simulate and integrate visual saliency in image quality metrics remain. This thesis presents psychophysical experiments and computational models relevant to the perceptually-optimised use of visual saliency in image quality metrics. We first systematically validated the capability of computational saliency in improving image quality metrics. Practical guidance regarding how to select suitable saliency models, which image quality metrics can benefit from saliency integration, and how the added value of saliency depends on image distortion type were provided. To better understand the relation between saliency and image quality, an eye-tracking experiment with a reliable experimental methodology was first designed to obtain ground truth fixation data. Significant findings on the interactions between saliency and visual distortion were then discussed. Based on these findings, a saliency integration approach taking into account the impact of distortion on the saliency deployment was proposed. We also devised an algorithm which adaptively incorporate saliency in image quality metrics based on saliency dispersion. Moreover, we further investigated the plausibility of measuring image quality based on the deviation of saliency induced by distortion. An image quality metric based on measuring saliency deviation was devised. This thesis demonstrates that the added value of saliency in image quality metrics can be optimised by taking into account the interactions between saliency and visual distortion. This thesis also demonstrates that the deviation of fixation deployment due to distortion can be used as a proxy for the prediction of image quality