Analyzing First-Person Stories Based on Socializing, Eating and Sedentary Patterns

First-person stories can be analyzed by means of egocentric pictures acquired throughout the whole active day with wearable cameras. This manuscript presents an egocentric dataset with more than 45,000 pictures from four people in different environments such as working or studying. All the images were manually labeled to identify three patterns of interest regarding people's lifestyle: socializing, eating and sedentary. Additionally, two different approaches are proposed to classify egocentric images into one of the 12 target categories defined to characterize these three patterns. The approaches are based on machine learning and deep learning techniques, including traditional classifiers and state-of-art convolutional neural networks. The experimental results obtained when applying these methods to the egocentric dataset demonstrated their adequacy for the problem at hand.Comment: Accepted at First International Workshop on Social Signal Processing and Beyond, 19th International Conference on Image Analysis and Processing (ICIAP), September 201

Feature Selection for Big Visual Data: Overview and Challenges

International Conference Image Analysis and Recognition (ICIAR 2018, Póvoa de Varzim, Portugal

Playing to distraction: towards a robust training of CNN classifiers through visual explanation techniques

The field of deep learning is evolving in different directions, with still the need for more efficient training strategies. In this work, we present a novel and robust training scheme that integrates visual explanation techniques in the learning process. Unlike the attention mechanisms that focus on the relevant parts of images, we aim to improve the robustness of the model by making it pay attention to other regions as well. Broadly speaking, the idea is to distract the classifier in the learning process to force it to focus not only on relevant regions but also on those that, a priori, are not so informative for the discrimination of the class. We tested the proposed approach by embedding it into the learning process of a convolutional neural network for the analysis and classification of two well-known datasets, namely Stanford cars and FGVC-Aircraft. Furthermore, we evaluated our model on a real-case scenario for the classification of egocentric images, allowing us to obtain relevant information about peoples' lifestyles. In particular, we work on the challenging EgoFoodPlaces dataset, achieving state-of-the-art results with a lower level of complexity. The obtained results indicate the suitability of our proposed training scheme for image classification, improving the robustness of the final model.Comment: 20 pages,3 figures, 4 table

Advancing the diagnosis of dry eye syndrome : development of automated assessments of tear film lipid layer patterns

[Resumen] El síndrome de ojo seco es una enfermedad sintomática que afecta a un amplio rango de la población, y tiene un impacto negativo en sus actividades diarias. Su diagnóstico es una tarea difícil debido a su etiología multifactorial, y por eso existen varias pruebas clínicas. Una de esas pruebas es la evaluación de los patrones interferenciales de la capa lipídica de la película lagrimal. Guillon dise˜nó un instrumento denominado Tearscope Plus para evaluar el grosor de la película lagrimal de forma rápida, y también definió una escala de clasificación compuesta de cinco categorías. La clasificación en uno de esos cinco patrones es una tarea clínica dificil, especialmente con las capas lipídicas más finas que carecen de características de color y/o morfológicas. Además, la interpretación subjetiva de los expertos mediante una revisión visual puede afectar a la clasificación, pudiendo producirse un alto grado de inter- e intra- variabilidad entre observadores. El desarrollo de un método sistemático y objetivo para análisis y clasificación es altamente deseable, permitiendo un diagnóstico homogéneo y liberando a los expertos de esta tediosa tarea. La propuesta de esta investigación es el diseño de un sistema automático para evaluar los patrones de la capa lipídica de la película lagrimal mediante la interpretación de las imágenes obtenidas con el Tearscope Plus. Por una parte, se presenta una metodología global para evaluar la capa lipídica de la película lagrimal mediante la clasificación automática de estas imágenes en una de las categorías de Guillon. El proceso se lleva a cabo mediante el uso de modelos de textura y color, y algoritmos de aprendizaje máquina. A continuación, esta metodología global se optimiza mediante la reducción de su complejidad computacional. Se utilizan técnicas de reducción de la dimensión para disminuir los requisitos de memoria/tiempo sin una degradación en su rendimiento. Por otra parte, se presenta una metodología local para crear mapas de la película lagrimal, que representan la distribución local de los patrones de la capa lipídica sobre la película lagrimal. Las diferentes evaluaciones automáticas que se proponen ahorran tiempo a los expertos, y proporcionan resultados imparciales que no están afectados por factores subjetivos.[Resumo] O síndrome de ollo seco é unha enfermidade sintomática que afecta a un amplo rango da poboación, e ten un impacto negativo nas súas actividades diarias. O seu diagnóstico é unha tarefa difícil debido á súa etioloxía multifactorial, e por iso existen varias probas clínicas. Unha desas probas é a avaliación dos patróns interferenciais da capa lipídica da película lagrimal. Guillon dese˜nou un instrumento denominado Tearscope Plus para avaliar o grosor da película lagrimal de forma rápida, e tamén definiu unha escala de clasificación composta de cinco categorías. A clasificación nun deses cinco patróns é unha tarefa clínica difícil, especialmente coas capas lipídicas máis finas que carecen de características de cor e/ou morfolóxicas. Ademais, a interpretación subxectiva dos expertos mediante una revisión visual pode afectar á clasificación, podendo producirse un alto grao de inter- e intra- variabilidade entre observadores. O desenvolvemento dun método sistemático e obxectivo para análise e clasificación é altamente desexable, permitindo un diagnóstico homoxéneo e liberando aos expertos desta tediosa tarefa. A proposta desta investigación é o deseño dun sistema automático para avaliar os patróns da capa lipídica da película lagrimal mediante a interpretación das imaxes obtidas co Tearscope Plus. Por unha parte, preséntase unha metodoloxía global para avaliar a capa lipídica da película lagrimal mediante a clasificación automática destas imaxes nunha das categorías de Guillon. O proceso é levado a cabo mediante o uso de modelos de textura e cor, e algoritmos de aprendizaxe máquina. A continuación, esta metodoloxía global é optimizada mediante a redución da súa complexidade computacional. Utilízanse técnicas de redución da dimensión para diminuír os requisitos de memoria/tempo sen unha degradación no seu rendemento. Por outra parte, preséntase unha metodoloxía local para crear mapas da película lagrimal, que representan a distribución local dos patróns da capa lipídica sobre a película lagrimal. As diferentes avaliacións automáticas que se propoñen aforran tempo aos expertos, e proporcionan resultados imparciais que non están afectados por factores subxectivos.[Abstract] Dry eye syndrome is a symptomatic disease which affects a wide range of population, and has a negative impact on their daily activities. Its diagnosis is a difficult task due to its multifactorial etiology, and so there exist several clinical tests. One of these tests is the evaluation of the interference patterns of the tear film lipid layer. Guillon designed an instrument known as Tearscope Plus which allows clinicians to rapidly assess the lipid layer thickness, and also defined a grading scale composed of five categories. The classification into these five patterns is a difficult clinical task, especially with thinner lipid layers which lack color and/or morphological features. Furthermore, the subjective interpretation of the experts via visual inspection may affect the classification, and so a high degree of inter- and also intra- observer variability can be produced. The development of a systematic, objective computerized method for analysis and classification is thus highly desirable, allowing for homogeneous diagnosis and relieving the experts from this tedious task. The proposal of this research is the design of an automatic system to assess the tear film lipid layer patterns through the interpretation of the images acquired with the Tearscope Plus. On the one hand, a global methodology is presented to assess the tear film lipid layer by automatically classifying these images into the Guillon categories. The process is carried out using texture and color models, and machine learning algorithms. Then, this global methodology is optimized through the reduction of its computational complexity. Dimensionality reduction techniques are used in order to diminish the memory/time requirements with no degradation in performance. On the other hand, a local methodology is also presented to create tear film maps, which represent the local distribution of the lipid layer patterns over the tear film. The different automated assessments proposed save time for experts, and provide unbiased results which are not affected by subjective factors

Feature Selection in Big Image Datasets

[Abstract] In computer vision, current feature extraction techniques generate high dimensional data. Both convolutional neural networks and traditional approaches like keypoint detectors are used as extractors of high-level features. However, the resulting datasets have grown in the number of features, leading into long training times due to the curse of dimensionality. In this research, some feature selection methods were applied to these image features through big data technologies. Additionally, we analyzed how image resolutions may affect to extracted features and the impact of applying a selection of the most relevant features. Experimental results show that making an important reduction of the extracted features provides classification results similar to those obtained with the full set of features and, in some cases, outperforms the results achieved using broad feature vectors.This research has been financially supported in part by European Union FEDER funds, by the Spanish Ministerio de Economía y Competitividad (research project PID2019-109238GB), by the Consellería de Industria of the Xunta de Galicia (research project GRC2014/035), and by the Principado de Asturias Regional Government (research project IDI-2018-000176). CITIC as a Research Centre of the Galician University System is financed by the Consellería de Educación, Universidades e Formación Profesional (Xunta de Galicia) through the ERDF (80%), Operational Programme ERDF Galicia 2014–2020, and the remaining 20% by the Secretaria Xeral de Universidades (ref. ED431G 2019/01).Xunta de Galicia; GRC2014/035Gobierno del Principado de Asturias; IDI-2018-000176Xunta de Galicia; ED431G 2019/0

Parallel definition of tear film maps on distributed-memory clusters for the support of dry eye diagnosis

[Abstract] Background and objectives The analysis of the interference patterns on the tear film lipid layer is a useful clinical test to diagnose dry eye syndrome. This task can be automated with a high degree of accuracy by means of the use of tear film maps. However, the time required by the existing applications to generate them prevents a wider acceptance of this method by medical experts. Multithreading has been previously successfully employed by the authors to accelerate the tear film map definition on multicore single-node machines. In this work, we propose a hybrid message-passing and multithreading parallel approach that further accelerates the generation of tear film maps by exploiting the computational capabilities of distributed-memory systems such as multicore clusters and supercomputers. Methods The algorithm for drawing tear film maps is parallelized using Message Passing Interface (MPI) for inter-node communications and the multithreading support available in the C++11 standard for intra-node parallelization. The original algorithm is modified to reduce the communications and increase the scalability. Results The hybrid method has been tested on 32 nodes of an Intel cluster (with two 12-core Haswell 2680v3 processors per node) using 50 representative images. Results show that maximum runtime is reduced from almost two minutes using the previous only-multithreaded approach to less than ten seconds using the hybrid method. Conclusions The hybrid MPI/multithreaded implementation can be used by medical experts to obtain tear film maps in only a few seconds, which will significantly accelerate and facilitate the diagnosis of the dry eye syndrome.Ministerio de Economía y Competitividad; TIN2013-42148-PPortugal. Fundação para a Ciência e a Tecnologia; POCI-01-0145-FEDER-006961Portugal. Fundação para a Ciência e a Tecnologia; UID/EEA/50014/2013Portugal. Fundação para a Ciência e a Tecnologia; SFRH/BPD/111177/2015

On the Effectiveness of Convolutional Autoencoders on Image-Based Personalized Recommender Systems

[Abstract] Over the years, the success of recommender systems has become remarkable. Due to the massive arrival of options that a consumer can have at his/her reach, a collaborative environment was generated, where users from all over the world seek and share their opinions based on all types of products. Specifically, millions of images tagged with users’ tastes are available on the web. Therefore, the application of deep learning techniques to solve these types of tasks has become a key issue, and there is a growing interest in the use of images to solve them, particularly through feature extraction. This work explores the potential of using only images as sources of information for modeling users’ tastes and proposes a method to provide gastronomic recommendations based on them. To achieve this, we focus on the pre-processing and encoding of the images, proposing the use of a pre-trained convolutional autoencoder as feature extractor. We compare our method with the standard approach of using convolutional neural networks and study the effect of applying transfer learning, reflecting how it is better to use only the specific knowledge of the target domain in this case, even if fewer examples are available.This research has been financially supported in part by European Union FEDER funds, by the Spanish Ministerio de Economía y Competitividad (research project PID2019-109238GB), by the Consellería de Industria of the Xunta de Galicia (research project GRC2014/035), and by the Principado de Asturias Regional Government (research project IDI-2018-000176). CITIC as a Research Center of the Galician University System is financed by the Consellería de Educación, Universidades e Formación Profesional (Xunta de Galicia) through the ERDF (80%), Operational Programme ERDF Galicia 2014–2020 and the remaining 20% by the Secretaria Xeral de Universidades (ref. ED431G 2019/01).Xunta de Galicia; GRC2014/035Gobierno del Principado de Asturias; IDI-2018-000176Xunta de Galicia; ED431G 2019/0

An end-to-end framework for intima media measurement and atherosclerotic plaque detection in the carotid artery

Background and objectives: The detection and delineation of atherosclerotic plaque are usually manually performed by medical experts on the carotid artery. Evidence suggests that this manual process is subject to errors and has a large variability between experts, equipment, and datasets. This paper proposes a robust end-to-end framework for automatic atherosclerotic plaque detection. Methods: The proposed framework is composed of: (1) a semantic segmentation model based on U-Net, with EfficientNet as the backbone, that obtains a segmentation mask with the carotid intima-media region; and (2) a convolutional neural network designed using Bayesian optimization that simultaneously performs a regression to get the average and maximum carotid intima media thickness, and a classification to determine the presence of plaque. Results: Our approach improves the state-of-the-art in both co and bulb territories in the REGICOR database, with more than 8000 images, while providing predictions in real-time. The correlation coefficient was 0.89 in the common carotid artery and 0.74 for bulb region, and the F1 score for atherosclerotic plaque detecting was 0.60 and 0.59, respectively. The experimentation carried out includes a comparison with other fully automatic methods for carotid intima media thickness estimation found in the literature. Additionally, we present an extensive experimental study to evaluate the robustness of our proposal, as well as its suitability and efficiency compared to different versions of the framework. Conclusions: The proposed end-to-end framework significantly improves the automatic characterization of atherosclerotic plaque. The generation of the segmented mask can be helpful for practitioners since it allows them to evaluate and interpret the model's results by visual inspection. Furthermore, the proposed framework overcomes the limitations of previous research based on ad-hoc post-processing, which could lead to overestimations in the case of oblique forms of the carotid artery

Correlation between tear osmolarity and tear meniscus

Purpose. To examine the relationship between tear meniscus height (TMH) and subjective meniscus grading (subjective TM) with tear osmolarity. Methods. Tear osmolarity measurements (using TearLab) and digital images of the tear meniscus were obtained in 177 consecutive patients undergoing an eye examination at our optometry clinic (Universidad de Santiago de Compostela, Spain) who fulfilled the study's inclusion criteria. Participants were also administered the McMonnies and Ocular Surface Disease Index (OSDI) questionnaires for the detection of dry eye disease. The lower tear meniscus was videotaped by a digital camera attached to a slit lamp in its central portion without fluorescein instillation. After the study, a masked observer extracted an image from each video, and measured the TMH using open source software (NIH ImageJ). Subsequently, the masked observer subjectively graded the appearance of each meniscus. For statistical analysis, subjects were stratified by age and by dry eye symptoms as indicated by their scores in the two questionnaires. Results. In the whole study population, a significant relationship was observed between osmolarity and TMH (-0.41, p<0.001) and osmolarity and subjective TM (r = 0.35, p<0.001). A cluster analysis revealed similar correlations when subjects were stratified by age or dry eye symptoms, these correlations being more pronounced in older and more symptomatic individuals. Objective TMH measurements and subjective meniscus quality were also correlated (r=-0.75, p<0.001). Conclusions. Osmolarity and both objective TMH measurements and subjective interpretation of the meniscus showed high correlation, especially in older symptomatic subjects.Estudio patrocinado por el Ministerio de Educación y Ciencia y el Instituto de Salud Carlos III a través del proyecto PI10/01098.S

Polyvascular Subclinical Atherosclerosis: Correlation Between Ankle Brachial Index and Carotid Atherosclerosis in a Population-Based Sample

We assessed the correlation between the biomarkers of lower limb atherosclerosis (eg, ankle-brachial index [ABI]) and of carotid atherosclerosis (eg, common carotid intima-media thickness (IMT) and presence of atherosclerotic plaque) in a population-based cohort from Girona (Northwest Spain) recruited in 2010. Ankle-brachial index and carotid ultrasound were performed in all participants. Generalized additive multivariable models were used to adjust a regression model of common carotid IMT on ABI. Logistic regression multivariable models were adjusted to assess the probability of carotid plaque in individuals with peripheral artery disease. We included 3307 individuals (54.2% women), mean age 60 years (standard deviation 11). Two patterns of association were observed between subclinical biomarkers of atherosclerosis at the lower limb and carotid artery. Ankle-brachial index and common carotid IMT showed a linear trend in men [beta coefficient (95% confidence interval) =-.068 (-.123; -.012); P = .016]. Women with peripheral artery disease presented with high risk of atherosclerotic plaque at the carotid artery [Odds ratio (95% confidence interval) = 2.61, (1.46; 4.69); P = .001]. Men showed a significant linear association between ABI levels and common carotid IMT values. Women with peripheral artery disease presented with high risk of atherosclerotic plaque at the carotid artery