f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning

When labeled training data is scarce, a promising data augmentation approach is to generate visual features of unknown classes using their attributes. To learn the class conditional distribution of CNN features, these models rely on pairs of image features and class attributes. Hence, they can not make use of the abundance of unlabeled data samples. In this paper, we tackle any-shot learning problems i.e. zero-shot and few-shot, in a unified feature generating framework that operates in both inductive and transductive learning settings. We develop a conditional generative model that combines the strength of VAE and GANs and in addition, via an unconditional discriminator, learns the marginal feature distribution of unlabeled images. We empirically show that our model learns highly discriminative CNN features for five datasets, i.e. CUB, SUN, AWA and ImageNet, and establish a new state-of-the-art in any-shot learning, i.e. inductive and transductive (generalized) zero- and few-shot learning settings. We also demonstrate that our learned features are interpretable: we visualize them by inverting them back to the pixel space and we explain them by generating textual arguments of why they are associated with a certain label.Comment: Accepted at CVPR 201

Gaze Embeddings for Zero-Shot Image Classification

Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification

Generating Counterfactual Explanations with Natural Language

Natural language explanations of deep neural network decisions provide an intuitive way for a AI agent to articulate a reasoning process. Current textual explanations learn to discuss class discriminative features in an image. However, it is also helpful to understand which attributes might change a classification decision if present in an image (e.g., "This is not a Scarlet Tanager because it does not have black wings.") We call such textual explanations counterfactual explanations, and propose an intuitive method to generate counterfactual explanations by inspecting which evidence in an input is missing, but might contribute to a different classification decision if present in the image. To demonstrate our method we consider a fine-grained image classification task in which we take as input an image and a counterfactual class and output text which explains why the image does not belong to a counterfactual class. We then analyze our generated counterfactual explanations both qualitatively and quantitatively using proposed automatic metrics.Comment: presented at 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018), Stockholm, Swede

Manipulating Attributes of Natural Scenes via Hallucination

In this study, we explore building a two-stage framework for enabling users to directly manipulate high-level attributes of a natural scene. The key to our approach is a deep generative network which can hallucinate images of a scene as if they were taken at a different season (e.g. during winter), weather condition (e.g. in a cloudy day) or time of the day (e.g. at sunset). Once the scene is hallucinated with the given attributes, the corresponding look is then transferred to the input image while preserving the semantic details intact, giving a photo-realistic manipulation result. As the proposed framework hallucinates what the scene will look like, it does not require any reference style image as commonly utilized in most of the appearance or style transfer approaches. Moreover, it allows to simultaneously manipulate a given scene according to a diverse set of transient attributes within a single model, eliminating the need of training multiple networks per each translation task. Our comprehensive set of qualitative and quantitative results demonstrate the effectiveness of our approach against the competing methods.Comment: Accepted for publication in ACM Transactions on Graphic

Predictors of mortality and survival in type 1 diabetes: a retrospective cohort study of type 1 diabetes mellitus (T1D) in the Wirral Peninsula

Background: The prevalence of T1D is rising, despite improvements in the management of this condition. It presents a risk of premature and excess mortality, which impacts survival and life expectancy. Aim: The study aim was to assess mortality, identify predicting risk factors for mortality and survival in T1D in the Wirral. A systematic review was done to establish present current evidence of all-cause and cause-specific mortality amongst T1D patients. Methods: A retrospective cohort study design, 1786 patients diagnosed with T1D extracted from the Wirral Diabetes Register (WDR). The follow-up period was between 1st of January, 2000 to 31st December, 2012. The primary outcome measured was all-cause mortality. Results: 1458 participants with T1D meet the inclusion criteria, after a follow-up period of 12 years, 113(7.75%) deaths were recorded. While the incidence rate was steady over the study period, the prevalence rate continued to increase over the study period. Significant predictors of mortality in this cohort were age of diagnosis, duration of diabetes, HbA1c,systolic blood pressure (SBP), diastolic blood pressure (DBP), and triglyceride levels. The predicting risk gender, age at diagnosis, duration of T1D, BMI, serum creatinine levels, SBP, total cholesterol, LDL, HDL, TC\HDL, and LDL\HDL showed a linear increase in mortality risk. IMD and DBP followed a U-shaped relationship with relative and absolute mortality, while HbA1c levels reveal a sinusoidal pattern with the highest risk of mortality at the levels ≤ 5.9% (41 mmol/mol). The risk of mortality for the predicting risk factors for this study ranged between 5% and 9%. Maximal risk of mortality of 9% was recorded in the predicting risks of smoking, BMI, SBP, and DBP. The risk of mortality of 8% was recorded for IMD, serum creatinine, total cholesterol, TG, LDL\HDL ratio, and TSH. The risk of mortality of 7% was recorded for the predicting variables of HbA1c, HDL, LDL, and TC\HDL ratio. The minimum risk of mortality of 5% was recorded for the predictor variable of the duration of diabetes. The significant predictors of mortality were the age at diagnosis, duration of diagnosis, systolic and diastolic blood pressure, HbA1c. The burden of mortality rest disproportionately with females who had higher relative risk of mortality of 4 times that of their male counterparts, however, the burden of premature mortality as recorded by the years of potential life lost was slightly higher in males (1797[53.6%]) as compare to females (1553[46.4%]). Of the 113 deaths recorded for the cohort that indicated a proportion of 7.75% of the total T1D patients, records for only 37 participants were retrieved. The principal cause of death in this cohort was malignancy-related 8 deaths (21.6%), this was followed by cardiovascular disease and sepsis, each having 6 deaths (16.2%) respectively. Cerebrovascular disease accounted for 5 deaths (13.5%). Death from diabetes complications (hypoglycaemia) was recorded in 1 patient (2.7%). There were marked reductions in life expectancy for this cohort. Life expectancy at 40 years for females was to an average age mortality of 66.2 years as compared to males 78.3 years. There has been improved survival for T1D in this cohort, 77.185 years [95% CI: 75.191 – 79.179] in males and 76.011 years [95% CI: 73.169 – 78.000] in females. The systematic review highlighted increased mortality in those with T1D as compared to the general population, females showed greater risk of vascular complications as compared to the males with T1D. 35 studies were included. Results showed all-cause mortality RR 3.73 (95% CI 3.19, 4.36) compared to general population, with gender specific mortality RR 1.17 (95% CI 1.06, 1.29). For cause specific mortality risk (overall and gender specific): cardiovascular v disease RR 3.48 (95% CI 3.14, 3.86) and RR 1.41 (95% CI 0.92, 2.17); renal disease RR 1.06 (95% CI 0.89, 1.26) and RR 0.63 (95% CI 0.38, 1.04); neoplasms RR 1.03 (95% CI 0.92, 1.16) and RR 1.18 (95% CI 0.75, 1.86); cerebrovascular disease according to gender RR 0.99 (95% CI 0.66, 1.48), and accidents and suicides according to gender RR 2.30 (95% CI 1.31, 4.06). Conclusion In conclusion, the study highlighted significant mortality risk in females as compared to their male counterparts; there has been progress in the survival of patients with T1D. However, life expectancy remains reduced as compared to those without the condition. Prevalence of T1D continues to increase, and the complex interplay of the predictor variables support the need for an individualised approach to care

Evaluation of Output Embeddings for Fine-Grained Image Classification

Image classification has advanced significantly in recent years with the availability of large-scale image sets. However, fine-grained classification remains a major challenge due to the annotation cost of large numbers of fine-grained categories. This project shows that compelling classification performance can be achieved on such categories even without labeled training data. Given image and class embeddings, we learn a compatibility function such that matching embeddings are assigned a higher score than mismatching ones; zero-shot classification of an image proceeds by finding the label yielding the highest joint compatibility score. We use state-of-the-art image features and focus on different supervised attributes and unsupervised output embeddings either derived from hierarchies or learned from unlabeled text corpora. We establish a substantially improved state-of-the-art on the Animals with Attributes and Caltech-UCSD Birds datasets. Most encouragingly, we demonstrate that purely unsupervised output embeddings (learned from Wikipedia and improved with fine-grained text) achieve compelling results, even outperforming the previous supervised state-of-the-art. By combining different output embeddings, we further improve results.Comment: @inproceedings {ARWLS15, title = {Evaluation of Output Embeddings for Fine-Grained Image Classification}, booktitle = {IEEE Computer Vision and Pattern Recognition}, year = {2015}, author = {Zeynep Akata and Scott Reed and Daniel Walter and Honglak Lee and Bernt Schiele}