Beyond saliency: understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation

Despite the tremendous achievements of deep convolutional neural networks (CNNs) in many computer vision tasks, understanding how they actually work remains a significant challenge. In this paper, we propose a novel two-step understanding method, namely Salient Relevance (SR) map, which aims to shed light on how deep CNNs recognize images and learn features from areas, referred to as attention areas, therein. Our proposed method starts out with a layer-wise relevance propagation (LRP) step which estimates a pixel-wise relevance map over the input image. Following, we construct a context-aware saliency map, SR map, from the LRP-generated map which predicts areas close to the foci of attention instead of isolated pixels that LRP reveals. In human visual system, information of regions is more important than of pixels in recognition. Consequently, our proposed approach closely simulates human recognition. Experimental results using the ILSVRC2012 validation dataset in conjunction with two well-established deep CNN models, AlexNet and VGG-16, clearly demonstrate that our proposed approach concisely identifies not only key pixels but also attention areas that contribute to the underlying neural network's comprehension of the given images. As such, our proposed SR map constitutes a convenient visual interface which unveils the visual attention of the network and reveals which type of objects the model has learned to recognize after training. The source code is available at https://github.com/Hey1Li/Salient-Relevance-Propagation.Comment: 35 pages, 15 figure

MILI: Multi-person inference from a low-resolution image

Existing multi-person reconstruction methods require the human bodies in the input image to occupy a considerable portion of the picture. However, low-resolution human objects are ubiquitous due to trade-off between the field of view and target distance given a limited camera resolution. In this paper, we propose an end-to-end multi-task framework for multi-person inference from a low-resolution image (MILI). To perceive more information from a low-resolution image, we use pair-wise images at high resolution and low resolution for training, and design a restoration network with a simple loss for better feature extraction from the low-resolution image. To address the occlusion problem in multi-person scenes, we propose an occlusion-aware mask prediction network to estimate the mask of each person during 3D mesh regression. Experimental results on both small-scale scenes and large-scale scenes demonstrate that our method outperforms the state-of-the-art methods both quantitatively and qualitatively. The code is available at http://cic.tju.edu.cn/faculty/likun/projects/MILI

The triglyceride to high-density lipoprotein ratio identifies children who may be at risk of developing cardiometabolic disease

Aim: It is important to develop simple, reliable methods to identify high-risk individuals who may benefit from intervention. This study investigated the association between the triglyceride to high-density lipoprotein cholesterol (TG/HDL) ratio and cardiometabolic risk, cardiorespiratory fitness and physical activity in children. Methods: Anthropometric, biochemical parameters, cardiorespiratory fitness and accelerometry determined physical activity were assessed in 155 children (80 girls) from 10 to 14 years of age from Bedfordshire, UK. Participants were grouped into high and low TG/HDL ratio groups, according to published thresholds. MANCOVA and logistic regression were used in the analysis. Results: Cardiometabolic risk factor levels were significantly higher in participants with a high TG/HDL ratio (p < 0.05). The odds of having high waist circumference (OR = 13.99; 95% CI 2.93, 69.25), elevated systolic blood pressure (5.27; 1.39, 20.01), high non-HDL cholesterol (19.47; 4.42, 85.81) and ≥2 cardiometabolic risk factors (15.32; 3.10, 75.79) were higher in participants with a high TG/HDL ratio. The TG/HDL ratio values were significantly lower in those with high cardiorespiratory fitness (p = 0.01), but there was no association with physical activity. Conclusion: These findings support the use of the TG/HDL ratio to identify children with cardiometabolic risk factors who may be at risk of developing cardiometabolic disease

Geometry-guided dense perspective network for speech-driven facial animation

Realistic speech-driven 3D facial animation is a challenging problem due to the complex relationship between speech and face. In this paper, we propose a deep architecture, called Geometry-guided Dense Perspective Network (GDPnet), to achieve speaker-independent realistic 3D facial animation. The encoder is designed with dense connections to strengthen feature propagation and encourage the re-use of audio features, and the decoder is integrated with an attention mechanism to adaptively recalibrate point-wise feature responses by explicitly modeling interdependencies between different neuron units. We also introduce a non-linear face reconstruction representation as a guidance of latent space to obtain more accurate deformation, which helps solve the geometry-related deformation and is good for generalization across subjects. Huber and HSIC (Hilbert-Schmidt Independence Criterion) constraints are adopted to promote the robustness of our model and to better exploit the non-linear and high-order correlations. Experimental results on the public dataset and real scanned dataset validate the superiority of our proposed GDPnet compared with state-of-the-art model. We will make the code available for research purposes

Gut microbiome-based noninvasive diagnostic model to predict acute coronary syndromes

BackgroundPrevious studies have shown that alterations in the gut microbiota are closely associated with Acute Coronary Syndrome (ACS) development. However, the value of gut microbiota for early diagnosis of ACS remains understudied.MethodsWe recruited 66 volunteers, including 29 patients with a first diagnosis of ACS and 37 healthy volunteers during the same period, collected their fecal samples, and sequenced the V4 region of the 16S rRNA gene. Functional prediction of the microbiota was performed using PICRUSt2. Subsequently, we constructed a nomogram and corresponding webpage based on microbial markers to assist in the diagnosis of ACS. The diagnostic performance and usefulness of the model were analyzed using boostrap internal validation, calibration curves, and decision curve analysis (DCA).ResultsCompared to that of healthy controls, the diversity and composition of microbial community of patients with ACS was markedly abnormal. Potentially pathogenic genera such as Streptococcus and Acinetobacter were significantly increased in the ACS group, whereas certain SCFA-producing genera such as Blautia and Agathobacter were depleted. In addition, in the correlation analysis with clinical indicators, the microbiota was observed to be associated with the level of inflammation and severity of coronary atherosclerosis. Finally, a diagnostic model for ACS based on gut microbiota and clinical variables was developed with an area under the receiver operating characteristic (ROC) curve (AUC) of 0.963 (95% CI: 0.925–1) and an AUC value of 0.948 (95% CI: 0.549–0.641) for bootstrap internal validation. The calibration curves of the model show good consistency between the actual and predicted probabilities. The DCA showed that the model had a high net clinical benefit for clinical applications.ConclusionOur study is the first to characterize the composition and function of the gut microbiota in patients with ACS and healthy populations in Southwest China and demonstrates the potential effect of the microbiota as a non-invasive marker for the early diagnosis of ACS

Revenue Management with Practical Restrictions and the New Challenge

Revenue management is the control of selling a limited quantity of a resource to a potential set of customers. Motivated by practical problems in three different sectors, we explore the strategies that managers employ to maximize revenue while navigating various capacity and pricing constraints.In the first essay, focusing on the context of a single live event, we define the managers' decision problem as the Event Ticket Pricing (ETP) problem and formulate it as a constrained non-linear optimization problem. The ETP problem we constructed incorporates constraints based on capacity considerations and pricing restrictions. Our formulation for the ETP problem is robust and can embed many demand functions. Importantly, our analysis of the optimal solution to the ETP problem reveals vital insights for live entertainment promotion planners facing insufficient supply and customer emotional concerns.Our second essay delves into the revenue management issue in the hotel industry, particularly investigating the effect of loyalty programs on hotel profits under various scenarios in the context of price and capacity restrictions. Given tourists' price and quality sensitivities, we assume the competition between two vertically differentiated hotels over prices and capacities and against the loyalty program. Through formulating unilateral/bilateral award sale problems, we probe the circumstances under which hotels can benefit from reward sales-factors that heavily depend on quality differentiation, the size of the loyal customer group, and the reimbursement rate.In the third essay, we shift our focus to the rapidly evolving world of live-streaming events-a highly promising promotional instrument and sales channel in the e-commerce landscape. We identifythat certain prevalent issues and challenges have seldom been approached from a modeling perspective. By constructing game theoretic models based on key characteristics of the live-streaming industry, we characterize equilibrium and examine how the live-streaming channel influences manufacturers' pricing decisions