Image captioning and visual question answering with external knowledge

The fields of computer vision and natural language processing have made significant advances in visual question answering (VQA) and image captioning. However, a limitation of models in use today is they typically perform poorly when the task requires common sense or external knowledge. Motivated by this observation, this work offers an exploration of the benefits of multi-source external knowledge for these two tasks. Three kinds of external knowledge are evaluated: knowledge base, reverse image search, and image search by text. This work demonstrates the advantage of these external knowledge sources via experiments on two image captioning datasets (COCO-Captions and VizWiz-Captions) and three visual question answering datasets (VQAv2, VizWiz-VQA, and OK-VQA).Informatio

Pneumonia Detection on Chest X-ray using Radiomic Features and Contrastive Learning

Chest X-ray becomes one of the most common medical diagnoses due to its noninvasiveness. The number of chest X-ray images has skyrocketed, but reading chest X-rays still have been manually performed by radiologists, which creates huge burnouts and delays. Traditionally, radiomics, as a subfield of radiology that can extract a large number of quantitative features from medical images, demonstrates its potential to facilitate medical imaging diagnosis before the deep learning era. With the rise of deep learning, the explainability of deep neural networks on chest X-ray diagnosis remains opaque. In this study, we proposed a novel framework that leverages radiomics features and contrastive learning to detect pneumonia in chest X-ray. Experiments on the RSNA Pneumonia Detection Challenge dataset show that our model achieves superior results to several state-of-the-art models (> 10% in F1-score) and increases the model's interpretability.Comment: Accepted for ISBI 202

Knowledge-Augmented Contrastive Learning for Abnormality Classification and Localization in Chest X-rays with Radiomics using a Feedback Loop

Building a highly accurate predictive model for classification and localization of abnormalities in chest X-rays usually requires a large number of manually annotated labels and pixel regions (bounding boxes) of abnormalities. However, it is expensive to acquire such annotations, especially the bounding boxes. Recently, contrastive learning has shown strong promise in leveraging unlabeled natural images to produce highly generalizable and discriminative features. However, extending its power to the medical image domain is under-explored and highly non-trivial, since medical images are much less amendable to data augmentations. In contrast, their prior knowledge, as well as radiomic features, is often crucial. To bridge this gap, we propose an end-to-end semi-supervised knowledge-augmented contrastive learning framework, that simultaneously performs disease classification and localization tasks. The key knob of our framework is a unique positive sampling approach tailored for the medical images, by seamlessly integrating radiomic features as a knowledge augmentation. Specifically, we first apply an image encoder to classify the chest X-rays and to generate the image features. We next leverage Grad-CAM to highlight the crucial (abnormal) regions for chest X-rays (even when unannotated), from which we extract radiomic features. The radiomic features are then passed through another dedicated encoder to act as the positive sample for the image features generated from the same chest X-ray. In this way, our framework constitutes a feedback loop for image and radiomic modality features to mutually reinforce each other. Their contrasting yields knowledge-augmented representations that are both robust and interpretable. Extensive experiments on the NIH Chest X-ray dataset demonstrate that our approach outperforms existing baselines in both classification and localization tasks.Comment: Accepted by WACV 202

Setting behavior, apatite-forming ability, mechanical strength of polymethylmethacrylate bone cement through bioactivity modification of phosphate functional groups combined with Ca2+ ions

Bioactivity modification helps polymethylmethacrylate (PMMA) bone cement to reinforce its interfacial adhesion to bone tissues through the chemical bonding of apatite. Since Si-OH groups combined with Ca2+ ions have succeeded in inducing apatite formation, more combinations of functional groups and active ions are being explored. In this study, Bis[2-(methacryloyloxy)ethyl] phosphate (B2meP) containing phosphate (=PO4H) groups and Ca(CH3COO)2 supplying Ca2+ ion were adopted to investigate the feasibility of equipping PMMA bone cement with apatite-forming ability in vitro, more effects under designed contents on setting behavior, injectability, contact angle, cytotoxicity and mechanical strength were also investigated. Results showed B2meP copolymerized with MMA and became one section of PMMA chains, surface = PO4H groups and released Ca2+ ions pushed spherical apatite individuals nucleating and agglomerating into layer horizontally, Increasing B2meP content lowered the contact angle and the peak temperature, enhanced the cell viability of MC3T3-E1, but prolonged apatite forming period. Injectability rate performed a similar trend to setting time. Lower adding content and deposited apatite layer contributed to reduce the strength loss in soaking. Taking biological performance and other properties into balance, cement added with B2meP of 10 wt% in MMA and Ca(CH3COO)2 of 20 wt% in PMMA performed better

Atmosphere-Mediated Superhydrophobicity of Rationally Designed Micro/Nanostructured Surfaces

Superhydrophobicity has received significant attention over the past three decades owing to its significant potential in self-cleaning, anti-icing and drag reduction surfaces, energy-harvesting devices, antibacterial coatings, and enhanced heat transfer applications. Superhydrophobicity can be obtained via the roughening of an intrinsically hydrophobic surface, the creation of a re-entrant geometry, or by the roughening of a hydrophilic surface followed by a conformal coating of a hydrophobic material. Intrinsically hydrophobic surfaces have poor thermophysical properties, such as thermal conductivity, and thus are not suitable for heat transfer applications. Re-entrant geometries, although versatile in applications where droplets are deposited, break down during spatially random nucleation and flood the surface. Chemical functionalization of rough metallic substrates, although promising, is not utilized because of the poor durability of conformal hydrophobic coatings. Here we develop a radically different approach to achieve stable superhydrophobicity. By utilizing laser processing and thermal oxidation of copper (Cu) to create a high surface energy hierarchical copper oxide (CuO), followed by repeatable and passive atmospheric adsorption of hydrophobic volatile organic compounds (VOCs), we show that stable superhydrophobicity with apparent advancing contact angles ≈160° and contact angle hysteresis as low as ≈20° can be achieved. We exploit the structure length scale and structure geometry-dependent VOC adsorption dynamics to rationally design CuO nanowires with enhanced superhydrophobicity. To gain an understanding of the VOC adsorption physics, we utilized X-ray photoelectron and ion mass spectroscopy to identify the chemical species deposited on our surfaces in two distinct locations: Urbana, IL, United States and Beijing, China. To test the stability of the atmosphere-mediated superhydrophobic surfaces during heterogeneous nucleation, we used high-speed optical microscopy to demonstrate the occurrence of dropwise condensation and stable coalescence-induced droplet jumping. Our work not only provides rational design guidelines for developing passively durable superhydrophobic surfaces with excellent flooding-resistance and self-healing capability but also sheds light on the key role played by the atmosphere in governing wetting

Programmable Ring Oscillator PUF Based on Switch Matrix

Configurable ring oscillator (CRO) physical unclonable functions (PUFs) which can improve the uniqueness and reliability of conventional RO PUFs have been widely studied. Especially, the multiplier, XOR gate and tristate inverter based CRO PUFs can improve the uniqueness and reliability. However the efficiency is remain at the same level when compared with the conventional RO PUFs. In this paper, a programmable RO PUF (PRO PUF), which can be programmed to change the structure of a typical RO PUF, is proposed. The proposed PRO PUF design is implemented based on the switch matrix of an FPGA and can be programmed as a chained RO PUF or a random looped RO PUF. The proposed PRO PUF is implemented on Xilinx Spartan 6 FPGAs. Experimental results demonstrate that the proposed PRO PUF design has good uniqueness and reliability metrics as well as a high hardware efficiency