TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis

Multi-modal skin lesion diagnosis (MSLD) has achieved remarkable success by modern computer-aided diagnosis technology based on deep convolutions. However, the information aggregation across modalities in MSLD remains challenging due to severity unaligned spatial resolution (dermoscopic image and clinical image) and heterogeneous data (dermoscopic image and patients' meta-data). Limited by the intrinsic local attention, most recent MSLD pipelines using pure convolutions struggle to capture representative features in shallow layers, thus the fusion across different modalities is usually done at the end of the pipelines, even at the last layer, leading to an insufficient information aggregation. To tackle the issue, we introduce a pure transformer-based method, which we refer to as ``Throughout Fusion Transformer (TFormer)", for sufficient information intergration in MSLD. Different from the existing approaches with convolutions, the proposed network leverages transformer as feature extraction backbone, bringing more representative shallow features. We then carefully design a stack of dual-branch hierarchical multi-modal transformer (HMT) blocks to fuse information across different image modalities in a stage-by-stage way. With the aggregated information of image modalities, a multi-modal transformer post-fusion (MTP) block is designed to integrate features across image and non-image data. Such a strategy that information of the image modalities is firstly fused then the heterogeneous ones enables us to better divide and conquer the two major challenges while ensuring inter-modality dynamics are effectively modeled. Experiments conducted on the public Derm7pt dataset validate the superiority of the proposed method. Our TFormer outperforms other state-of-the-art methods. Ablation experiments also suggest the effectiveness of our designs

ECL: Class-Enhancement Contrastive Learning for Long-tailed Skin Lesion Classification

Skin image datasets often suffer from imbalanced data distribution, exacerbating the difficulty of computer-aided skin disease diagnosis. Some recent works exploit supervised contrastive learning (SCL) for this long-tailed challenge. Despite achieving significant performance, these SCL-based methods focus more on head classes, yet ignoring the utilization of information in tail classes. In this paper, we propose class-Enhancement Contrastive Learning (ECL), which enriches the information of minority classes and treats different classes equally. For information enhancement, we design a hybrid-proxy model to generate class-dependent proxies and propose a cycle update strategy for parameters optimization. A balanced-hybrid-proxy loss is designed to exploit relations between samples and proxies with different classes treated equally. Taking both "imbalanced data" and "imbalanced diagnosis difficulty" into account, we further present a balanced-weighted cross-entropy loss following curriculum learning schedule. Experimental results on the classification of imbalanced skin lesion data have demonstrated the superiority and effectiveness of our method

Dense Pixel-to-Pixel Harmonization via Continuous Image Representation

High-resolution (HR) image harmonization is of great significance in real-world applications such as image synthesis and image editing. However, due to the high memory costs, existing dense pixel-to-pixel harmonization methods are mainly focusing on processing low-resolution (LR) images. Some recent works resort to combining with color-to-color transformations but are either limited to certain resolutions or heavily depend on hand-crafted image filters. In this work, we explore leveraging the implicit neural representation (INR) and propose a novel image Harmonization method based on Implicit neural Networks (HINet), which to the best of our knowledge, is the first dense pixel-to-pixel method applicable to HR images without any hand-crafted filter design. Inspired by the Retinex theory, we decouple the MLPs into two parts to respectively capture the content and environment of composite images. A Low-Resolution Image Prior (LRIP) network is designed to alleviate the Boundary Inconsistency problem, and we also propose new designs for the training and inference process. Extensive experiments have demonstrated the effectiveness of our method compared with state-of-the-art methods. Furthermore, some interesting and practical applications of the proposed method are explored. Our code is available at https://github.com/WindVChen/INR-Harmonization.Comment: Accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT

A Rotation Meanout Network with Invariance for Dermoscopy Image Classification and Retrieval

The computer-aided diagnosis (CAD) system can provide a reference basis for the clinical diagnosis of skin diseases. Convolutional neural networks (CNNs) can not only extract visual elements such as colors and shapes but also semantic features. As such they have made great improvements in many tasks of dermoscopy images. The imaging of dermoscopy has no principal orientation, indicating that there are a large number of skin lesion rotations in the datasets. However, CNNs lack rotation invariance, which is bound to affect the robustness of CNNs against rotations. To tackle this issue, we propose a rotation meanout (RM) network to extract rotation-invariant features from dermoscopy images. In RM, each set of rotated feature maps corresponds to a set of outputs of the weight-sharing convolutions and they are fused using meanout strategy to obtain the final feature maps. Through theoretical derivation, the proposed RM network is rotation-equivariant and can extract rotation-invariant features when followed by the global average pooling (GAP) operation. The extracted rotation-invariant features can better represent the original data in classification and retrieval tasks for dermoscopy images. The RM is a general operation, which does not change the network structure or increase any parameter, and can be flexibly embedded in any part of CNNs. Extensive experiments are conducted on a dermoscopy image dataset. The results show our method outperforms other anti-rotation methods and achieves great improvements in dermoscopy image classification and retrieval tasks, indicating the potential of rotation invariance in the field of dermoscopy images

CS-SELEX Generates High-Affinity ssDNA Aptamers as Molecular Probes for Hepatitis C Virus Envelope Glycoprotein E2

Currently, the development of effective diagnostic reagents as well as treatments against Hepatitis C virus (HCV) remains a high priority. In this study, we have described the development of an alive cell surface -Systematic Evolution of Ligands by Exponential Enrichment (CS-SELEX) technique and screened the functional ssDNA aptamers that specifically bound to HCV envelope surface glycoprotein E2. Through 13 rounds of selection, the CS-SELEX generated high-affinity ssDNA aptamers, and the selected ssDNA aptamer ZE2 demonstrated the highest specificity and affinity to E2-positive cells. HCV particles could be specifically captured and diagnosed using the aptamer ZE2. A good correlation was observed in HCV patients between HCV E2 antigen-aptamer assay and assays for HCV RNA quantities or HCV antibody detection. Moreover, the selected aptamers, especially ZE2, could competitively inhibit E2 protein binding to CD81, an important HCV receptor, and significantly block HCV cell culture (HCVcc) infection of human hepatocytes (Huh7.5.1) in vitro. Our data demonstrate that the newly selected ssDNA aptamers, especially aptamer ZE2, hold great promise for developing new molecular probes, as an early diagnostic reagent for HCV surface antigen, or a therapeutic drug specifically for HCV

Spatial heterogeneity in implicit housing prices: evidence from Hangzhou, China

Estimated coefficients in hedonic price models are generally assumed to be constant throughout the entire study area. However, increasing evidence reveals that the marginal prices of housing characteristics may vary over space and that the spatial heterogeneity problem in implicit housing prices should be given attention. Taking Hangzhou, China, as an example, this study uses the micro data of 603 residential communities in 2014 to examine spatial heterogeneity in implicit housing prices. On the basis of the traditional hedonic price model, we establish spatial expansion and geographically weighted regression (GWR) models for comparative analysis. Results show that the spatial expansion and GWR models have excellent goodness of fit and can improve the traditional hedonic price model. The mixed geographically weighted regression (MGWR) model further reveals that the implicit prices of nine housing characteristics vary significantly over space and that the impacts of the four remaining housing characteristics on housing prices are fixed throughout the entire study area. Unlike the traditional hedonic price model and spatial expansion model, the GWR/MGWR model has the unique advantage of visually providing the spatial distribution of implicit housing prices and accurately describing spatial heterogeneity