Align before Search: Aligning Ads Image to Text for Accurate Cross-Modal Sponsored Search

Cross-Modal sponsored search displays multi-modal advertisements (ads) when consumers look for desired products by natural language queries in search engines. Since multi-modal ads bring complementary details for query-ads matching, the ability to align ads-specific information in both images and texts is crucial for accurate and flexible sponsored search. Conventional research mainly studies from the view of modeling the implicit correlations between images and texts for query-ads matching, ignoring the alignment of detailed product information and resulting in suboptimal search performance.In this work, we propose a simple alignment network for explicitly mapping fine-grained visual parts in ads images to the corresponding text, which leverages the co-occurrence structure consistency between vision and language spaces without requiring expensive labeled training data. Moreover, we propose a novel model for cross-modal sponsored search that effectively conducts the cross-modal alignment and query-ads matching in two separate processes. In this way, the model matches the multi-modal input in the same language space, resulting in a superior performance with merely half of the training data. Our model outperforms the state-of-the-art models by 2.57% on a large commercial dataset. Besides sponsored search, our alignment method is applicable for general cross-modal search. We study a typical cross-modal retrieval task on the MSCOCO dataset, which achieves consistent performance improvement and proves the generalization ability of our method. Our code is available at https://github.com/Pter61/AlignCMSS

Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrieval

Different from Composed Image Retrieval task that requires expensive labels for training task-specific models, Zero-Shot Composed Image Retrieval (ZS-CIR) involves diverse tasks with a broad range of visual content manipulation intent that could be related to domain, scene, object, and attribute. The key challenge for ZS-CIR tasks is to learn a more accurate image representation that has adaptive attention to the reference image for various manipulation descriptions. In this paper, we propose a novel context-dependent mapping network, named Context-I2W, for adaptively converting description-relevant Image information into a pseudo-word token composed of the description for accurate ZS-CIR. Specifically, an Intent View Selector first dynamically learns a rotation rule to map the identical image to a task-specific manipulation view. Then a Visual Target Extractor further captures local information covering the main targets in ZS-CIR tasks under the guidance of multiple learnable queries. The two complementary modules work together to map an image to a context-dependent pseudo-word token without extra supervision. Our model shows strong generalization ability on four ZS-CIR tasks, including domain conversion, object composition, object manipulation, and attribute manipulation. It obtains consistent and significant performance boosts ranging from 1.88% to 3.60% over the best methods and achieves new state-of-the-art results on ZS-CIR. Our code is available at https://github.com/Pter61/context_i2w

Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service

Recent advances in vision-language pre-trained models (VLPs) have significantly increased visual understanding and cross-modal analysis capabilities. Companies have emerged to provide multi-modal Embedding as a Service (EaaS) based on VLPs (e.g., CLIP-based VLPs), which cost a large amount of training data and resources for high-performance service. However, existing studies indicate that EaaS is vulnerable to model extraction attacks that induce great loss for the owners of VLPs. Protecting the intellectual property and commercial ownership of VLPs is increasingly crucial yet challenging. A major solution of watermarking model for EaaS implants a backdoor in the model by inserting verifiable trigger embeddings into texts, but it is only applicable for large language models and is unrealistic due to data and model privacy. In this paper, we propose a safe and robust backdoor-based embedding watermarking method for VLPs called VLPMarker. VLPMarker utilizes embedding orthogonal transformation to effectively inject triggers into the VLPs without interfering with the model parameters, which achieves high-quality copyright verification and minimal impact on model performance. To enhance the watermark robustness, we further propose a collaborative copyright verification strategy based on both backdoor trigger and embedding distribution, enhancing resilience against various attacks. We increase the watermark practicality via an out-of-distribution trigger selection approach, removing access to the model training data and thus making it possible for many real-world scenarios. Our extensive experiments on various datasets indicate that the proposed watermarking approach is effective and safe for verifying the copyright of VLPs for multi-modal EaaS and robust against model extraction attacks. Our code is available at https://github.com/Pter61/vlpmarker

Strain Induced One-Dimensional Landau-Level Quantization in Corrugated Graphene

Theoretical research has predicted that ripples of graphene generates effective gauge field on its low energy electronic structure and could lead to zero-energy flat bands, which are the analog of Landau levels in real magnetic fields. Here we demonstrate, using a combination of scanning tunneling microscopy and tight-binding approximation, that the zero-energy Landau levels with vanishing Fermi velocities will form when the effective pseudomagnetic flux per ripple is larger than the flux quantum. Our analysis indicates that the effective gauge field of the ripples results in zero-energy flat bands in one direction but not in another. The Fermi velocities in the perpendicular direction of the ripples are not renormalized at all. The condition to generate the ripples is also discussed according to classical thin-film elasticity theory.Comment: 4 figures, Phys. Rev.

Mechanisms connecting square dance to sleep quality among middle-aged and older Chinese females: serial mediation roles of social support and depressive symptoms

BackgroundSquare dance is gaining increasing popularity among middle-aged and older Chinese women who are also at high risk of sleep disturbance. Although previous studies have shown exercise could improve sleep quality, the association between square dance and sleep quality remains to be discussed, and even less is known about the potential mechanism underlying this association.PurposeThis study aims to investigate the relationship between square dance and sleep quality and test if social support and depressive symptoms together play a serial mediating role in the influence of square dance on sleep quality.MethodsA cross-sectional study was conducted among 549 middle-aged and older Chinese females from September to December 2020 in Shao Yang City, Hunan Province of China, with ethics approval granted (SYU [2020]002). Square dance involvement was assessed by three questions about the time participants spent in square dance. Social support, depressive symptoms, and sleep quality were measured using the Pittsburgh Sleep Quality Index (PSQI), Social Support Self-Rating Scale (SSRS), and 9-item Patient Health Questionnaire (PHQ-9), respectively. The serial mediation model was analyzed by the bootstrapping method to assess whether social support and depressive symptoms mediate the relationship between square dance and sleep quality.ResultsTwo-thirds of the participants had high involvement in square dance and most reported a moderate and high level of social support (98.54%). The prevalence of depressive symptoms and sleep disturbance was 19.49 and 26.78%, respectively. The serial mediation model showed a significant association between square dance and sleep quality, which was fully mediated by social support and depressive symptoms in a serial model (total effect c = −0.114, 95%CI = −0.227 to −0.001; direct effect c’ = −0.036, 95% CI = −0.138 to 0.065; total indirect effect ab = −0.077, 95% CI = -0.139 to-0.016).ConclusionOur study extends the understanding of how square dance is associated with sleep quality through the serial mediating roles of social support and depressive symptoms. It provides crucial implications for developing square dance interventions to improve sleep quality among middle-aged and older Chinese females

Effect of tea intake on genetic predisposition to gout and uric acid: a Mendelian randomization study

ObjectiveThe effect of tea on gout and uric acid is still controversial. This study aims to analyze the effect of tea intake on genetic predisposition to gout, idiopathic gout, gout due to impairment of renal function as well as uric acid by Mendelian randomization (MR).MethodsForty independent single nucleotide polymorphisms (SNPs) associated with tea intake were selected from UK Biobank. SNPs for uric acid were obtained from BioBank Japan, SNPs for gout were obtained from UK Biobank, and SNPs for gout due to impairment of renal function and idiopathic gout were derived from FinnGen. The causal relationship of exposure-outcome was tested using inverse variance weighted, MR-Egger and weighted median. MR-Egger intercept was employed to assess horizontal pleiotropy, Cochran’s Q test was used to assess heterogeneity, and leave-one-out sensitivity analysis was utilized to analyze the stability of the results.ResultsThe results of MR analysis showed that tea intake was negatively associated with gout due to impairment of renal function (OR 0.997, 95% CI 0.994 to 0.999, P = 0.017), whereas there was no causal association with gout, idiopathic gout, and uric acid (P > 0.05), for which sensitivity analysis suggested that these results were robust.ConclusionsThere was a genetic predisposition effect of increased tea intake on the reduced risk of gout due to impairment of renal function, whereas there was no such effect on gout, idiopathic gout, and uric acid. Tea intake may become an important option in the dietary treatment of gout due to impairment of renal function

The genome of\u3ci\u3e Orychophragmus\u3c/i\u3e violaceus provides genomic insights into the evolution of Brassicaceaepolyploidizationandits distinct traits

Orychophragmus violaceus, referred to as ‘‘eryuelan’’ (February orchid) in China, is an early-flowering ornamental plant. The high oil content and abundance of unsaturated fatty acids in O. violaceus seeds make it a potential high-quality oilseed crop. Here, we generated a whole-genome assembly for O. violaceus using Nanopore and Hi-C sequencing technologies. The assembled genome of O. violaceus was ~1.3 Gb in size, with 12 pairs of chromosomes. Through investigation of ancestral genome evolution, we determined that the genome of O. violaceus experienced a tetraploidization event from a diploid progenitor with the translocated proto-Calepineae karyotype. Comparisons between the reconstructed subgenomes of O. violaceus identified indicators of subgenome dominance, indicating that subgenomes likely originated via allotetraploidy. O. violaceus was phylogenetically close to the Brassica genus, and tetraploidy in O. violaceus occurred approximately 8.57 million years ago, close in time to the whole-genome triplication of Brassica that likely arose via an intermediate tetraploid lineage. However, the tetraploidization in Orychophragmus was independent of the hexaploidization in Brassica, as evidenced by the results from detailed phylogenetic analyses and comparisons of the break and fusion points of ancestral genomic blocks. Moreover, identification of multi-copy genes regulating the production of high-quality oil highlighted the contributions of both tetraploidization and tandem duplication to functional innovation in O. violaceus. These findings provide novel insights into the polyploidization evolution of plant species and will promote both functional genomic studies and domestication/breeding efforts in O. violaceus