Align before Search: Aligning Ads Image to Text for Accurate Cross-Modal Sponsored Search

Cross-Modal sponsored search displays multi-modal advertisements (ads) when consumers look for desired products by natural language queries in search engines. Since multi-modal ads bring complementary details for query-ads matching, the ability to align ads-specific information in both images and texts is crucial for accurate and flexible sponsored search. Conventional research mainly studies from the view of modeling the implicit correlations between images and texts for query-ads matching, ignoring the alignment of detailed product information and resulting in suboptimal search performance.In this work, we propose a simple alignment network for explicitly mapping fine-grained visual parts in ads images to the corresponding text, which leverages the co-occurrence structure consistency between vision and language spaces without requiring expensive labeled training data. Moreover, we propose a novel model for cross-modal sponsored search that effectively conducts the cross-modal alignment and query-ads matching in two separate processes. In this way, the model matches the multi-modal input in the same language space, resulting in a superior performance with merely half of the training data. Our model outperforms the state-of-the-art models by 2.57% on a large commercial dataset. Besides sponsored search, our alignment method is applicable for general cross-modal search. We study a typical cross-modal retrieval task on the MSCOCO dataset, which achieves consistent performance improvement and proves the generalization ability of our method. Our code is available at https://github.com/Pter61/AlignCMSS

PlaceRaider: Virtual Theft in Physical Spaces with Smartphones

As smartphones become more pervasive, they are increasingly targeted by malware. At the same time, each new generation of smartphone features increasingly powerful onboard sensor suites. A new strain of sensor malware has been developing that leverages these sensors to steal information from the physical environment (e.g., researchers have recently demonstrated how malware can listen for spoken credit card numbers through the microphone, or feel keystroke vibrations using the accelerometer). Yet the possibilities of what malware can see through a camera have been understudied. This paper introduces a novel visual malware called PlaceRaider, which allows remote attackers to engage in remote reconnaissance and what we call virtual theft. Through completely opportunistic use of the camera on the phone and other sensors, PlaceRaider constructs rich, three dimensional models of indoor environments. Remote burglars can thus download the physical space, study the environment carefully, and steal virtual objects from the environment (such as financial documents, information on computer monitors, and personally identifiable information). Through two human subject studies we demonstrate the effectiveness of using mobile devices as powerful surveillance and virtual theft platforms, and we suggest several possible defenses against visual malware

Nonlinear multigrid based on local spectral coarsening for heterogeneous diffusion problems

This work develops a nonlinear multigrid method for diffusion problems discretized by cell-centered finite volume methods on general unstructured grids. The multigrid hierarchy is constructed algebraically using aggregation of degrees of freedom and spectral decomposition of reference linear operators associated with the aggregates. For rapid convergence, it is important that the resulting coarse spaces have good approximation properties. In our approach, the approximation quality can be directly improved by including more spectral degrees of freedom in the coarsening process. Further, by exploiting local coarsening and a piecewise-constant approximation when evaluating the nonlinear component, the coarse level problems are assembled and solved without ever re-visiting the fine level, an essential element for multigrid algorithms to achieve optimal scalability. Numerical examples comparing relative performance of the proposed nonlinear multigrid solvers with standard single-level approaches -- Picard's and Newton's methods -- are presented. Results show that the proposed solver consistently outperforms the single-level methods, both in efficiency and robustness

A reduced basis localized orthogonal decomposition

In this work we combine the framework of the Reduced Basis method (RB) with the framework of the Localized Orthogonal Decomposition (LOD) in order to solve parametrized elliptic multiscale problems. The idea of the LOD is to split a high dimensional Finite Element space into a low dimensional space with comparably good approximation properties and a remainder space with negligible information. The low dimensional space is spanned by locally supported basis functions associated with the node of a coarse mesh obtained by solving decoupled local problems. However, for parameter dependent multiscale problems, the local basis has to be computed repeatedly for each choice of the parameter. To overcome this issue, we propose an RB approach to compute in an "offline" stage LOD for suitable representative parameters. The online solution of the multiscale problems can then be obtained in a coarse space (thanks to the LOD decomposition) and for an arbitrary value of the parameters (thanks to a suitable "interpolation" of the selected RB). The online RB-LOD has a basis with local support and leads to sparse systems. Applications of the strategy to both linear and nonlinear problems are given