Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

The multimedia community has shown a significant interest in perceiving and representing the physical world with multimodal pretrained neural network models, and among them, the visual-language pertaining (VLP) is, currently, the most captivating topic. However, there have been few endeavors dedicated to the exploration of 1) whether essential linguistic knowledge (e.g., semantics and syntax) can be extracted during VLP, and 2) how such linguistic knowledge impact or enhance the multimodal alignment. In response, here we aim to elucidate the impact of comprehensive linguistic knowledge, including semantic expression and syntactic structure, on multimodal alignment. Specifically, we design and release the SNARE, the first large-scale multimodal alignment probing benchmark, to detect the vital linguistic components, e.g., lexical, semantic, and syntax knowledge, containing four tasks: Semantic structure, Negation logic, Attribute ownership, and Relationship composition. Based on our proposed probing benchmarks, our holistic analyses of five advanced VLP models illustrate that the VLP model: i) shows insensitivity towards complex syntax structures and relies on content words for sentence comprehension; ii) demonstrates limited comprehension of combinations between sentences and negations; iii) faces challenges in determining the presence of actions or spatial relationships within visual information and struggles with verifying the correctness of triple combinations. We make our benchmark and code available at \url{https://github.com/WangFei-2019/SNARE/}.Comment: [TL;DR] we design and release the SNARE, the first large-scale multimodal alignment probing benchmark for current vision-language pretrained model

Negated Complementary Commonsense using Large Language Models

Larger language models, such as GPT-3, have shown to be excellent in many tasks. However, we demonstrate that out-of-ordinary questions can throw the model off guard. This work focuses on finding answers to negated complementary questions in commonsense scenarios. We illustrate how such questions adversely affect the model responses. We propose a model-agnostic methodology to improve the performance in negated complementary scenarios. Our method outperforms few-shot generation from GPT-3 (by more than 11 points) and, more importantly, highlights the significance of studying the response of large language models in negated complementary questions. The code, data, and experiments are available under: https://github.com/navidre/negated_complementary_commonsense.Comment: Appeared in Natural Language Reasoning and Structured Explanations Workshop (NLRSE) - ACL 202

Logics and Their Galaxies

This article introduces some concepts that help exploring the ontological import of universal logic. It studies the notions of an antilogic and counterlogic associated to each logic and shows some of their properties. It presents the notion of galaxy, as the class of possible worlds compatible with a given logic.We explore some consequences of these developments

Logic Programming with Default, Weak and Strict Negations

This paper treats logic programming with three kinds of negation: default, weak and strict negations. A 3-valued logic model theory is discussed for logic programs with three kinds of negation. The procedure is constructed for negations so that a soundness of the procedure is guaranteed in terms of 3-valued logic model theory.Comment: 14 pages, to appear in Theory and Practice of Logic Programming (TPLP

Phase-Tunable Thermal Logic: Computation with Heat

Boolean algebra, the branch of mathematics where variables can assume only true or false value, is the theoretical basis of classical computation. The analogy between Boolean operations and electronic switching circuits, highlighted by Shannon in 1938, paved the way to modern computation based on electronic devices. The grow of computational power of such devices, after an exciting exponential -Moore trend, is nowadays blocked by heat dissipation due to computational tasks, very demanding after the chips miniaturization. Heat is often a detrimental form of energy which increases the systems entropy decreasing the efficiency of logic operations. Here, we propose a physical system able to perform thermal logic operations by reversing the old heat-disorder epitome into a novel heat-order paradigm. We lay the foundations of heat computation by encoding logic state variables in temperature and introducing the thermal counterparts of electronic logic gates. Exploiting quantum effects in thermally biased Josephson junctions (JJs), we propound a possible realization of a functionally complete dissipationless logic. Our architecture ensures high operation stability and robustness with switching frequencies reaching the GHz

Studies in optical parallel processing

Threshold and A/D devices for converting a gray scale image into a binary one were investigated for all-optical and opto-electronic approaches to parallel processing. Integrated optical logic circuits (IOC) and optical parallel logic devices (OPA) were studied as an approach to processing optical binary signals. In the IOC logic scheme, a single row of an optical image is coupled into the IOC substrate at a time through an array of optical fibers. Parallel processing is carried out out, on each image element of these rows, in the IOC substrate and the resulting output exits via a second array of optical fibers. The OPAL system for parallel processing which uses a Fabry-Perot interferometer for image thresholding and analog-to-digital conversion, achieves a higher degree of parallel processing than is possible with IOC