Do humans and machines have the same eyes? Human-machine perceptual differences on image classification

Trained computer vision models are assumed to solve vision tasks by imitating human behavior learned from training labels. Most efforts in recent vision research focus on measuring the model task performance using standardized benchmarks. Limited work has been done to understand the perceptual difference between humans and machines. To fill this gap, our study first quantifies and analyzes the statistical distributions of mistakes from the two sources. We then explore human vs. machine expertise after ranking tasks by difficulty levels. Even when humans and machines have similar overall accuracies, the distribution of answers may vary. Leveraging the perceptual difference between humans and machines, we empirically demonstrate a post-hoc human-machine collaboration that outperforms humans or machines alone.Comment: Paper under revie

M2C: Towards Automatic Multimodal Manga Complement

Multimodal manga analysis focuses on enhancing manga understanding with visual and textual features, which has attracted considerable attention from both natural language processing and computer vision communities. Currently, most comics are hand-drawn and prone to problems such as missing pages, text contamination, and aging, resulting in missing comic text content and seriously hindering human comprehension. In other words, the Multimodal Manga Complement (M2C) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for vision and language understanding. To this end, we first propose the Multimodal Manga Complement task by establishing a new M2C benchmark dataset covering two languages. First, we design a manga argumentation method called MCoT to mine event knowledge in comics with large language models. Then, an effective baseline FVP-M$^{2}$ using fine-grained visual prompts is proposed to support manga complement. Extensive experimental results show the effectiveness of FVP-M$^{2}$ method for Multimodal Mange Complement.Comment: EMNLP2023. arXiv admin note: text overlap with arXiv:2210.1546

Interaction induced decay of a heteronuclear two-atom system

Two-atom systems in small traps are of fundamental interest, first of all for understanding the role of interactions in degenerate cold gases and for the creation of quantum gates in quantum information processing with single-atom traps. One of the key quantities is the inelastic relaxation (decay) time when one of the atoms or both are in a higher hyperfine state. Here we measure this quantity in a heteronuclear system of $^{87}$Rb and $^{85}$Rb in a micro optical trap and demonstrate experimentally and theoretically the presence of both fast and slow relaxation processes, depending on the choice of the initial hyperfine states. The developed experimental method allows us to single out a particular relaxation process and, in this sense, our experiment is a "superclean platform" for collisional physics studies. Our results have also implications for engineering of quantum states via controlled collisions and creation of two-qubit quantum gates.Comment: 8 pages, 3 figure