Missing Modality meets Meta Sampling (M3S): An Efficient Universal Approach for Multimodal Sentiment Analysis with Missing Modality

Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may include incomplete data. Most existing works that address missing modalities usually assume a particular modality is completely missing and seldom consider a mixture of missing across multiple modalities. In this paper, we propose a simple yet effective meta-sampling approach for multimodal sentiment analysis with missing modalities, namely Missing Modality-based Meta Sampling (M3S). To be specific, M3S formulates a missing modality sampling strategy into the modal agnostic meta-learning (MAML) framework. M3S can be treated as an efficient add-on training component on existing models and significantly improve their performances on multimodal data with a mixture of missing modalities. We conduct experiments on IEMOCAP, SIMS and CMU-MOSI datasets, and superior performance is achieved compared with recent state-of-the-art methods

Tuning 2D perovskite–graphene layered composite for photocatalysis †

The augmentation of photocatalytic activity in layered perovskite oxides via the integration of graphene-like materials presents a promising pathway for the optimization of solar energy conversion. The electron-rich nature of graphene, coupled with its high electron conductivity, functions as an effective photosensitizer, thereby enhancing visible light harvesting. In this investigation, we have, for the first time, assembled ultrathin exfoliated Dion–Jacobson perovskite layers with reduced graphene oxide (rGO) layers, resulting in a high surface area layered nanocomposite, achieved through a tailored electrostatic approach. To further refine the electron properties of the layered perovskite–reduced graphene oxide composites, we have explored the use of various lanthanides as A-site cations in the Dion–Jacobson perovskites, including LaNb2O7 (LNO), PrNb2O7 (PNO), and NdNb2O7 (NNO). The synthesized composites demonstrate exceptional performance in photocatalytic H2 production, with rGO/NNO exhibiting the highest activity, achieving a hydrogen evolution rate (HER) of 835 μmol g−1 under light illumination, attributable to optimal interfacial effects. Our experimental and theoretical analyses indicate that hydrogen production is predominantly influenced by the A-site cation charge density at the materials' interface, as dictated by the charge transfer dynamics. This research potentially contributes to the comprehension and enhancement of photocatalytic processes for applications in solar energy conversion

MovieChat: From Dense Token to Sparse Memory for Long Video Understanding

Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing systems can only handle videos with very few frames. For long videos, the computation complexity, memory cost, and long-term temporal connection impose additional challenges. Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose the MovieChat to overcome these challenges. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video and 14K manual annotations for validation of the effectiveness of our method.Comment: CVPR 2024. First three authors contribute equally to this work. Project Website https://rese1f.github.io/MovieChat

Unconventional Charge-to-Spin Conversion in Graphene/MoTe2 van der Waals Heterostructures

Spin-charge interconversion (SCI) is a central phenomenon to the development of spintronic devices from materials with strong spin-orbit coupling (SOC). In the case of materials with high crystal symmetry, the only allowed SCI processes are those where the spin-current, charge-current, and spin-polarization directions are orthogonal to each other. Consequently, standard SCI experiments are designed to maximize the signals arising from the SCI processes with conventional mutually orthogonal geometry. However, in low-symmetry materials, certain nonorthogonal SCI processes are also allowed. Since the standard SCI experiment is limited to charge current flowing only in one direction in the SOC material, certain allowed SCI configurations remain unexplored. Here, we perform a thorough SCI study in a graphene-based lateral spin valve combined with low-symmetry MoTe2. Due to a very low contact resistance between the two materials, we can detect SCI signals using both a standard configuration, where the charge current is applied along MoTe2, and a recently introduced [three-dimensional- (3D) current] configuration, where the charge-current flow can be controlled in three directions within the heterostructure. As a result, we observe three different SCI components, one orthogonal and two nonorthogonal, adding valuable insight into the SCI processes in low-symmetry materials. The large SCI signals obtained at room temperature, along with the versatility of the 3D-current configuration, provide feasibility and flexibility to the design of the next generation of spin-based devices.This work is supported by the Spanish MICINN under Projects No. RTI2018-094861-B-I00, No. PGC2018-101988-B-C21, No. PID2019-109905GB-C21, No. MAT2017-88377-C2-2-R, and the Maria de Maeztu Units of Excellence Programme (Grants No. MDM-2016-0618 and No. CEX2020-001038-M); the “Valleytronics” Intel Science Technology Center; the Gipuzkoa Regional Council under Projects No. 2021-CIEN-000037-01 and No. 2021-CIEN-000070-01; and the European Union H2020 under the Marie Sklodowska-Curie Actions (Grants No. 0766025-QuESTech and No. 794982-2DSTOP). N.O. thanks the Spanish MICINN for support from a Ph.D. fellowship (Grant No. BES-2017-07963). J.I.-A. acknowledges support from the “Juan de la Cierva-Formación” program by the Spanish MICINN (Grant No. FJC2018-038688-I) for a postdoctoral fellowship. R.C. acknowledges funding from Generalitat Valenciana through Grant No. CIDEGENT/2018/004 M.G.V. and I.R. thanks support from the Spanish MICINN (grant PID2019-109905GBC21), the German Research Foundation DFG (grant nr. GA3314/1-1-FOR 5249 QUAST) and the European Research Council ERC (Grant No. 101020833)

Segment Anything Model for Medical Images?

The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It designed a novel promotable segmentation task, ensuring zero-shot image segmentation using the pre-trained model via two main modes including automatic everything and manual prompt. SAM has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging due to the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. SAM has achieved impressive results on various natural image segmentation tasks. Meanwhile, zero-shot and efficient MIS can well reduce the annotation time and boost the development of medical image analysis. Hence, SAM seems to be a potential tool and its performance on large medical datasets should be further validated. We collected and sorted 52 open-source datasets, and build a large medical segmentation dataset with 16 modalities, 68 objects, and 553K slices. We conducted a comprehensive analysis of different SAM testing strategies on the so-called COSMOS 553K dataset. Extensive experiments validate that SAM performs better with manual hints like points and boxes for object perception in medical images, leading to better performance in prompt mode compared to everything mode. Additionally, SAM shows remarkable performance in some specific objects and modalities, but is imperfect or even totally fails in other situations. Finally, we analyze the influence of different factors (e.g., the Fourier-based boundary complexity and size of the segmented objects) on SAM's segmentation performance. Extensive experiments validate that SAM's zero-shot segmentation capability is not sufficient to ensure its direct application to the MIS.Comment: 23 pages, 14 figures, 12 table

Gate-tunable spin hall effect in an all-light-element heterostructure: Graphene with copper oxide

Graphene is a light material for long-distance spin transport due to its low spin–orbit coupling, which at the same time is the main drawback for exhibiting a sizable spin Hall effect. Decoration by light atoms has been predicted to enhance the spin Hall angle in graphene while retaining a long spin diffusion length. Here, we combine a light metal oxide (oxidized Cu) with graphene to induce the spin Hall effect. Its efficiency, given by the product of the spin Hall angle and the spin diffusion length, can be tuned with the Fermi level position, exhibiting a maximum (1.8 ± 0.6 nm at 100 K) around the charge neutrality point. This all-light-element heterostructure shows a larger efficiency than conventional spin Hall materials. The gate-tunable spin Hall effect is observed up to room temperature. Our experimental demonstration provides an efficient spin-to-charge conversion system free from heavy metals and compatible with large-scale fabrication.We acknowledge funding by the “Valleytronics” Intel Science Technology Center, by the Spanish MICINN (Project No. PID2021-122511OB-I00 and “Maria de Maeztu” Units of Excellence Programme No. CEX2020-001038-M), by the European Union H2020 under the Marie Sklodowska–Curie Actions (Project Nos. 0766025-QuESTech and 955671-SPEAR), and by Diputación de Gipuzkoa (Project No. 2021-CIEN-000037-01). Z.C. acknowledges postdoctoral fellowship support from the “Juan de la Cierva” Programme by the Spanish MICINN (grant No. FJC2021-047257-I). N.O. acknowledges the Spanish MICINN for support from a Ph.D.fellowship (Grant No. BES-2017-07963).With funding from the Spanish government through the "Severo Ochoa Centre of Excellence" accreditation (CEX2020-001038-M).Peer reviewe

Segment anything model for medical images?

The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: (1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. (2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. (3) SAM performed better with manual hints, especially box, than the Everything mode. (4) SAM could help human annotation with high labeling quality and less time. (5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. (6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. (7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. (8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. Codes and models are available at: https://github.com/yuhoo0302/Segment-Anything-Model-for-Medical-Images. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.</p

Uncovering the phonon spectra and lattice dynamics of plastically deformable InSe van der Waals crystals

Abstract Stacking two-dimensional (2D) van der Waals (vdW) materials in a layered bulk structure provides an appealing platform for the emergence of exotic physical properties. As a vdW crystal with exceptional plasticity, InSe offers the opportunity to explore various effects arising from the coupling of its peculiar mechanical behaviors and other physical properties. Here, we employ neutron scattering techniques to investigate the correlations of plastic interlayer slip, lattice anharmonicity, and thermal transport in InSe crystals. Not only are the interlayer slip direction and magnitude well captured by shifts in the Bragg reflections, but we also observe a deviation from the expected Debye behaviour in the heat capacity and lattice thermal conductivity. Combining the experimental data with first-principles calculations, we tentatively attribute the observed evidence of strong phonon-phonon interactions to a combination of a large acoustic-optical frequency resonance and a nesting effect. These findings correlate the macroscopic plastic slip and the microscopic lattice dynamics, providing insights into the mechano-thermo coupling and modulation in 2D vdW materials