Rethinking Multi-Modal Alignment in Video Question Answering from Feature and Sample Perspectives

Reasoning about causal and temporal event relations in videos is a new destination of Video Question Answering (VideoQA).The major stumbling block to achieve this purpose is the semantic gap between language and video since they are at different levels of abstraction. Existing efforts mainly focus on designing sophisticated architectures while utilizing frame- or object-level visual representations. In this paper, we reconsider the multi-modal alignment problem in VideoQA from feature and sample perspectives to achieve better performance. From the view of feature,we break down the video into trajectories and first leverage trajectory feature in VideoQA to enhance the alignment between two modalities. Moreover, we adopt a heterogeneous graph architecture and design a hierarchical framework to align both trajectory-level and frame-level visual feature with language feature. In addition, we found that VideoQA models are largely dependent on language priors and always neglect visual-language interactions. Thus, two effective yet portable training augmentation strategies are designed to strengthen the cross-modal correspondence ability of our model from the view of sample. Extensive results show that our method outperforms all the state-of-the-art models on the challenging NExT-QA benchmark, which demonstrates the effectiveness of the proposed method

Analytical and experimental investigation on eigenfrequency-based damage diagnosis of cantilever beam

This paper presents two eigenfrequency-based damage diagnosis methods in a cantilever beam. The analytical relationship has been established between the eigenfrequency and damage parameters, including relative damage location and severity. On the premise that pre-damaged eigenfrequencies are known, a diagnosis algorithm without requirement of material properties is proposed based on change ratios of the first three eigenfrequencies. If pre-damaged eigenfrequencies are unfeasible to be acquired, a three-contour method based on only post-damaged eigenfrequencies is introduced to estimate damage parameters. The uniqueness of solution is discussed. Both the numerical simulation by the finite element method and the experiment on real beams are conducted and result in a good agreement between actual damage parameters and calculated values by using the proposed methods

Protocol for bacterial typing using Fourier transform infrared spectroscopy

[EN]The Fourier transform infrared (FT-IR) signals obtained from bacterial samples are specific and reproducible, making FT-IR an efficient tool for bacterial typing at a subspecies level. However, the typing accuracy could be affected by many factors, including sample preparation and spectral acquisition. Here, we present a unified protocol for bacterial typing based on FT-IR spectroscopy. We describe sample preparation from bacterial culture and FT-IR spectrum collection. We then detail FT-IR spectrum preprocessing and multivariate analysis of spectral data for bacterial typing.SIThis work was supported by Moutai Group Research and Development Project (no. 2018023) and the National Natural Science Foundation of China (nos. 31470786, U1904196, 82073699, and 21275032)

The metabolic adaptation evoked by arginine enhances the effect of radiation in brain metastases

Selected patients with brain metastases (BM) are candidates for radiotherapy. A lactatogenic metabolism, common in BM, has been associated with radioresistance. We demonstrated that BM express nitric oxide (NO) synthase 2 and that administration of its substrate l-arginine decreases tumor lactate in BM patients. In a placebo-controlled trial, we showed that administration of l-arginine before each fraction enhanced the effect of radiation, improving the control of BM. Studies in preclinical models demonstrated that l-arginine radiosensitization is a NO-mediated mechanism secondary to the metabolic adaptation induced in cancer cells. We showed that the decrease in tumor lactate was a consequence of reduced glycolysis that also impacted ATP and NAD+ levels. These effects were associated with NO-dependent inhibition of GAPDH and hyperactivation of PARP upon nitrosative DNA damage. These metabolic changes ultimately impaired the repair of DNA damage induced by radiation in cancer cells while greatly sparing tumor-infiltrating lymphocytes.Fil: Marullo, Rossella. Cornell University; Estados UnidosFil: Castro, Monica. Universidad de Buenos Aires; ArgentinaFil: Yomtoubian, Shira. Cornell University; Estados UnidosFil: Nieves Calvo Vidal, M.. Cornell University; Estados UnidosFil: Revuelta, María Victoria. Consejo Nacional de Investigaciones Científicas y Técnicas; ArgentinaFil: Krumsiek, Jan. Cornell University; Estados UnidosFil: Nicholas, Andrew P.. Cornell University; Estados UnidosFil: Cresta Morgado, Pablo. Universidad de Buenos Aires; ArgentinaFil: Yang, ShaoNing. Cornell University; Estados UnidosFil: Medina, Vanina Araceli. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Houssay. Instituto de Investigaciones Biomédicas. Universidad de Buenos Aires. Facultad de Medicina. Instituto de Investigaciones Biomédicas; ArgentinaFil: Roth, Berta María Cristina. Universidad de Buenos Aires; ArgentinaFil: Bonomi, Marcelo. Ohio State University; Estados UnidosFil: Keshari, Kayvan R.. Memorial Sloan Kettering Cancer Center; Estados UnidosFil: Mittal, Vivek. Cornell University; Estados UnidosFil: Navigante, Alfredo Hugo. Universidad de Buenos Aires; ArgentinaFil: Cerchietti, Leandro. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina. Universidad de Buenos Aires. Facultad de Medicina. Instituto de Oncología "Ángel H. Roffo"; Argentin

A Soft-YoloV4 for High-Performance Head Detection and Counting

Blockage of pedestrians will cause inaccurate people counting, and people’s heads are easily blocked by each other in crowded occasions. To reduce missed detections as much as possible and improve the capability of the detection model, this paper proposes a new people counting method, named Soft-YoloV4, by attenuating the score of adjacent detection frames to prevent the occurrence of missed detection. The proposed Soft-YoloV4 improves the accuracy of people counting and reduces the incorrect elimination of the detection frames when heads are blocked by each other. Compared with the state-of-the-art YoloV4, the AP value of the proposed head detection method is increased from 88.52 to 90.54%. The Soft-YoloV4 model has much higher robustness and a lower missed detection rate for head detection, and therefore it dramatically improves the accuracy of people counting

A Soft-YoloV4 for High-Performance Head Detection and Counting

Blockage of pedestrians will cause inaccurate people counting, and people’s heads are easily blocked by each other in crowded occasions. To reduce missed detections as much as possible and improve the capability of the detection model, this paper proposes a new people counting method, named Soft-YoloV4, by attenuating the score of adjacent detection frames to prevent the occurrence of missed detection. The proposed Soft-YoloV4 improves the accuracy of people counting and reduces the incorrect elimination of the detection frames when heads are blocked by each other. Compared with the state-of-the-art YoloV4, the AP value of the proposed head detection method is increased from 88.52 to 90.54%. The Soft-YoloV4 model has much higher robustness and a lower missed detection rate for head detection, and therefore it dramatically improves the accuracy of people counting

MCSNet+ : enhanced convolutional neural network for detection and classification of tribolium and sitophilus sibling species in actual wheat storage environments

Insect pests like Tribolium and Sitophilus siblings are major threats to grain storage and processing, causing quality and quantity losses that endanger food security. These closely related species, having very similar morphological and biological characteristics, often exhibit variations in biology and pesticide resistance, complicating control efforts. Accurate pest species identification is essential for effective control, but workplace safety in the grain bin associated with grain deterioration, clumping, fumigator hazards, and air quality create challenges. Therefore, there is a pressing need for an online automated detection system. In this work, we enriched the stored-grain pest sibling image dataset, which includes 25,032 annotated Tribolium samples of two species and five geographical strains from real warehouse and another 1774 from the lab. As previously demonstrated on the Sitophilus family, Convolutional Neural Networks demonstrate distinct advantages over other model architectures in detecting Tribolium. Our CNN model, MCSNet+, integrates Soft-NMS for better recall in dense object detection, a Position-Sensitive Prediction Model to handle translation issues, and anchor parameter fine-tuning for improved matching and speed. This approach significantly enhances mean Average Precision (mAP) for Sitophilus and Tribolium, reaching a minimum of 92.67 ± 1.74% and 94.27 ± 1.02%, respectively. Moreover, MCSNet+ exhibits significant improvements in prediction speed, advancing from 0.055 s/img to 0.133 s/img, and elevates the recognition rates of moving insect sibling species in real wheat storage and visible light, rising from 2.32% to 2.53%. The detection performance of the model on laboratory-captured images surpasses that of real storage facilities, with better results for Tribolium compared to Sitophilus. Although inter-strain variances are less pronounced, the model achieves acceptable detection results across different Tribolium geographical strains, with a minimum recognition rate of 82.64 ± 1.27%. In real-time monitoring videos of grain storage facilities with wheat backgrounds, the enhanced deep learning model based on Convolutional Neural Networks successfully detects and identifies closely related stored-grain pest images. This achievement provides a viable solution for establishing an online pest management system in real storage facilities. © 2023 by the authors

Auto-identification of two Sitophilus sibling species on stored wheat using deep convolutional neural network

BACKGROUND: Sitophilus oryzae and Sitophilus zeamais are the two main insect pests that infest stored grain worldwide. Accurate and rapid identification of the two pests is challenging because of their similar appearances. The S. zeamais adults are darker and shinier than S. oryzae in visible light. Convolutional neural network (CNN) can be applied for the effective differentiation due to its high effectiveness in object recognition. RESULTS: We propose a multilayer convolutional structure (MCS) feature extractor to extract insect characteristics within each layer of the CNN architecture. A region proposal network is adopted to determine the location of a potential pest in the wheat background. The precision of classification and the robustness of bounding box regression are increased by including deeper layer variables into the classification and bounding box regression subnets, as well as combining loss functions softmax and smooth L1. The proposed multilayer convolutional structure network (MCSNet) achieves the mean average precision of 87.89 ± 2.36% from the laboratory test, with an average detection speed of 0.182 ± 0.005 s per test. The model was further assessed with the field trials, and the obtained accuracy was 90.35 ± 3.12%. For all test conditions, the average precision for S. oryzae was higher than that for S. zeamais. CONCLUSION: The proposed MCSNet model has demonstrated that it is a fast and accurate method for detecting sibling species from visible light images in both laboratory and field trials. This will ultimately be applied for pest management together with an upgraded industrial camera system, which has been installed in over 100 000 grain depots of China. © 2022 Society of Chemical Industry