Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition

Human emotion recognition plays an important role in human-computer interaction. In this paper, we present our approach to the Valence-Arousal (VA) Estimation Challenge, Expression (Expr) Classification Challenge, and Action Unit (AU) Detection Challenge of the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). Specifically, we propose a novel multi-modal fusion model that leverages Temporal Convolutional Networks (TCN) and Transformer to enhance the performance of continuous emotion recognition. Our model aims to effectively integrate visual and audio information for improved accuracy in recognizing emotions. Our model outperforms the baseline and ranks 3 in the Expression Classification challenge.Comment: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW

Predicting Temporal Sets with Deep Neural Networks

Given a sequence of sets, where each set contains an arbitrary number of elements, the problem of temporal sets prediction aims to predict the elements in the subsequent set. In practice, temporal sets prediction is much more complex than predictive modelling of temporal events and time series, and is still an open problem. Many possible existing methods, if adapted for the problem of temporal sets prediction, usually follow a two-step strategy by first projecting temporal sets into latent representations and then learning a predictive model with the latent representations. The two-step approach often leads to information loss and unsatisfactory prediction performance. In this paper, we propose an integrated solution based on the deep neural networks for temporal sets prediction. A unique perspective of our approach is to learn element relationship by constructing set-level co-occurrence graph and then perform graph convolutions on the dynamic relationship graphs. Moreover, we design an attention-based module to adaptively learn the temporal dependency of elements and sets. Finally, we provide a gated updating mechanism to find the hidden shared patterns in different sequences and fuse both static and dynamic information to improve the prediction performance. Experiments on real-world data sets demonstrate that our approach can achieve competitive performances even with a portion of the training data and can outperform existing methods with a significant margin.Comment: 9 pages, 6 figures, Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '2020

Stress Analysis Of Key Components And Vibration Property Research Of The Meshing Pair In Single Screw Compressors

The single screw compressor(SSC) is widely applied to air compression, refrigeration, petrochemical industry, waste heat recovery, etc. Among SSCs, the respective strength and stiffness of the casing, screw rotor and gaterotor, where relative motions occur, play a key role on the machine running and clearance fit. Dynamic properties of the meshing pair directly affect the SSC’s vibration, as well as gaterotor’s wear-out failure. In this thesis, the strength, stiffness and dynamic characteristics of the meshing pair under different operation conditions or with components made of different materials were analyzed. Analytical method and FEM were combined to calculate and analyze the issues above. Main contents and conclusions are as follows: Stress and deformation analysis of key components were implemented by ANSYS Workbench. The results show that both the maximum stress of the casing and the deformation of the gaterotor are basically linear to the discharge pressure. The first six natural frequencies and the corresponding vibration modes of the screw rotor and gaterotor were obtained to analyze and predict their respective vibration properties. It turned out that natural properties of the screw rotor change little on account of rotation speed and damping. Neither the screw rotor nor the gaterotor would resonate. Some exploratory work about the coupling interaction between gaterotor and its support was done. It is concluded that the gaterotor would suffer the support’s collision excitations consequently

Automated Ortho-Rectification of UAV-Based Hyperspectral Data over an Agricultural Field Using Frame RGB Imagery

Low-cost Unmanned Airborne Vehicles (UAVs) equipped with consumer-grade imaging systems have emerged as a potential remote sensing platform that could satisfy the needs of a wide range of civilian applications. Among these applications, UAV-based agricultural mapping and monitoring have attracted significant attention from both the research and professional communities. The interest in UAV-based remote sensing for agricultural management is motivated by the need to maximize crop yield. Remote sensing-based crop yield prediction and estimation are primarily based on imaging systems with different spectral coverage and resolution (e.g., RGB and hyperspectral imaging systems). Due to the data volume, RGB imaging is based on frame cameras, while hyperspectral sensors are primarily push-broom scanners. To cope with the limited endurance and payload constraints of low-cost UAVs, the agricultural research and professional communities have to rely on consumer-grade and light-weight sensors. However, the geometric fidelity of derived information from push-broom hyperspectral scanners is quite sensitive to the available position and orientation established through a direct geo-referencing unit onboard the imaging platform (i.e., an integrated Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS). This paper presents an automated framework for the integration of frame RGB images, push-broom hyperspectral scanner data and consumer-grade GNSS/INS navigation data for accurate geometric rectification of the hyperspectral scenes. The approach relies on utilizing the navigation data, together with a modified Speeded-Up Robust Feature (SURF) detector and descriptor, for automating the identification of conjugate features in the RGB and hyperspectral imagery. The SURF modification takes into consideration the available direct geo-referencing information to improve the reliability of the matching procedure in the presence of repetitive texture within a mechanized agricultural field. Identified features are then used to improve the geometric fidelity of the previously ortho-rectified hyperspectral data. Experimental results from two real datasets show that the geometric rectification of the hyperspectral data was improved by almost one order of magnitude

Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture

Either RGB images or inertial signals have been used for the task of motion capture (mocap), but combining them together is a new and interesting topic. We believe that the combination is complementary and able to solve the inherent difficulties of using one modality input, including occlusions, extreme lighting/texture, and out-of-view for visual mocap and global drifts for inertial mocap. To this end, we propose a method that fuses monocular images and sparse IMUs for real-time human motion capture. Our method contains a dual coordinate strategy to fully explore the IMU signals with different goals in motion capture. To be specific, besides one branch transforming the IMU signals to the camera coordinate system to combine with the image information, there is another branch to learn from the IMU signals in the body root coordinate system to better estimate body poses. Furthermore, a hidden state feedback mechanism is proposed for both two branches to compensate for their own drawbacks in extreme input cases. Thus our method can easily switch between the two kinds of signals or combine them in different cases to achieve a robust mocap. %The two divided parts can help each other for better mocap results under different conditions. Quantitative and qualitative results demonstrate that by delicately designing the fusion method, our technique significantly outperforms the state-of-the-art vision, IMU, and combined methods on both global orientation and local pose estimation. Our codes are available for research at https://shaohua-pan.github.io/robustcap-page/.Comment: Accepted by SIGGRAPH ASIA 2023. Project page: https://shaohua-pan.github.io/robustcap-page

SynDB: a Synapse protein DataBase based on synapse ontology

A synapse is the junction across which a nerve impulse passes from an axon terminal to a neuron, muscle cell or gland cell. The functions and building molecules of the synapse are essential to almost all neurobiological processes. To describe synaptic structures and functions, we have developed Synapse Ontology (SynO), a hierarchical representation that includes 177 terms with hundreds of synonyms and branches up to eight levels deep. associated 125 additional protein keywords and 109 InterPro domains with these SynO terms. Using a combination of automated keyword searches, domain searches and manual curation, we collected 14 000 non-redundant synapse-related proteins, including 3000 in human. We extensively annotated the proteins with information about sequence, structure, function, expression, pathways, interactions and disease associations and with hyperlinks to external databases. The data are stored and presented in the Synapse protein DataBase (SynDB, ). SynDB can be interactively browsed by SynO, Gene Ontology (GO), domain families, species, chromosomal locations or Tribe-MCL clusters. It can also be searched by text (including Boolean operators) or by sequence similarity. SynDB is the most comprehensive database to date for synaptic proteins

Robust estimation of bacterial cell count from optical density

Optical density (OD) is widely used to estimate the density of cells in liquid culture, but cannot be compared between instruments without a standardized calibration protocol and is challenging to relate to actual cell count. We address this with an interlaboratory study comparing three simple, low-cost, and highly accessible OD calibration protocols across 244 laboratories, applied to eight strains of constitutive GFP-expressing E. coli. Based on our results, we recommend calibrating OD to estimated cell count using serial dilution of silica microspheres, which produces highly precise calibration (95.5% of residuals <1.2-fold), is easily assessed for quality control, also assesses instrument effective linear range, and can be combined with fluorescence calibration to obtain units of Molecules of Equivalent Fluorescein (MEFL) per cell, allowing direct comparison and data fusion with flow cytometry measurements: in our study, fluorescence per cell measurements showed only a 1.07-fold mean difference between plate reader and flow cytometry data

Feature-Based Approach for the Registration of Pushbroom Imagery with Existing Orthophotos

Low-cost Unmanned Airborne Vehicles (UAVs) are rapidly becoming suitable platforms for acquiring remote sensing data for a wide range of applications. For example, a UAV-based mobile mapping system (MMS) is emerging as a novel phenotyping tool that delivers several advantages to alleviate the drawbacks of conventional manual plant trait measurements. Moreover, UAVs equipped with direct geo-referenced frame cameras and pushbroom scanners can acquire geospatial data for comprehensive high-throughput phenotyping. UAVs for mobile mapping platforms are low-cost and easy to use, can fly closer to the objects, and are filling an important gap between ground wheel-based and traditional manned-airborne platforms. However, consumer-grade UAVs are capable of carrying only equipment with a relatively light payload and their flying time is determined by a limited battery life. These restrictions of UAVs unfortunately force potential users to adopt lower-quality direct geo-referencing and imaging systems that may negatively impact the quality of the deliverables. Recent advances in sensor calibration and automated triangulation have made it feasible to obtain accurate mapping using low-cost camera systems equipped with consumer-grade GNSS/INS units. However, ortho-rectification of the data from a linear-array scanner is challenging for low-cost UAV systems, because the derived geo-location information from pushbroom sensors is quite sensitive to the performance of the implemented direct geo-referencing unit. This thesis presents a novel approach for improving the ortho-rectification of hyperspectral pushbroom scanner imagery with the aid of orthophotos generated from frame cameras through the identification of conjugate features while modeling the impact of residual artifacts in the direct geo-referencing information. The experimental results qualitatively and quantitatively proved the feasibility of the proposed methodology in improving the geo-referencing accuracy of real datasets collected over an agricultural field