Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation

Video scene graph generation (VidSGG) aims to identify objects in visual scenes and infer their relationships for a given video. It requires not only a comprehensive understanding of each object scattered on the whole scene but also a deep dive into their temporal motions and interactions. Inherently, object pairs and their relationships enjoy spatial co-occurrence correlations within each image and temporal consistency/transition correlations across different images, which can serve as prior knowledge to facilitate VidSGG model learning and inference. In this work, we propose a spatial-temporal knowledge-embedded transformer (STKET) that incorporates the prior spatial-temporal knowledge into the multi-head cross-attention mechanism to learn more representative relationship representations. Specifically, we first learn spatial co-occurrence and temporal transition correlations in a statistical manner. Then, we design spatial and temporal knowledge-embedded layers that introduce the multi-head cross-attention mechanism to fully explore the interaction between visual representation and the knowledge to generate spatial- and temporal-embedded representations, respectively. Finally, we aggregate these representations for each subject-object pair to predict the final semantic labels and their relationships. Extensive experiments show that STKET outperforms current competing algorithms by a large margin, e.g., improving the mR@50 by 8.1%, 4.7%, and 2.1% on different settings over current algorithms.Comment: Technical Repor

The causal association between smoking initiation, alcohol and coffee consumption, and women’s reproductive health: A two-sample Mendelian randomization analysis

Objective: A number of epidemiological studies have demonstrated that smoking initiation and alcohol and coffee consumption were closely related to women’s reproductive health. However, there was still insufficient evidence supporting their direct causality effect.Methods: We utilized two-sample Mendelian randomization (TSMR) analysis with summary datasets from genome-wide association study (GWAS) to investigate the causal relationship between smoking initiation, alcohol and coffee consumption, and women’s reproductive health-related traits. Exposure genetic instruments were used as variants significantly related to traits. The inverse-variance weighted (IVW) method was used as the main analysis approach, and we also performed MR-PRESSO, MR-Egger, weighted median, and weighted mode to supplement the sensitivity test. Then, the horizontal pleiotropy was detected by using MRE intercept and MR-PRESSO methods, and the heterogeneity was assessed using Cochran’s Q statistics.Results: We found evidence that smoking women showed a significant inverse causal association with the sex hormone-binding globulin (SHBG) levels (corrected β = −0.033, p = 9.05E-06) and age at menopause (corrected β = −0.477, p = 6.60E-09) and a potential positive correlation with the total testosterone (TT) levels (corrected β = 0.033, p = 1.01E-02). In addition, there was suggestive evidence for the alcohol drinking effect on the elevated TT levels (corrected β = 0.117, p = 5.93E-03) and earlier age at menopause (corrected β = −0.502, p = 4.14E-02) among women, while coffee consumption might decrease the female SHBG levels (corrected β = −0.034, p = 1.33E-03).Conclusion: Our findings suggested that smoking in women significantly decreased their SHBG concentration, promoted earlier menopause, and possibly reduced the TT levels. Alcohol drinking had a potential effect on female higher TT levels and earlier menopause, while coffee consumption might lead to lower female SHBG levels