LSTM Pose Machines

We observed that recent state-of-the-art results on single image human pose estimation were achieved by multi-stage Convolution Neural Networks (CNN). Notwithstanding the superior performance on static images, the application of these models on videos is not only computationally intensive, it also suffers from performance degeneration and flicking. Such suboptimal results are mainly attributed to the inability of imposing sequential geometric consistency, handling severe image quality degradation (e.g. motion blur and occlusion) as well as the inability of capturing the temporal correlation among video frames. In this paper, we proposed a novel recurrent network to tackle these problems. We showed that if we were to impose the weight sharing scheme to the multi-stage CNN, it could be re-written as a Recurrent Neural Network (RNN). This property decouples the relationship among multiple network stages and results in significantly faster speed in invoking the network for videos. It also enables the adoption of Long Short-Term Memory (LSTM) units between video frames. We found such memory augmented RNN is very effective in imposing geometric consistency among frames. It also well handles input quality degradation in videos while successfully stabilizes the sequential outputs. The experiments showed that our approach significantly outperformed current state-of-the-art methods on two large-scale video pose estimation benchmarks. We also explored the memory cells inside the LSTM and provided insights on why such mechanism would benefit the prediction for video-based pose estimations.Comment: Poster in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 201

Nucleosome Positioning and Its Role in Gene Regulation in Yeast

Nucleosome, composed of a 147-bp segment of DNA helix wrapped around a histone protein octamer, serves as the basic unit of chromatin. Nucleosome positioning refers to the relative position of DNA double helix with respect to the histone octamer. The positioning has an important role in transcription, DNA replication and other DNA transactions since packing DNA into nucleosomes occludes the binding site of proteins. Moreover, the nucleosomes bear histone modifications thus having a profound effect in regulation. Nucleosome positioning and its roles are extensively studied in model organism yeast. In this chapter, nucleosome organization and its roles in gene regulation are reviewed. Typically, nucleosomes are depleted around transcription start sites (TSSs), resulting in a nucleosome-free region (NFR) that is flanked by two well-positioned H2A.Z-containing nucleosomes. The nucleosomes downstream of the TSS are equally spaced in a nucleosome array. DNA sequences, especially 10–11 bp periodicities of some specific dinucleotides, partly determine the nucleosome positioning. Nucleosome occupancy can be determined with high throughput sequencing techniques. Importantly, nucleosome positions are dynamic in different cell types and different environments. Histones depletions, histones mutations, heat shock and changes in carbon source will profoundly change nucleosome organization. In the yeast cells, upon mutating the histones, the nucleosomes change drastically at promoters and the highly expressed genes, such as ribosome genes, undergo more change. The changes of nucleosomes tightly associate the transcription initiation, elongation and termination. H2A.Z is contained in the +1 and −1 nucleosomes and thus in transcription. Chaperon Chz1 and elongation factor Spt16 function in H2A.Z deposition on chromatin. The chapter covers the basic concept of nucleosomes, nucleosome determinant, the techniques of mapping nucleosomes, nucleosome alteration upon stress and mutation, and Htz1 dynamics on chromatin

Alternative Telescopic Displacement: An Efficient Multimodal Alignment Method

Feature alignment is the primary means of fusing multimodal data. We propose a feature alignment method that fully fuses multimodal information, which alternately shifts and expands feature information from different modalities to have a consistent representation in a feature space. The proposed method can robustly capture high-level interactions between features of different modalities, thus significantly improving the performance of multimodal learning. We also show that the proposed method outperforms other popular multimodal schemes on multiple tasks. Experimental evaluation of ETT and MIT-BIH-Arrhythmia, datasets shows that the proposed method achieves state of the art performance.Comment: 8 pages,7 figure

Enhancing Deep Knowledge Tracing with Auxiliary Tasks

Knowledge tracing (KT) is the problem of predicting students' future performance based on their historical interactions with intelligent tutoring systems. Recent studies have applied multiple types of deep neural networks to solve the KT problem. However, there are two important factors in real-world educational data that are not well represented. First, most existing works augment input representations with the co-occurrence matrix of questions and knowledge components\footnote{\label{ft:kc}A KC is a generalization of everyday terms like concept, principle, fact, or skill.} (KCs) but fail to explicitly integrate such intrinsic relations into the final response prediction task. Second, the individualized historical performance of students has not been well captured. In this paper, we proposed \emph{AT-DKT} to improve the prediction performance of the original deep knowledge tracing model with two auxiliary learning tasks, i.e., \emph{question tagging (QT) prediction task} and \emph{individualized prior knowledge (IK) prediction task}. Specifically, the QT task helps learn better question representations by predicting whether questions contain specific KCs. The IK task captures students' global historical performance by progressively predicting student-level prior knowledge that is hidden in students' historical learning interactions. We conduct comprehensive experiments on three real-world educational datasets and compare the proposed approach to both deep sequential KT models and non-sequential models. Experimental results show that \emph{AT-DKT} outperforms all sequential models with more than 0.9\% improvements of AUC for all datasets, and is almost the second best compared to non-sequential models. Furthermore, we conduct both ablation studies and quantitative analysis to show the effectiveness of auxiliary tasks and the superior prediction outcomes of \emph{AT-DKT}.Comment: Accepted at WWW'23: The 2023 ACM Web Conference, 202

Online Map Vectorization for Autonomous Driving: A Rasterization Perspective

Vectorized high-definition (HD) map is essential for autonomous driving, providing detailed and precise environmental information for advanced perception and planning. However, current map vectorization methods often exhibit deviations, and the existing evaluation metric for map vectorization lacks sufficient sensitivity to detect these deviations. To address these limitations, we propose integrating the philosophy of rasterization into map vectorization. Specifically, we introduce a new rasterization-based evaluation metric, which has superior sensitivity and is better suited to real-world autonomous driving scenarios. Furthermore, we propose MapVR (Map Vectorization via Rasterization), a novel framework that applies differentiable rasterization to vectorized outputs and then performs precise and geometry-aware supervision on rasterized HD maps. Notably, MapVR designs tailored rasterization strategies for various geometric shapes, enabling effective adaptation to a wide range of map elements. Experiments show that incorporating rasterization into map vectorization greatly enhances performance with no extra computational cost during inference, leading to more accurate map perception and ultimately promoting safer autonomous driving.Comment: [NeurIPS 2023