Multiple Object Tracking with Correlation Filters and Deep Features

This thesis studies on-line multiple object tracking (MOT) problem which has been developed in numerous real-world applications, such as emerging self-driving car agents or estimating a target's trajectory over time to identify its movement pattern. The challenges that an on-line MOT tracker always faces are: (1) being able to consistently and smoothly track the same target over time with the presence of occlusions, (2) being able to recover from fragmented tracks, (3) handling identity switches of the same target, and (4) being able to operate in real-time. This work aims to provide an efficient detect-and-track framework to address these challenges. To narrow down the classes of objects to be studied, but without losing the tracker's extendibility to a generic object, we pick \textit{pedestrians} as the primary objects of interest. The proposed framework consists of four building blocks, i.e. object detection, object tracking, data association, and object re-identification. While most of the MOT frameworks make the assumption of the availability of the detector in every frame, the proposed MOT tracker operates with the detector being triggered only periodically, e.g. in every three frames, leading to improved efficiency. As for each building block, the detection is performed by Single Shot Detector (SSD), which has proven efficiency and efficacy on generic object classes. When the detector is triggered and active tracks exist, data association module identifies the correspondence of the objects detected by the detector and tracked by the tracker. In cases where newly detected objects cannot be identified as any of current tracks, the re-identification module then attempts to find the correspondence for them in the history track. The experiments show that the proposed framework is outperformed by the recently published on-line MOT trackers which are based on different object detectors. However, the results suggest that the proposed framework's performance does not degrade when the detector is partially unavailable and improves in certain conditions due to better temporal consistency. Based on these experiments, we are able to identify major shortcomings of the current framework, providing possible ways to improve it and directions for the future work

Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models

Predicting a scene graph that captures visual entities and their interactions in an image has been considered a crucial step towards full scene comprehension. Recent scene graph generation (SGG) models have shown their capability of capturing the most frequent relations among visual entities. However, the state-of-the-art results are still far from satisfactory, e.g. models can obtain 31% in overall recall R@100, whereas the likewise important mean class-wise recall mR@100 is only around 8% on Visual Genome (VG). The discrepancy between R and mR results urges to shift the focus from pursuing a high R to a high mR with a still competitive R. We suspect that the observed discrepancy stems from both the annotation bias and sparse annotations in VG, in which many visual entity pairs are either not annotated at all or only with a single relation when multiple ones could be valid. To address this particular issue, we propose a novel SGG training scheme that capitalizes on self-learned knowledge. It involves two relation classifiers, one offering a less biased setting for the other to base on. The proposed scheme can be applied to most of the existing SGG models and is straightforward to implement. We observe significant relative improvements in mR (between +6.6% and +20.4%) and competitive or better R (between -2.4% and 0.3%) across all standard SGG tasks.Comment: accepted to BMVC202

PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting

Vision-language (VL) Pre-training (VLP) has shown to well generalize VL models over a wide range of VL downstream tasks, especially for cross-modal retrieval. However, it hinges on a huge amount of image-text pairs, which requires tedious and costly curation. On the contrary, weakly-supervised VLP (W-VLP) explores means with object tags generated by a pre-trained object detector (OD) from images. Yet, they still require paired information, i.e. images and object-level annotations, as supervision to train an OD. To further reduce the amount of supervision, we propose Prompts-in-The-Loop (PiTL) that prompts knowledge from large language models (LLMs) to describe images. Concretely, given a category label of an image, e.g. refinery, the knowledge, e.g. a refinery could be seen with large storage tanks, pipework, and ..., extracted by LLMs is used as the language counterpart. The knowledge supplements, e.g. the common relations among entities most likely appearing in a scene. We create IN14K, a new VL dataset of 9M images and 1M descriptions of 14K categories from ImageNet21K with PiTL. Empirically, the VL models pre-trained with PiTL-generated pairs are strongly favored over other W-VLP works on image-to-text (I2T) and text-to-image (T2I) retrieval tasks, with less supervision. The results reveal the effectiveness of PiTL-generated pairs for VLP

Novel Mutation in Boy With Cartilage-hair Hypoplasia

BackgroundCartilage-hair hypoplasia (MIM 250250) is an autosomal recessive disease with diverse clinical manifestations. The clinical phenotypes include variable degrees of bone and hair dysplasia, deficient cellular and/or humoral immunity, and a predisposition to malignancy.MethodsWe performed genetic studies of a patient with disproportionate short stature and brittle scalp hair. Genetic studies were also carried out in the patient's parents.ResultsA novel maternal mutation that consisted of a duplication of 14 nucleotides at position −13 of the RNA component of the RNA component of mitochondrial RNA processing endoribonuclease gene (RMRP; g. −26 to −13 dupTACTACTCTGTGAA, promoter region) and a paternal mutation base substitution of C to T at nucleotide + 230 (designated as + 1 in the transcription initiation site) in the coding sequence of RMRP were detected in this patient.ConclusionA novel maternal RMRP mutation was found in a Chinese boy with typical cartilage-hair hypoplasia

Learning by Hallucinating: Vision-Language Pre-training with Weak Supervision

Weakly-supervised vision-language (V-L) pre-training (W-VLP) aims at learning cross-modal alignment with little or no paired data, such as aligned images and captions. Recent W-VLP methods, which pair visual features with object tags, help achieve performances comparable with some VLP models trained with aligned pairs in various V-L downstream tasks. This, however, is not the case in cross-modal retrieval (XMR). We argue that the learning of such a W-VLP model is curbed and biased by the object tags of limited semantics. We address the lack of paired V-L data for model supervision with a novel Visual Vocabulary based Feature Hallucinator (WFH), which is trained via weak supervision as a W-VLP model, not requiring images paired with captions. WFH generates visual hallucinations from texts, which are then paired with the originally unpaired texts, allowing more diverse interactions across modalities. Empirically, WFH consistently boosts the prior W-VLP works, e.g. U-VisualBERT (U-VB), over a variety of V-L tasks, i.e. XMR, Visual Question Answering, etc. Notably, benchmarked with recall@{1,5,10}, it consistently improves U-VB on image-to-text and text-to-image retrieval on two popular datasets Flickr30K and MSCOCO. Meanwhile, it gains by at least 14.5% in cross-dataset generalization tests on these XMR tasks. Moreover, in other V-L downstream tasks considered, our WFH models are on par with models trained with paired V-L data, revealing the utility of unpaired data. These results demonstrate greater generalization of the proposed W-VLP model with WFH.Comment: Accepted to WACV'23. Please find supplementary material at https://drive.google.com/file/d/1SmCBGsUgkYLAhmK83RZqY03bq4j3214p/view?usp=sharin

dbPTM: an information repository of protein post-translational modification

dbPTM is a database that compiles information on protein post-translational modifications (PTMs), such as the catalytic sites, solvent accessibility of amino acid residues, protein secondary and tertiary structures, protein domains and protein variations. The database includes all of the experimentally validated PTM sites from Swiss-Prot, PhosphoELM and O-GLYCBASE. Only a small fraction of Swiss-Prot proteins are annotated with experimentally verified PTM. Although the Swiss-Prot provides rich information about the PTM, other structural properties and functional information of proteins are also essential for elucidating protein mechanisms. The dbPTM systematically identifies three major types of protein PTM (phosphorylation, glycosylation and sulfation) sites against Swiss-Prot proteins by refining our previously developed prediction tool, KinasePhos (). Solvent accessibility and secondary structure of residues are also computationally predicted and are mapped to the PTM sites. The resource is now freely available at

Loss of stearoyl-CoA desaturase 2 disrupts inflammatory response in macrophages

Macrophages are innate immune cells that patrol tissues and are the first responders to detect infection. They orchestrate the host immune response in eliminating invading pathogens and the subsequent transition from inflammation to tissue repair. Macrophage dysfunction contributes to age-related pathologies, including low-grade inflammation in advanced age that is termed inflammaging. Our laboratory has previously identified that macrophage expression of a fatty acid desaturase, stearoyl-CoA desaturase 2 (SCD2), declines with age. Herein, we delineate the precise cellular effects of SCD2 deficiency in murine macrophages. We found that deletion o

The Relationship between Stasis-Stagnation Constitution and Peripheral Arterial Disease in Patients with Type 2 Diabetes

Objectives. In traditional Chinese medicine, Yu-Zhi (YZ, indicating stasis and stagnation) constitution describes a body that tends to express abnormal circulatory conditions. This study identified the linkage between YZ constitution and peripheral arterial disease (PAD) in patients with type 2 diabetes. Methods. Patients over 20 years of age who had had type 2 diabetes for 5 years or longer were recruited. PAD was diagnosed if the ankle-brachial index score was ≤0.9 in either leg. Level of YZ constitution was accessed by an YZ Constitution Questionnaire. Results. A total of 712 patients (354 men and 358 women) with a mean age of 61.5±10.6 years and diabetes duration of 13.1±6.7 years were recruited. The prevalence of PAD among our patients was 7.2%. Multivariate logistic regression revealed significant correlations between PAD and, respectively, YZ score, age, diabetes duration, current smoking, and hs-CRP. Conclusion. In addition to traditional risk factors, YZ constitution was statistically associated with PAD in patients with type 2 diabetes. This result invites further research into the effectiveness of traditional Chinese medicine to treat YZ constitution

Evaluation of an Epitypified Ophiocordyceps formosana

The substantial merit of Cordyceps s.l. spp. in terms of medicinal benefits is largely appreciated. Nevertheless, only few studies have characterized and examined the clinical complications of the use of health tonics containing these species. Here, we epitypified C. formosana isolates that were collected and characterized as Ophiocordyceps formosana based on morphological characteristics, molecular phylogenetic analyses, and metabolite profiling. Thus, we renamed and transferred C. formosana to the new protologue Ophiocordyceps formosana (Kobayasi & Shimizu) Wang, Tsai, Tzean & Shen comb. nov. Additionally, the pharmacological potential of O. formosana was evaluated based on the hot-water extract from its mycelium. The relative amounts of the known bioactive ingredients that are unique to Cordyceps s.l. species in O. formosana were found to be similar to the amounts in O. sinensis and C. militaris, indicating the potential applicability of O. formosana for pharmacological uses. Additionally, we found that O. formosana exhibited antioxidation activities in vitro and in vivo that were similar to those of O. sinensis and C. militaris. Furthermore, O. formosana also displayed conspicuously effective antitumor activity compared with the tested Cordyceps s.l. species. Intrinsically, O. formosana exhibited less toxicity than the other Cordyceps species. Together, our data suggest that the metabolites of O. formosana may play active roles in complementary medicine