311 research outputs found
Visually Grounded Compound PCFGs
Exploiting visual groundings for language understanding has recently been
drawing much attention. In this work, we study visually grounded grammar
induction and learn a constituency parser from both unlabeled text and its
visual groundings. Existing work on this task (Shi et al., 2019) optimizes a
parser via Reinforce and derives the learning signal only from the alignment of
images and sentences. While their model is relatively accurate overall, its
error distribution is very uneven, with low performance on certain constituents
types (e.g., 26.2% recall on verb phrases, VPs) and high on others (e.g., 79.6%
recall on noun phrases, NPs). This is not surprising as the learning signal is
likely insufficient for deriving all aspects of phrase-structure syntax and
gradient estimates are noisy. We show that using an extension of probabilistic
context-free grammar model we can do fully-differentiable end-to-end visually
grounded learning. Additionally, this enables us to complement the image-text
alignment loss with a language modeling objective. On the MSCOCO test captions,
our model establishes a new state of the art, outperforming its non-grounded
version and, thus, confirming the effectiveness of visual groundings in
constituency grammar induction. It also substantially outperforms the previous
grounded model, with largest improvements on more `abstract' categories (e.g.,
+55.1% recall on VPs).Comment: Accepted to EMNLP 2020. Our code is available at
https://github.com/zhaoyanpeng/vpcf
An Empirical Study of Compound PCFGs
Compound probabilistic context-free grammars (C-PCFGs) have recently
established a new state of the art for phrase-structure grammar induction.
However, due to the high time-complexity of chart-based representation and
inference, it is difficult to investigate them comprehensively. In this work,
we rely on a fast implementation of C-PCFGs to conduct evaluation complementary
to that of~\citet{kim-etal-2019-compound}. We highlight three key findings: (1)
C-PCFGs are data-efficient, (2) C-PCFGs make the best use of global
sentence-level information in preterminal rule probabilities, and (3) the best
configurations of C-PCFGs on English do not always generalize to
morphology-rich languages.Comment: Accepted to Adapt-NLP at EACL 2021. Our code is available at
https://github.com/zhaoyanpeng/cpcf
On the Transferability of Visually Grounded PCFGs
There has been a significant surge of interest in visually grounded grammar
induction in recent times. While a variety of models have been developed for
the task and have demonstrated impressive performance, they have not been
evaluated on text domains that are different from the training domain, so it is
unclear if the improvements brought by visual groundings are transferable. Our
study aims to fill this gap and assess the degree of transferability. We start
by extending VC-PCFG (short for Visually-grounded Compound
PCFG~\citep{zhao-titov-2020-visually}) in such a way that it can transfer
across text domains. We consider a zero-shot transfer learning setting where a
model is trained on the source domain and is directly applied to target
domains, without any further training. Our experimental results suggest that:
the benefits from using visual groundings transfer to text in a domain similar
to the training domain but fail to transfer to remote domains. Further, we
conduct data and result analysis; we find that the lexicon overlap between the
source domain and the target domain is the most important factor in the
transferability of VC-PCFG.Comment: Accepted to EMNLP Findings 2023. Our code is available at
https://github.com/zhaoyanpeng/cpcf
Unsupervised structure induction and multimodal grounding
Structured representations build upon symbolic abstraction (e.g., words in natural language and visual concepts in natural images), offer a principled way of encoding our perceptions about the physical world, and enable the human-like generalization of machine learning systems. The predominant paradigm for learning structured representations of the observed data has been supervised learning, but it is limited in several respects. First, supervised learning is challenging given the scarcity of labeled data. Second, conventional approaches to structured prediction have been relying on a single modality (e.g., either images or text), ignoring the learning cues that may have been specified in and can be readily obtained from other modalities of data. In this thesis, we investigate unsupervised approaches to structure induction in a multimodal setting.
Unsupervised learning is inherently difficult in general, let alone inducing complex and discrete structures from data without direct supervision. By considering the multimodal setting, we leverage the alignments between different data modalities (e.g., text, audio, and images) to facilitate the learning of structure-induction models, e.g., knowing that the individual words in ``a white pigeon'' always appear with the same visual object, a language parser is likely to treat them as a whole (i.e., phrase). The multimodal learning setting is practically viable because multimodal alignments are generally abundant. For example, they can be found in online posts such as news and tweets that usually contain images and associated text, and in (YouTube) videos, where audio, scripts, and scenes are synchronized and grounded in each other.
We develop structure-induction models, which are capable of exploiting bimodal image-text alignments, for two modalities: (1) for natural language, we consider unsupervised syntactic parsing with phrase-structure grammars and regularize the parser by using visual image groundings; and (2) for visual images, we induce scene graph representations by mapping arguments and predicates in the text to their visual counterparts (i.e., visual objects and relations among them) in an unsupervised manner. While useful, crossmodal alignments are not always abundantly available on the web, e.g., the alignments between non-speech audio and text. We tackle the challenge by sharing the visual modality between image-text alignment and image-audio alignment; images function as a pivot and connect audio and text. The contributions of this thesis span from model development to data collection. We demonstrated the feasibility of applying multimodal learning techniques to unsupervised structure induction and multimodal alignment collection. Our work opens up new avenues for multimodal and unsupervised structured representation learning
Unsupervised Object-Centric Voxelization for Dynamic Scene Understanding
Understanding the compositional dynamics of multiple objects in unsupervised
visual environments is challenging, and existing object-centric representation
learning methods often ignore 3D consistency in scene decomposition. We propose
DynaVol, an inverse graphics approach that learns object-centric volumetric
representations in a neural rendering framework. DynaVol maintains time-varying
3D voxel grids that explicitly represent the probability of each spatial
location belonging to different objects, and decouple temporal dynamics and
spatial information by learning a canonical-space deformation field. To
optimize the volumetric features, we embed them into a fully differentiable
neural network, binding them to object-centric global features and then driving
a compositional NeRF for scene reconstruction. DynaVol outperforms existing
methods in novel view synthesis and unsupervised scene decomposition and allows
for the editing of dynamic scenes, such as adding, deleting, replacing objects,
and modifying their trajectories
Is ChatGPT Equipped with Emotional Dialogue Capabilities?
This report presents a study on the emotional dialogue capability of ChatGPT,
an advanced language model developed by OpenAI. The study evaluates the
performance of ChatGPT on emotional dialogue understanding and generation
through a series of experiments on several downstream tasks. Our findings
indicate that while ChatGPT's performance on emotional dialogue understanding
may still lag behind that of supervised models, it exhibits promising results
in generating emotional responses. Furthermore, the study suggests potential
avenues for future research directions
Transcriptome, microRNA, and degradome analyses of the gene expression of Paulownia with phytoplamsa
Primers of P. tomentosa miRNAs for qRT-PCR analysis. (DOCX 20.7 kb
- …