6 research outputs found
ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oracles
Automatically generating textual content with desired attributes is an
ambitious task that people have pursued long. Existing works have made a series
of progress in incorporating unimodal controls into language models (LMs),
whereas how to generate controllable sentences with multimodal signals and high
efficiency remains an open question. To tackle the puzzle, we propose a new
paradigm of zero-shot controllable text generation with multimodal signals
(\textsc{ZeroGen}). Specifically, \textsc{ZeroGen} leverages controls of text
and image successively from token-level to sentence-level and maps them into a
unified probability space at decoding, which customizes the LM outputs by
weighted addition without extra training. To achieve better inter-modal
trade-offs, we further introduce an effective dynamic weighting mechanism to
regulate all control weights. Moreover, we conduct substantial experiments to
probe the relationship of being in-depth or in-width between signals from
distinct modalities. Encouraging empirical results on three downstream tasks
show that \textsc{ZeroGen} not only outperforms its counterparts on captioning
tasks by a large margin but also shows great potential in multimodal news
generation with a higher degree of control. Our code will be released at
https://github.com/ImKeTT/ZeroGen.Comment: 17 pages, preprin
Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics
Multi-modal large language models (MLLMs) are trained based on large language
models (LLM), with an enhanced capability to comprehend multi-modal inputs and
generate textual responses. While they excel in multi-modal tasks, the pure NLP
abilities of MLLMs are often underestimated and left untested. In this study,
we get out of the box and unveil an intriguing characteristic of MLLMs -- our
preliminary results suggest that visual instruction tuning, a prevailing
strategy for transitioning LLMs into MLLMs, unexpectedly and interestingly
helps models attain both improved truthfulness and ethical alignment in the
pure NLP context. For example, a visual-instruction-tuned LLaMA2 7B model
surpasses the performance of the LLaMA2-chat 7B model, fine-tuned with over one
million human annotations, on TruthfulQA-mc and Ethics benchmarks. Further
analysis reveals that the improved alignment can be attributed to the superior
instruction quality inherent to visual-text data. In releasing our code at
github.com/UCSC-VLAA/Sight-Beyond-Text, we aspire to foster further exploration
into the intrinsic value of visual-text synergies and, in a broader scope,
multi-modal interactions in alignment research
ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue
Incorporating visual knowledge into text-only dialogue systems has become a
potential direction to imitate the way humans think, imagine, and communicate.
However, existing multimodal dialogue systems are either confined by the scale
and quality of available datasets or the coarse concept of visual knowledge. To
address these issues, we provide a new paradigm of constructing multimodal
dialogues as well as two datasets extended from text-only dialogues under such
paradigm (ReSee-WoW, ReSee-DD). We propose to explicitly split the visual
knowledge into finer granularity (``turn-level'' and ``entity-level''). To
further boost the accuracy and diversity of augmented visual information, we
retrieve them from the Internet or a large image dataset. To demonstrate the
superiority and universality of the provided visual knowledge, we propose a
simple but effective framework ReSee to add visual representation into vanilla
dialogue models by modality concatenations. We also conduct extensive
experiments and ablations w.r.t. different model configurations and visual
knowledge settings. Empirical, encouraging results not only demonstrate the
effectiveness of introducing visual knowledge at both entity and turn level but
also verify the proposed model ReSee outperforms several state-of-the-art
methods on automatic and human evaluations. By leveraging text and vision
knowledge, ReSee can produce informative responses with real-world visual
concepts.Comment: 15 pages, preprin
PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation
Controllable text generation has taken a gigantic step forward these days.
Yet existing methods are either constrained in a one-off pattern or not
efficient enough for receiving multiple conditions at every generation stage.
We propose a model-agnostic framework Plug-in Conditional Auto-Encoder for
Controllable Text Generation (PCAE) towards flexible and semi-supervised text
generation. Our framework is "plug-and-play" with partial parameters to be
fine-tuned in the pre-trained model (less than a half). Crucial to the success
of PCAE is the proposed broadcasting label fusion network for navigating the
global latent code to a specified local and confined space. Visualization of
the local latent prior well confirms the primary devotion in hidden space of
the proposed model. Moreover, extensive experiments across five related
generation tasks (from 2 conditions up to 10 conditions) on both RNN- based and
pre-trained BART [26] based auto-encoders reveal the high capability of PCAE,
which enables generation that is highly manipulable, syntactically diverse and
time-saving with minimum labeled samples. We will release our code at
https://github.com/ImKeTT/pcae.Comment: Knowledge-Based System
AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling
Variational Auto-Encoder (VAE) has become the de-facto learning paradigm in
achieving representation learning and generation for natural language at the
same time. Nevertheless, existing VAE-based language models either employ
elementary RNNs, which is not powerful to handle complex works in the
multi-task situation, or fine-tunes two pre-trained language models (PLMs) for
any downstream task, which is a huge drain on resources. In this paper, we
propose the first VAE framework empowered with adaptive GPT-2s (AdaVAE).
Different from existing systems, we unify both the encoder\&decoder of the VAE
model using GPT-2s with adaptive parameter-efficient components, and further
introduce Latent Attention operation to better construct latent space from
transformer models. Experiments from multiple dimensions validate that AdaVAE
is competent to effectively organize language in three related tasks (language
modeling, representation modeling and guided text generation) even with less
than activated parameters in training. Our code is available at
\url{https://github.com/ImKeTT/AdaVAE}
How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs
This work focuses on the potential of Vision LLMs (VLLMs) in visual
reasoning. Different from prior studies, we shift our focus from evaluating
standard performance to introducing a comprehensive safety evaluation suite,
covering both out-of-distribution (OOD) generalization and adversarial
robustness. For the OOD evaluation, we present two novel VQA datasets, each
with one variant, designed to test model performance under challenging
conditions. In exploring adversarial robustness, we propose a straightforward
attack strategy for misleading VLLMs to produce visual-unrelated responses.
Moreover, we assess the efficacy of two jailbreaking strategies, targeting
either the vision or language component of VLLMs. Our evaluation of 21 diverse
models, ranging from open-source VLLMs to GPT-4V, yields interesting
observations: 1) Current VLLMs struggle with OOD texts but not images, unless
the visual information is limited; and 2) These VLLMs can be easily misled by
deceiving vision encoders only, and their vision-language training often
compromise safety protocols. We release this safety evaluation suite at
https://github.com/UCSC-VLAA/vllm-safety-benchmark.Comment: H.T., C.C., and Z.W. contribute equally. Work done during H.T. and
Z.W.'s internship at UCSC, and C.C. and Y.Z.'s internship at UN