169 research outputs found
Trading cookies with a random walk
The mathematical problem of determining a gambler’s risk of ruin involves analyzing decisions of only one agent, namely the “gambler”. In this work we consider an extension that introduces two additional players, so called “sellers”. These two new agents can boost the probability of success for the gambler by selling to him (using a jargon borrowed from the theory of excited random walks) a “cookie” which is used to increase the probability of moving forward in the next step. The generalized gambler’s ruin scenario considers an excited random walk on a finite interval of integer line with two “cookie store” locations and unlimited supply of cookies at each. Each time when the buyer (walker) visits a store location, he has an opportunity to decide whether he is willing to consume the cookie or not. We wish to determine the equilibrium prices and cookie store locations in a formal game associated with this generalized gambler’s ruin scenario
#DiminishingDiscrimination:The symbolic annihilation of race and racism in news hashtags of “calling 911 on black people"
This critical textual analysis examines the symbolic annihilation of race and racisms by the use of hashtags, such as #BBQBecky and #CornerstoreCaroline, in news coverage of White people in the United States who called or threatened to call 911 to report everyday behavior of Black individuals in 2018. This study argues that the use of humorous and virtual hashtags to represent the callers contributed to overall coverage of racially charged incidents in ways that reduced the events’ seriousness in terms of social policing of Black individuals. Collectively, this coverage formed a type of symbolic annihilation of racist interpretations of callers’ acts and Black resistance and meanings embedded within the hashtags themselves
Towards Omni-supervised Referring Expression Segmentation
Referring Expression Segmentation (RES) is an emerging task in computer
vision, which segments the target instances in images based on text
descriptions. However, its development is plagued by the expensive segmentation
labels. To address this issue, we propose a new learning task for RES called
Omni-supervised Referring Expression Segmentation (Omni-RES), which aims to
make full use of unlabeled, fully labeled and weakly labeled data, e.g.,
referring points or grounding boxes, for efficient RES training. To accomplish
this task, we also propose a novel yet strong baseline method for Omni-RES
based on the recently popular teacher-student learning, where the weak labels
are not directly transformed into supervision signals but used as a yardstick
to select and refine high-quality pseudo-masks for teacher-student learning. To
validate the proposed Omni-RES method, we apply it to a set of state-of-the-art
RES models and conduct extensive experiments on a bunch of RES datasets. The
experimental results yield the obvious merits of Omni-RES than the
fully-supervised and semi-supervised training schemes. For instance, with only
10% fully labeled data, Omni-RES can help the base model achieve 100% fully
supervised performance, and it also outperform the semi-supervised alternative
by a large margin, e.g., +14.93% on RefCOCO and +14.95% on RefCOCO+,
respectively. More importantly, Omni-RES also enable the use of large-scale
vision-langauges like Visual Genome to facilitate low-cost RES training, and
achieve new SOTA performance of RES, e.g., 80.66 on RefCOCO
Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
With ever increasing parameters and computation, vision-language pre-trained
(VLP) models exhibit prohibitive expenditure in downstream task adaption.
Recent endeavors mainly focus on parameter efficient transfer learning (PETL)
for VLP models by only updating a small number of parameters. However,
excessive computational overhead still plagues the application of VLPs. In this
paper, we aim at parameter and computation efficient transfer learning (PCETL)
for VLP models. In particular, PCETL not only needs to limit the number of
trainable parameters in VLP models, but also to reduce the computational
redundancy during inference, thus enabling a more efficient transfer. To
approach this target, we propose a novel dynamic architecture skipping (DAS)
approach towards effective PCETL. Instead of directly optimizing the intrinsic
architectures of VLP models, DAS first observes the significances of their
modules to downstream tasks via a reinforcement learning (RL) based process,
and then skips the redundant ones with lightweight networks, i.e., adapters,
according to the obtained rewards. In this case, the VLP model can well
maintain the scale of trainable parameters while speeding up its inference on
downstream tasks. To validate DAS, we apply it to two representative VLP
models, namely ViLT and METER, and conduct extensive experiments on a bunch of
VL tasks. The experimental results not only show the great advantages of DAS in
reducing computational complexity, e.g. -11.97% FLOPs of METER on VQA2.0, but
also confirm its competitiveness against existing PETL methods in terms of
parameter scale and performance. Our source code is given in our appendix
What Goes beyond Multi-modal Fusion in One-stage Referring Expression Comprehension: An Empirical Study
Most of the existing work in one-stage referring expression comprehension
(REC) mainly focuses on multi-modal fusion and reasoning, while the influence
of other factors in this task lacks in-depth exploration. To fill this gap, we
conduct an empirical study in this paper. Concretely, we first build a very
simple REC network called SimREC, and ablate 42 candidate designs/settings,
which covers the entire process of one-stage REC from network design to model
training. Afterwards, we conduct over 100 experimental trials on three
benchmark datasets of REC. The extensive experimental results not only show the
key factors that affect REC performance in addition to multi-modal fusion,
e.g., multi-scale features and data augmentation, but also yield some findings
that run counter to conventional understanding. For example, as a vision and
language (V&L) task, REC does is less impacted by language prior. In addition,
with a proper combination of these findings, we can improve the performance of
SimREC by a large margin, e.g., +27.12% on RefCOCO+, which outperforms all
existing REC methods. But the most encouraging finding is that with much less
training overhead and parameters, SimREC can still achieve better performance
than a set of large-scale pre-trained models, e.g., UNITER and VILLA,
portraying the special role of REC in existing V&L research
Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
Recently, growing interest has been aroused in extending the multimodal
capability of large language models (LLMs), e.g., vision-language (VL)
learning, which is regarded as the next milestone of artificial general
intelligence. However, existing solutions are prohibitively expensive, which
not only need to optimize excessive parameters, but also require another
large-scale pre-training before VL instruction tuning. In this paper, we
propose a novel and affordable solution for the effective VL adaption of LLMs,
called Mixture-of-Modality Adaptation (MMA). Instead of using large neural
networks to connect the image encoder and LLM, MMA adopts lightweight modules,
i.e., adapters, to bridge the gap between LLMs and VL tasks, which also enables
the joint optimization of the image and language models. Meanwhile, MMA is also
equipped with a routing algorithm to help LLMs achieve an automatic shift
between single- and multi-modal instructions without compromising their ability
of natural language understanding. To validate MMA, we apply it to a recent LLM
called LLaMA and term this formed large vision-language instructed model as
LaVIN. To validate MMA and LaVIN, we conduct extensive experiments under two
setups, namely multimodal science question answering and multimodal dialogue.
The experimental results not only demonstrate the competitive performance and
the superior training efficiency of LaVIN than existing multimodal LLMs, but
also confirm its great potential as a general-purpose chatbot. More
importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4
training hours with 3.8M trainable parameters, greatly confirming the
effectiveness of MMA. Our project is released at
https://luogen1996.github.io/lavin
Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches
The emerging trend of AR/VR places great demands on 3D content. However, most
existing software requires expertise and is difficult for novice users to use.
In this paper, we aim to create sketch-based modeling tools for user-friendly
3D modeling. We introduce Reality3DSketch with a novel application of an
immersive 3D modeling experience, in which a user can capture the surrounding
scene using a monocular RGB camera and can draw a single sketch of an object in
the real-time reconstructed 3D scene. A 3D object is generated and placed in
the desired location, enabled by our novel neural network with the input of a
single sketch. Our neural network can predict the pose of a drawing and can
turn a single sketch into a 3D model with view and structural awareness, which
addresses the challenge of sparse sketch input and view ambiguity. We conducted
extensive experiments synthetic and real-world datasets and achieved
state-of-the-art (SOTA) results in both sketch view estimation and 3D modeling
performance. According to our user study, our method of performing 3D modeling
in a scene is 5x faster than conventional methods. Users are also more
satisfied with the generated 3D model than the results of existing methods.Comment: IEEE Transactions on MultiMedi
PixelFolder: An Efficient Progressive Pixel Synthesis Network for Image Generation
Pixel synthesis is a promising research paradigm for image generation, which
can well exploit pixel-wise prior knowledge for generation. However, existing
methods still suffer from excessive memory footprint and computation overhead.
In this paper, we propose a progressive pixel synthesis network towards
efficient image generation, coined as PixelFolder. Specifically, PixelFolder
formulates image generation as a progressive pixel regression problem and
synthesizes images by a multi-stage paradigm, which can greatly reduce the
overhead caused by large tensor transformations. In addition, we introduce
novel pixel folding operations to further improve model efficiency while
maintaining pixel-wise prior knowledge for end-to-end regression. With these
innovative designs, we greatly reduce the expenditure of pixel synthesis, e.g.,
reducing 90% computation and 57% parameters compared to the latest pixel
synthesis method called CIPS. To validate our approach, we conduct extensive
experiments on two benchmark datasets, namely FFHQ and LSUN Church. The
experimental results show that with much less expenditure, PixelFolder obtains
new state-of-the-art (SOTA) performance on two benchmark datasets, i.e., 3.77
FID and 2.45 FID on FFHQ and LSUN Church, respectively. Meanwhile, PixelFolder
is also more efficient than the SOTA methods like StyleGAN2, reducing about 74%
computation and 36% parameters, respectively. These results greatly validate
the effectiveness of the proposed PixelFolder.Comment: 11 pages, 7 figure
- …