30 research outputs found
Competitive Spectrum Management with Incomplete Information
This paper studies an interference interaction (game) between selfish and
independent wireless communication systems in the same frequency band. Each
system (player) has incomplete information about the other player's channel
conditions. A trivial Nash equilibrium point in this game is where players
mutually full spread (FS) their transmit spectrum and interfere with each
other. This point may lead to poor spectrum utilization from a global network
point of view and even for each user individually.
In this paper, we provide a closed form expression for a non pure-FS
epsilon-Nash equilibrium point; i.e., an equilibrium point where players choose
FDM for some channel realizations and FS for the others. We show that operating
in this non pure-FS epsilon-Nash equilibrium point increases each user's
throughput and therefore improves the spectrum utilization, and demonstrate
that this performance gain can be substantial. Finally, important insights are
provided into the behaviour of selfish and rational wireless users as a
function of the channel parameters such as fading probabilities, the
interference-to-signal ratio
Knowledge is a Region in Weight Space for Fine-tuned Language Models
Research on neural networks has focused on understanding a single model
trained on a single dataset. However, relatively little is known about the
relationships between different models, particularly those trained or tested on
different datasets. We address this by studying how the weight space and the
underlying loss landscape of different models are interconnected.
Specifically, we demonstrate that finetuned models that were optimized for
high performance, reside in well-defined regions in weight space, and vice
versa -- that any model that resides anywhere in those regions also exhibits
high performance. Notably, we show that language models that have been
finetuned on the same dataset form a tight cluster in the weight space, while
models finetuned on different datasets from the same underlying task form a
looser cluster. Moreover, traversing around the region between the models leads
to new models that perform comparably or even better than models obtained via
finetuning, even on tasks that the original models were not finetuned on.
Our findings provide insight into the relationships between models,
demonstrating that a model positioned between two similar models can acquire
the knowledge of both. We leverage this and design a method for selecting a
better model for efficient finetuning. Specifically, we show that starting from
the center of the region is as effective, if not more, than using the
pretrained model in 11 out of 12 datasets, resulting in an average accuracy
improvement of 3.06
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
We propose a new paradigm to continually evolve pretrained models, denoted
ColD Fusion. It provides the benefits of multitask learning but leverages
distributed computation with limited communication and eliminates the need for
shared data. Consequentially, ColD Fusion can give rise to a synergistic loop,
where finetuned models can be recycled to continually improve the pretrained
model they are based upon. We show that ColD Fusion yields comparable benefits
to multitask training by producing a model that (a) attains strong performance
on all of the datasets it was trained on; and (b) is a better starting point
for finetuning on unseen datasets. We show that ColD Fusion outperforms RoBERTa
and even previous multitask models. Specifically, when training and testing on
35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.33 points
on average without any changes to the architecture.Comment: ACL 2
Efficient Benchmarking (of Language Models)
The increasing versatility of language models LMs has given rise to a new
class of benchmarks that comprehensively assess a broad range of capabilities.
Such benchmarks are associated with massive computational costs reaching
thousands of GPU hours per model. However the efficiency aspect of these
evaluation efforts had raised little discussion in the literature. In this work
we present the problem of Efficient Benchmarking namely intelligently reducing
the computation costs of LM evaluation without compromising reliability. Using
the HELM benchmark as a test case we investigate how different benchmark design
choices affect the computation-reliability tradeoff. We propose to evaluate the
reliability of such decisions by using a new measure Decision Impact on
Reliability DIoR for short. We find for example that the current leader on HELM
may change by merely removing a low-ranked model from the benchmark and observe
that a handful of examples suffice to obtain the correct benchmark ranking.
Conversely a slightly different choice of HELM scenarios varies ranking widely.
Based on our findings we outline a set of concrete recommendations for more
efficient benchmark design and utilization practices leading to dramatic cost
savings with minimal loss of benchmark reliability often reducing computation
by x100 or more
Corpus Wide Argument Mining -- a Working Solution
One of the main tasks in argument mining is the retrieval of argumentative
content pertaining to a given topic. Most previous work addressed this task by
retrieving a relatively small number of relevant documents as the initial
source for such content. This line of research yielded moderate success, which
is of limited use in a real-world system. Furthermore, for such a system to
yield a comprehensive set of relevant arguments, over a wide range of topics,
it requires leveraging a large and diverse corpus in an appropriate manner.
Here we present a first end-to-end high-precision, corpus-wide argument mining
system. This is made possible by combining sentence-level queries over an
appropriate indexing of a very large corpus of newspaper articles, with an
iterative annotation scheme. This scheme addresses the inherent label bias in
the data and pinpoints the regions of the sample space whose manual labeling is
required to obtain high-precision among top-ranked candidates
Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours
Text classification can be useful in many real-world scenarios, saving a lot
of time for end users. However, building a custom classifier typically requires
coding skills and ML knowledge, which poses a significant barrier for many
potential users. To lift this barrier, we introduce Label Sleuth, a free open
source system for labeling and creating text classifiers. This system is unique
for (a) being a no-code system, making NLP accessible to non-experts, (b)
guiding users through the entire labeling process until they obtain a custom
classifier, making the process efficient -- from cold start to classifier in a
few hours, and (c) being open for configuration and extension by developers. By
open sourcing Label Sleuth we hope to build a community of users and developers
that will broaden the utilization of NLP models.Comment: 7 pages, 2 figure