16,053 research outputs found
RACE: Large-scale ReAding Comprehension Dataset From Examinations
We present RACE, a new dataset for benchmark evaluation of methods in the
reading comprehension task. Collected from the English exams for middle and
high school Chinese students in the age range between 12 to 18, RACE consists
of near 28,000 passages and near 100,000 questions generated by human experts
(English instructors), and covers a variety of topics which are carefully
designed for evaluating the students' ability in understanding and reasoning.
In particular, the proportion of questions that requires reasoning is much
larger in RACE than that in other benchmark datasets for reading comprehension,
and there is a significant gap between the performance of the state-of-the-art
models (43%) and the ceiling human performance (95%). We hope this new dataset
can serve as a valuable resource for research and evaluation in machine
comprehension. The dataset is freely available at
http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at
https://github.com/qizhex/RACE_AR_baselines.Comment: EMNLP 201
Embodied Question Answering
We present a new AI task -- Embodied Question Answering (EmbodiedQA) -- where
an agent is spawned at a random location in a 3D environment and asked a
question ("What color is the car?"). In order to answer, the agent must first
intelligently navigate to explore the environment, gather information through
first-person (egocentric) vision, and then answer the question ("orange").
This challenging task requires a range of AI skills -- active perception,
language understanding, goal-driven navigation, commonsense reasoning, and
grounding of language into actions. In this work, we develop the environments,
end-to-end-trained reinforcement learning agents, and evaluation protocols for
EmbodiedQA.Comment: 20 pages, 13 figures, Webpage: https://embodiedqa.org
Using online linear classifiers to filter spam Emails
The performance of two online linear classifiers - the Perceptron and Littlestone’s Winnow – is explored for two anti-spam filtering benchmark corpora - PU1 and Ling-Spam. We study the performance for varying numbers of features, along with three different feature selection methods: Information Gain (IG), Document Frequency (DF) and Odds Ratio. The size of the training set and the number of training iterations are also investigated for both classifiers. The experimental results show that both the Perceptron and Winnow perform much better when using IG or DF than using Odds Ratio. It is further demonstrated that when using IG or DF, the classifiers are insensitive to the number of features and the number of training iterations, and not greatly sensitive to the size of training set. Winnow is shown to slightly outperform the Perceptron. It is also demonstrated that both of these online classifiers perform much better than a standard Naïve Bayes method. The theoretical and implementation computational complexity of these two classifiers are very low, and they are very easily adaptively updated. They outperform most of the published results, while being significantly easier to train and adapt. The analysis and promising experimental results indicate that the Perceptron and Winnow are two very competitive classifiers for anti-spam filtering
Exploring the Limitations of Behavior Cloning for Autonomous Driving
Driving requires reacting to a wide variety of complex environment conditions
and agent behaviors. Explicitly modeling each possible scenario is unrealistic.
In contrast, imitation learning can, in theory, leverage data from large fleets
of human-driven cars. Behavior cloning in particular has been successfully used
to learn simple visuomotor policies end-to-end, but scaling to the full
spectrum of driving behaviors remains an unsolved problem. In this paper, we
propose a new benchmark to experimentally investigate the scalability and
limitations of behavior cloning. We show that behavior cloning leads to
state-of-the-art results, including in unseen environments, executing complex
lateral and longitudinal maneuvers without these reactions being explicitly
programmed. However, we confirm well-known limitations (due to dataset bias and
overfitting), new generalization issues (due to dynamic objects and the lack of
a causal model), and training instability requiring further research before
behavior cloning can graduate to real-world driving. The code of the studied
behavior cloning approaches can be found at
https://github.com/felipecode/coiltraine
MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning
Recently, the astonishing performance of large language models (LLMs) in
natural language comprehension and generation tasks triggered lots of
exploration of using them as central controllers to build agent systems.
Multiple studies focus on bridging the LLMs to external tools to extend the
application scenarios. However, the current LLMs' perceiving tool-use ability
is limited to a single text query, which may result in ambiguity in
understanding the users' real intentions. LLMs are expected to eliminate that
by perceiving the visual- or auditory-grounded instructions' information.
Therefore, in this paper, we propose MLLM-Tool, a system incorporating
open-source LLMs and multi-modal encoders so that the learnt LLMs can be
conscious of multi-modal input instruction and then select the function-matched
tool correctly. To facilitate the evaluation of the model's capability, we
collect a dataset featured by consisting of multi-modal input tools from
HuggingFace. Another important feature of our dataset is that our dataset also
contains multiple potential choices for the same instruction due to the
existence of identical functions and synonymous functions, which provides more
potential solutions for the same query. The experiments reveal that our
MLLM-Tool is capable of recommending appropriate tools for multi-modal
instructions. Codes and data are available at
https://github.com/MLLM-Tool/MLLM-Tool.Comment: 21 pages, 9 figures, 10 table
- …