19 research outputs found
Generating Executable Action Plans with Environmentally-Aware Language Models
Large Language Models (LLMs) trained using massive text datasets have
recently shown promise in generating action plans for robotic agents from high
level text queries. However, these models typically do not consider the robot's
environment, resulting in generated plans that may not actually be executable,
due to ambiguities in the planned actions or environmental constraints. In this
paper, we propose an approach to generate environmentally-aware action plans
that agents are better able to execute. Our approach involves integrating
environmental objects and object relations as additional inputs into LLM action
plan generation to provide the system with an awareness of its surroundings,
resulting in plans where each generated action is mapped to objects present in
the scene. We also design a novel scoring function that, along with generating
the action steps and associating them with objects, helps the system
disambiguate among object instances and take into account their states. We
evaluated our approach using the VirtualHome simulator and the ActivityPrograms
knowledge base and found that action plans generated from our system had a 310%
improvement in executability and a 147% improvement in correctness over prior
work. The complete code and a demo of our method is publicly available at
https://github.com/hri-ironlab/scene_aware_language_planner
Virtual-to-Real-World Transfer Learning for Robots on Wilderness Trails
Robots hold promise in many scenarios involving outdoor use, such as
search-and-rescue, wildlife management, and collecting data to improve
environment, climate, and weather forecasting. However, autonomous navigation
of outdoor trails remains a challenging problem. Recent work has sought to
address this issue using deep learning. Although this approach has achieved
state-of-the-art results, the deep learning paradigm may be limited due to a
reliance on large amounts of annotated training data. Collecting and curating
training datasets may not be feasible or practical in many situations,
especially as trail conditions may change due to seasonal weather variations,
storms, and natural erosion. In this paper, we explore an approach to address
this issue through virtual-to-real-world transfer learning using a variety of
deep learning models trained to classify the direction of a trail in an image.
Our approach utilizes synthetic data gathered from virtual environments for
model training, bypassing the need to collect a large amount of real images of
the outdoors. We validate our approach in three main ways. First, we
demonstrate that our models achieve classification accuracies upwards of 95% on
our synthetic data set. Next, we utilize our classification models in the
control system of a simulated robot to demonstrate feasibility. Finally, we
evaluate our models on real-world trail data and demonstrate the potential of
virtual-to-real-world transfer learning.Comment: iROS 201
Extrinisic Calibration of a Camera-Arm System Through Rotation Identification
Determining extrinsic calibration parameters is a necessity in any robotic
system composed of actuators and cameras. Once a system is outside the lab
environment, parameters must be determined without relying on outside artifacts
such as calibration targets. We propose a method that relies on structured
motion of an observed arm to recover extrinsic calibration parameters. Our
method combines known arm kinematics with observations of conics in the image
plane to calculate maximum-likelihood estimates for calibration extrinsics.
This method is validated in simulation and tested against a real-world model,
yielding results consistent with ruler-based estimates. Our method shows
promise for estimating the pose of a camera relative to an articulated arm's
end effector without requiring tedious measurements or external artifacts.
Index Terms: robotics, hand-eye problem, self-calibration, structure from
motio
Non-Invasive BCI through EEG
Thesis advisor: Robert SignorileIt has long been known that as neurons fire within the brain they produce measurable electrical activity. Electroencephalography (EEG) is the measurement and recording of these electrical signals using sensors arrayed across the scalp. Though there is copious research in using EEG technology in the fields of neuroscience and cognitive psychology, it is only recently that the possibility of utilizing EEG measurements as inputs in the control of computers has emerged. The idea of Brain-Computer Interfaces (BCIs) which allow the control of devices using brain signals evolved from the realm of science fiction to simple devices that currently exist. BCIs naturally present themselves to many extremely useful applications including prosthetic devices, restoring or aiding in communication and hearing, military applications, video gaming and virtual reality, and robotic control, and have the possibility of significantly improving the quality of life of many disabled individuals. However, current BCIs suffer from many problems including inaccuracies, delays between thought, detection, and action, exorbitant costs, and invasive surgeries. The purpose of this research is to examine the Emotiv EPOC© System as a cost-effective gateway to non-invasive portable EEG measurements and utilize it to build a thought-based BCI to control the Parallax Scribbler® robot. This research furthers the analysis of the current pros and cons of EEG technology as it pertains to BCIs and offers a glimpse of the future potential capabilities of BCI systems.Thesis (BA) — Boston College, 2010.Submitted to: Boston College. College of Arts and Sciences.Discipline: Computer Science Honors Program.Discipline: Computer Science
TOBY: A Tool for Exploring Data in Academic Survey Papers
This paper describes TOBY, a visualization tool that helps a user explore the
contents of an academic survey paper. The visualization consists of four
components: a hierarchical view of taxonomic data in the survey, a document
similarity view in the space of taxonomic classes, a network view of citations,
and a new paper recommendation tool. In this paper, we will discuss these
features in the context of three separate deployments of the tool
Recommended from our members
ARC-LfD: Using Augmented Reality for Interactive Long-Term Robot Skill Maintenance via Constrained Learning from Demonstration
Learning from Demonstration (LfD) enables novice users to teach robots new skills. However, many LfD methods do not facilitate skill maintenance and adaptation. Changes in task requirements or in the environment often reveal the lack of resiliency and adaptability in the skill model. To overcome these limitations, we introduce ARC-LfD: an Augmented Reality (AR) interface for constrained Learning from Demonstration that allows users to maintain, update, and adapt learned skills. This is accomplished through in-situ visualizations of learned skills and constraint-based editing of existing skills without requiring further demonstration. We describe the existing algorithmic basis for this system as well as our Augmented Reality interface and the novel capabilities it provides. Finally, we provide three case studies that demonstrate how ARC-LfD enables users to adapt to changes in the environment or task which require a skill to be altered after initial teaching has taken place.</p