Robobarista: Learning to Manipulate Novel Objects via Deep Multimodal Embedding

There is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors, and so on. It is challenging for a roboticist to program a robot for each of these object types and for each of their instantiations. In this work, we present a novel approach to manipulation planning based on the idea that many household objects share similarly-operated object parts. We formulate the manipulation planning as a structured prediction problem and learn to transfer manipulation strategy across different objects by embedding point-cloud, natural language, and manipulation trajectory data into a shared embedding space using a deep neural network. In order to learn semantically meaningful spaces throughout our network, we introduce a method for pre-training its lower layers for multimodal feature embedding and a method for fine-tuning this embedding space using a loss-based margin. In order to collect a large number of manipulation demonstrations for different objects, we develop a new crowd-sourcing platform called Robobarista. We test our model on our dataset consisting of 116 objects and appliances with 249 parts along with 250 language instructions, for which there are 1225 crowd-sourced manipulation demonstrations. We further show that our robot with our model can even prepare a cup of a latte with appliances it has never seen before.Comment: Journal Versio

Robobarista: Object Part based Transfer of Manipulation Trajectories from Crowd-sourcing in 3D Pointclouds

There is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors, and so on. It is challenging for a roboticist to program a robot for each of these object types and for each of their instantiations. In this work, we present a novel approach to manipulation planning based on the idea that many household objects share similarly-operated object parts. We formulate the manipulation planning as a structured prediction problem and design a deep learning model that can handle large noise in the manipulation demonstrations and learns features from three different modalities: point-clouds, language and trajectory. In order to collect a large number of manipulation demonstrations for different objects, we developed a new crowd-sourcing platform called Robobarista. We test our model on our dataset consisting of 116 objects with 249 parts along with 250 language instructions, for which there are 1225 crowd-sourced manipulation demonstrations. We further show that our robot can even manipulate objects it has never seen before.Comment: In International Symposium on Robotics Research (ISRR) 201

Data-based prediction and causality inference of nonlinear dynamics

Natural systems are typically nonlinear and complex, and it is of great interest to be able to reconstruct a system in order to understand its mechanism, which can not only recover nonlinear behaviors but also predict future dynamics. Due to the advances of modern technology, big data becomes increasingly accessible and consequently the problem of reconstructing systems from measured data or time series plays a central role in many scientific disciplines. In recent decades, nonlinear methods rooted in state space reconstruction have been developed, and they do not assume any model equations but can recover the dynamics purely from the measured time series data. In this review, the development of state space reconstruction techniques will be introduced and the recent advances in systems prediction and causality inference using state space reconstruction will be presented. Particularly, the cutting-edge method to deal with short-term time series data will be focused. Finally, the advantages as well as the remaining problems in this field are discussed.Comment: 19 pages, 9 figures, it has been accepted for publication in SCIENCE CHINA Mathematic

An ASP-Based Architecture for Autonomous UAVs in Dynamic Environments: Progress Report

Traditional AI reasoning techniques have been used successfully in many domains, including logistics, scheduling and game playing. This paper is part of a project aimed at investigating how such techniques can be extended to coordinate teams of unmanned aerial vehicles (UAVs) in dynamic environments. Specifically challenging are real-world environments where UAVs and other network-enabled devices must communicate to coordinate---and communication actions are neither reliable nor free. Such network-centric environments are common in military, public safety and commercial applications, yet most research (even multi-agent planning) usually takes communications among distributed agents as a given. We address this challenge by developing an agent architecture and reasoning algorithms based on Answer Set Programming (ASP). ASP has been chosen for this task because it enables high flexibility of representation, both of knowledge and of reasoning tasks. Although ASP has been used successfully in a number of applications, and ASP-based architectures have been studied for about a decade, to the best of our knowledge this is the first practical application of a complete ASP-based agent architecture. It is also the first practical application of ASP involving a combination of centralized reasoning, decentralized reasoning, execution monitoring, and reasoning about network communications. This work has been empirically validated using a distributed network-centric software evaluation testbed and the results provide guidance to designers in how to understand and control intelligent systems that operate in these environments.Comment: Proceedings of the 15th International Workshop on Non-Monotonic Reasoning (NMR 2014

MortonNet: Self-Supervised Learning of Local Features in 3D Point Clouds

We present a self-supervised task on point clouds, in order to learn meaningful point-wise features that encode local structure around each point. Our self-supervised network, named MortonNet, operates directly on unstructured/unordered point clouds. Using a multi-layer RNN, MortonNet predicts the next point in a point sequence created by a popular and fast Space Filling Curve, the Morton-order curve. The final RNN state (coined Morton feature) is versatile and can be used in generic 3D tasks on point clouds. In fact, we show how Morton features can be used to significantly improve performance (+3% for 2 popular semantic segmentation algorithms) in the task of semantic segmentation of point clouds on the challenging and large-scale S3DIS dataset. We also show how MortonNet trained on S3DIS transfers well to another large-scale dataset, vKITTI, leading to an improvement over state-of-the-art of 3.8%. Finally, we use Morton features to train a much simpler and more stable model for part segmentation in ShapeNet. Our results show how our self-supervised task results in features that are useful for 3D segmentation tasks, and generalize well to other datasets

Parking Sensing and Information System: Sensors, Deployment, and Evaluation

This paper describes a smart parking sensing and information system that disseminates the parking availability information for public users in a cost-effective and efficient manner. The hardware framework of the system is built on advanced wireless sensor networks and cloud service over the Internet, and the system is highly scalable. The parking information provided to the users is set in the form of occupancy rates and expected cruising time. Both are obtained from our analytical algorithm processing both historical and real-time data, and are thereafter visualized in a color theme. The entire parking system is deployed and extensively evaluated at Stanford University Parking Structure-1

Social Behavior in Bacterial Nanonetworks: Challenges and Opportunities

Molecular communication promises to enable communication between nanomachines with a view to increasing their functionalities and open up new possible applications. Due to some of the biological properties, bacteria have been proposed as a possible information carrier for molecular communication, and the corresponding communication networks are known as \textit{bacterial nanonetworks}. The biological properties include the ability for bacteria to mobilize between locations and carry the information encoded in Deoxyribonucleic Acid (DNA) molecules. However, similar to most organisms, bacteria have complex social properties that govern their colony. These social characteristics enable the bacteria to evolve through various fluctuating environmental conditions by utilizing cooperative and non-cooperative behaviors. This article provides an overview of the different types of cooperative and non-cooperative social behavior of bacteria. The challenges (due to non-cooperation) and the opportunities (due to cooperation) these behaviors can bring to the reliability of communication in bacterial nanonetworks are also discussed. Finally, simulation results on the impact of bacterial cooperative social behavior on the end-to-end reliability of a single-link bacterial nanonetwork are presented. The article concludes with highlighting the potential future research opportunities in this emerging field.Comment: Accepted for publication in IEEE Network Magazine as an open call articl

Blade: A Data Center Garbage Collector

An increasing number of high-performance distributed systems are written in garbage collected languages. This removes a large class of harmful bugs from these systems. However, it also introduces high tail-latency do to garbage collection pause times. We address this problem through a new technique of garbage collection avoidance which we call Blade. Blade is an API between the collector and application developer that allows developers to leverage existing failure recovery mechanisms in distributed systems to coordinate collection and bound the latency impact. We describe Blade and implement it for the Go programming language. We also investigate two different systems that utilize Blade, a HTTP load-balancer and the Raft consensus algorithm. For the load-balancer, we eliminate any latency introduced by the garbage collector, for Raft, we bound the latency impact to a single network round-trip, (48 {\mu}s in our setup). In both cases, latency at the tail using Blade is up to three orders of magnitude better.Comment: 14 pages, 9 figure

Incorporating Privileged Information to Unsupervised Anomaly Detection

We introduce a new unsupervised anomaly detection ensemble called SPI which can harness privileged information - data available only for training examples but not for (future) test examples. Our ideas build on the Learning Using Privileged Information (LUPI) paradigm pioneered by Vapnik et al. [19,17], which we extend to unsupervised learning and in particular to anomaly detection. SPI (for Spotting anomalies with Privileged Information) constructs a number of frames/fragments of knowledge (i.e., density estimates) in the privileged space and transfers them to the anomaly scoring space through "imitation" functions that use only the partial information available for test examples. Our generalization of the LUPI paradigm to unsupervised anomaly detection shepherds the field in several key directions, including (i) domain knowledge-augmented detection using expert annotations as PI, (ii) fast detection using computationally-demanding data as PI, and (iii) early detection using "historical future" data as PI. Through extensive experiments on simulated and real datasets, we show that augmenting privileged information to anomaly detection significantly improves detection performance. We also demonstrate the promise of SPI under all three settings (i-iii); with PI capturing expert knowledge, computationally expensive features, and future data on three real world detection tasks

GreyFiber: A System for Providing Flexible Access to Wide-Area Connectivity

Access to fiber-optic connectivity in the Internet is traditionally offered either via lit circuits or dark fiber. Economic (capex vs. opex) and operational considerations (latency, capacity) dictate the choice between these two offerings, but neither may effectively address the specific needs of modern-day enterprises or service providers over a range of use scenarios. In this paper, we describe a new approach for fiber-optic connectivity in the Internet that we call GreyFiber. The core idea of GreyFiber is to offer flexible access to fiber-optic paths between end points (e.g., datacenters or colocation facilities) over a range of timescales. We identify and discuss operational issues and systems challenges that need to be addressed to make GreyFiber a viable and realistic option for offering flexible access to infrastructure (similar to cloud computing). We investigate the efficacy of GreyFiber with a prototype implementation deployed in the GENI and CloudLab testbeds. Our scaling experiments show that 50 circuits can be provisioned within a minute. We also show that backup paths can be provisioned 28 times faster than an OSPF-based solution during failure/maintenance events. Our experiments also examine GreyFiber overhead demands and show that the time spent in circuit creation is dependent on the network infrastructure, indicating avenues for future improvements