Autonomous Marker-less Rapid Aerial Grasping

In a future with autonomous robots, visual and spatial perception is of utmost importance for robotic systems. Particularly for aerial robotics, there are many applications where utilizing visual perception is necessary for any real-world scenarios. Robotic aerial grasping using drones promises fast pick-and-place solutions with a large increase in mobility over other robotic solutions. Utilizing Mask R-CNN scene segmentation (detectron2), we propose a vision-based system for autonomous rapid aerial grasping which does not rely on markers for object localization and does not require the appearance of the object to be previously known. Combining segmented images with spatial information from a depth camera, we generate a dense point cloud of the detected objects and perform geometry-based grasp planning to determine grasping points on the objects. In real-world experiments on a dynamically grasping aerial platform, we show that our system can replicate the performance of a motion capture system for object localization up to 94.5 % of the baseline grasping success rate. With our results, we show the first use of geometry-based grasping techniques with a flying platform and aim to increase the autonomy of existing aerial manipulation platforms, bringing them further towards real-world applications in warehouses and similar environments.Comment: 8 pages, 10 figures, accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2023. Video https://www.youtube.com/watch?v=6hbhAT4l90

A Compact Acoustic Communication Module for Remote Control Underwater

This paper describes an end-to-end compact acoustic communication system designed for easy integration into remotely controlled underwater operations. The system supports up to 2048 commands that are encoded as 16 bit words. We present the design, hardware, and supporting algorithms for this system. A pulse-based FSK modulation scheme is presented, along with a method of demodulation requiring minimal processing power that leverages the Goertzel algorithm and dynamic peak detection. We packaged the system together with an intuitive user interface for remotely controlling an autonomous underwater vehicle. We evaluated this system in the pool and in the open ocean. We present the communication data collected during experiments using the system to control an underwater robot.National Science Foundation (U.S.) (NSF 1117178)National Science Foundation (U.S.) (NSF IIS1226883)National Science Foundation (U.S.) (Award 112237

Safe Local Navigation for Visually Impaired Users With a Time-of-Flight and Haptic Feedback Device

This paper presents ALVU (Array of Lidars and Vibrotactile Units), a contactless, intuitive, hands-free, and discreet wearable device that allows visually impaired users to detect low- and high-hanging obstacles, as well as physical boundaries in their immediate environment. The solution allows for safe local navigation in both confined and open spaces by enabling the user to distinguish free space from obstacles. The device presented is composed of two parts: a sensor belt and a haptic strap. The sensor belt is an array of time-of-flight distance sensors worn around the front of a user's waist, and the pulses of infrared light provide reliable and accurate measurements of the distances between the user and surrounding obstacles or surfaces. The haptic strap communicates the measured distances through an array of vibratory motors worn around the user's upper abdomen, providing haptic feedback. The linear vibration motors are combined with a point-loaded pretensioned applicator to transmit isolated vibrations to the user. We validated the device's capability in an extensive user study entailing 162 trials with 12 blind users. Users wearing the device successfully walked through hallways, avoided obstacles, and detected staircases.Andrea Bocelli FoundationNational Science Foundation (U.S.) (Grant NSF IIS1226883

Meta-Learning via Classifier(-free) Guidance

State-of-the-art meta-learning techniques do not optimize for zero-shot adaptation to unseen tasks, a setting in which humans excel. On the contrary, meta-learning algorithms learn hyperparameters and weight initializations that explicitly optimize for few-shot learning performance. In this work, we take inspiration from recent advances in generative modeling and language-conditioned image synthesis to propose meta-learning techniques that use natural language guidance to achieve higher zero-shot performance compared to the state-of-the-art. We do so by recasting the meta-learning problem as a multi-modal generative modeling problem: given a task, we consider its adapted neural network weights and its natural language description as equivalent multi-modal task representations. We first train an unconditional generative hypernetwork model to produce neural network weights; then we train a second "guidance" model that, given a natural language task description, traverses the hypernetwork latent space to find high-performance task-adapted weights in a zero-shot manner. We explore two alternative approaches for latent space guidance: "HyperCLIP"-based classifier guidance and a conditional Hypernetwork Latent Diffusion Model ("HyperLDM"), which we show to benefit from the classifier-free guidance technique common in image generation. Finally, we demonstrate that our approaches outperform existing meta-learning methods with zero-shot learning experiments on our Meta-VQA dataset, which we specifically constructed to reflect the multi-modal meta-learning setting

Perforated red blood cells enable compressible and injectable hydrogels as therapeutic vehicles

Hydrogels engineered for medical use within the human body need to be delivered in a minimally invasive fashion without altering their biochemical and mechanical properties to maximize their therapeutic outcomes. In this regard, key strategies applied for creating such medical hydrogels include formulating precursor solutions that can be crosslinked in situ with physical or chemical cues following their delivery or forming macroporous hydrogels at sub-zero temperatures via cryogelation prior to their delivery. Here, we present a new class of injectable composite materials with shape recovery ability. The shape recovery is derived from the physical properties of red blood cells (RBCs) that are first modified via hypotonic swelling and then integrated into the hydrogel scaffolds before polymerization. The RBCs' hypotonic swelling induces the formation of nanometer-sized pores on their cell membranes, which enable fast liquid release under compression. The resulting biocomposite hydrogel scaffolds display high deformability and shape-recovery ability. The scaffolds can repeatedly compress up to ~87% of their original volumes during injection and subsequent retraction through syringe needles of different sizes; this cycle of injection and retraction can be repeated up to ten times without causing any substantial mechanical damage to the scaffolds. Our biocomposite material system and fabrication approach for injectable materials will be foundational for the minimally invasive delivery of drug-loaded scaffolds, tissue-engineered constructs, and personalized medical platforms that could be administered to the human body with conventional needle-syringe systems

Vandermonde Neural Operators

Fourier Neural Operators (FNOs) have emerged as very popular machine learning architectures for learning operators, particularly those arising in PDEs. However, as FNOs rely on the fast Fourier transform for computational efficiency, the architecture can be limited to input data on equispaced Cartesian grids. Here, we generalize FNOs to handle input data on non-equispaced point distributions. Our proposed model, termed as Vandermonde Neural Operator (VNO), utilizes Vandermonde-structured matrices to efficiently compute forward and inverse Fourier transforms, even on arbitrarily distributed points. We present numerical experiments to demonstrate that VNOs can be significantly faster than FNOs, while retaining comparable accuracy, and improve upon accuracy of comparable non-equispaced methods such as the Geo-FNO.Comment: 21 pages, 10 figure

ViSE: Vision-Based 3D Online Shape Estimation of Continuously Deformable Robots

The precise control of soft and continuum robots requires knowledge of their shape. The shape of these robots has, in contrast to classical rigid robots, infinite degrees of freedom. To partially reconstruct the shape, proprioceptive techniques use built-in sensors resulting in inaccurate results and increased fabrication complexity. Exteroceptive methods so far rely on placing reflective markers on all tracked components and triangulating their position using multiple motion-tracking cameras. Tracking systems are expensive and infeasible for deformable robots interacting with the environment due to marker occlusion and damage. Here, we present a regression approach for 3D shape estimation using a convolutional neural network. The proposed approach takes advantage of data-driven supervised learning and is capable of real-time marker-less shape estimation during inference. Two images of a robotic system are taken simultaneously at 25 Hz from two different perspectives, and are fed to the network, which returns for each pair the parameterized shape. The proposed approach outperforms marker-less state-of-the-art methods by a maximum of 4.4% in estimation accuracy while at the same time being more robust and requiring no prior knowledge of the shape. The approach can be easily implemented due to only requiring two color cameras without depth and not needing an explicit calibration of the extrinsic parameters. Evaluations on two types of soft robotic arms and a soft robotic fish demonstrate our method's accuracy and versatility on highly deformable systems in real-time. The robust performance of the approach against different scene modifications (camera alignment and brightness) suggests its generalizability to a wider range of experimental setups, which will benefit downstream tasks such as robotic grasping and manipulation