Learning Sim-to-Real Dense Object Descriptors for Robotic Manipulation

It is crucial to address the following issues for ubiquitous robotics manipulation applications: (a) vision-based manipulation tasks require the robot to visually learn and understand the object with rich information like dense object descriptors; and (b) sim-to-real transfer in robotics aims to close the gap between simulated and real data. In this paper, we present Sim-to-Real Dense Object Nets (SRDONs), a dense object descriptor that not only understands the object via appropriate representation but also maps simulated and real data to a unified feature space with pixel consistency. We proposed an object-to-object matching method for image pairs from different scenes and different domains. This method helps reduce the effort of training data from real-world by taking advantage of public datasets, such as GraspNet. With sim-to-real object representation consistency, our SRDONs can serve as a building block for a variety of sim-to-real manipulation tasks. We demonstrate in experiments that pre-trained SRDONs significantly improve performances on unseen objects and unseen visual environments for various robotic tasks with zero real-world training.Comment: Accepted to International Conference on Robotics and Automation (ICRA) 202

FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue

Pre-trained language models based on general text enable huge success in the NLP scenario. But the intrinsical difference of linguistic patterns between general text and task-oriented dialogues makes existing pre-trained language models less useful in practice. Current dialogue pre-training methods rely on a contrastive framework and face the challenges of both selecting true positives and hard negatives. In this paper, we propose a novel dialogue pre-training model, FutureTOD, which distills future knowledge to the representation of the previous dialogue context using a self-training framework. Our intuition is that a good dialogue representation both learns local context information and predicts future information. Extensive experiments on diverse downstream dialogue tasks demonstrate the effectiveness of our model, especially the generalization, robustness, and learning discriminative dialogue representations capabilities.Comment: ACL 2023 Main Conferenc

Real-time spatial frequency domain imaging by single snapshot multiple frequency demodulation technique

We have presented a novel Single Snapshot Multiple Frequency Demodulation (SSMD) method enabling single snapshot wide field imaging of optical properties of turbid media in the Spatial Frequency Domain. SSMD makes use of the orthogonality of harmonic functions and extracts the modulation transfer function (MTF) at multiple modulation frequencies and of arbitrary orientations and amplitudes simultaneously from a single structured-illuminated image at once. SSMD not only increases significantly the data acquisition speed and reduces motion artifacts but also exhibits excellent noise suppression in imaging as well. The performance of SSMD-SFDI is demonstrated with experiments on both tissue mimicking phantoms and in vivo for recovering optical properties. SSMD is ideal in the implementation of a real-time spatial frequency domain imaging platform, which will open up SFDI for vast applications in, for example, mapping the optical properties of a dynamic turbid medium or monitoring fast temporal evolutions. © (2017) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only

Single snapshot multiple frequency modulated imaging of subsurface optical properties of turbid media with structured light

We report a novel demodulation method that enables single snapshot wide field imaging of optical properties of turbid media in the Spatial Frequency Domain (SFD). This Single Snapshot Multiple frequency Demodulation (SSMD) method makes use of the orthogonality of harmonic functions to extract the modulation transfer function (MTF) at multiple modulation frequencies simultaneously from a single structured-illuminated image at once. The orientation, frequency, and amplitude of each modulation can be set arbitrarily subject to the limitation of the implementation device. We first validate and compare SSMD to the existing demodulation methods by numerical simulations. The performance of SSMD is then demonstrated with experiments on both tissue mimicking phantoms and in vivo for recovering optical properties by comparing to the standard three-phase demodulation approach. The results show that SSMD increases significantly the data acquisition speed and reduces motion artefacts. SSMD exhibits excellent noise suppression in imaging as well at the rate proportional to the square root of the number of pixels contained in its kernel. SSMD is ideal in the implementation of a real-time spatial frequency domain imaging platform and will open up SFDI for vast applications in imaging and monitoring dynamic turbid medium and processes

In vivo real-time imaging of cutaneous hemoglobin concentration, oxygen saturation, scattering properties, melanin content, and epidermal thickness with visible spatially modulated light

We present the real-time single snapshot multiple frequency demodulation - spatial frequency domain imaging (SSMD-SFDI) platform implemented with a visible digital mirror device that is capable of imaging and monitoring dynamic turbid medium and processes over a large field of view. One challenge in quantitative imaging of biological tissue such as the skin is the complex structure rendering techniques based on homogeneous medium models to fail. To address this difficulty we have also developed a novel method that maps the layered structure to a homogeneous medium for spatial frequency domain imaging. The varying penetration depth of spatially modulated light on its wavelength and modulation frequency is used to resolve the layered structure. The efficacy of the real-time SSMD-SFDI platform and this two-layer model is demonstrated by imaging forearms of 6 healthy subjects under the reactive hyperemia protocol. The results show that our approach not only successfully decouples light absorption by melanin from that by hemoglobin and yields accurate determination of cutaneous hemoglobin concentration and oxygen saturation, but also provides reliable estimation of the scattering properties, the melanin content and the epidermal thickness in real time. Potential applications of our system in imaging skin physiological and functional states, cancer screening, and microcirculation monitoring are discussed at the end. © 2017 Optical Society of Americ

Seen to Unseen: Exploring Compositional Generalization of Multi-Attribute Controllable Dialogue Generation

Existing controllable dialogue generation work focuses on the single-attribute control and lacks generalization capability to out-of-distribution multiple attribute combinations. In this paper, we explore the compositional generalization for multi-attribute controllable dialogue generation where a model can learn from seen attribute values and generalize to unseen combinations. We propose a prompt-based disentangled controllable dialogue generation model, DCG. It learns attribute concept composition by generating attribute-oriented prompt vectors and uses a disentanglement loss to disentangle different attributes for better generalization. Besides, we design a unified reference-free evaluation framework for multiple attributes with different levels of granularities. Experiment results on two benchmarks prove the effectiveness of our method and the evaluation metric.Comment: ACL 2023 Main Conferenc

Overexpression of a MYB Family Gene, OsMYB6, Increases Drought and Salinity Stress Tolerance in Transgenic Rice

MYB transcription factors have been demonstrated to play key regulatory roles in plant growth, development and abiotic stress response. However, knowledge concerning the involvement of rice MYB genes in salinity and drought stress resistance are largely unknown. In the present study, we cloned and characterized the OsMYB6 gene, which was induced by drought and salinity stress. Subcellular localization of OsMYB6-YFP fusion protein in protoplast cells indicated that OsMYB6 was localized in the nucleus. Overexpression of OsMYB6 in rice did not suggest a negative effect on the growth and development of transgenic plants, but OsMYB6-overexpressing plants showed increased tolerance to drought and salt stress compared with wild-type plants, as are evaluated by higher proline content, higher CAT and SOD activities, lower REL and MDA content in transgenic plants under drought and salt stress conditions. In addition, the expression of abiotic stress-responsive genes were significantly higher in OsMYB6 transgenic plants than that in wild-type plants under drought and salt stress conditions. These results indicate that OsMYB6 gene functions as a stress-responsive transcription factor which plays a positive regulatory role in response to drought and salt stress resistance, and may be used as a candidate gene for molecular breeding of salt-tolerant and drought-tolerant crop varieties