DaXBench: Benchmarking Deformable Object Manipulation with Differentiable Physics

Deformable Object Manipulation (DOM) is of significant importance to both daily and industrial applications. Recent successes in differentiable physics simulators allow learning algorithms to train a policy with analytic gradients through environment dynamics, which significantly facilitates the development of DOM algorithms. However, existing DOM benchmarks are either single-object-based or non-differentiable. This leaves the questions of 1) how a task-specific algorithm performs on other tasks and 2) how a differentiable-physics-based algorithm compares with the non-differentiable ones in general. In this work, we present DaXBench, a differentiable DOM benchmark with a wide object and task coverage. DaXBench includes 9 challenging high-fidelity simulated tasks, covering rope, cloth, and liquid manipulation with various difficulty levels. To better understand the performance of general algorithms on different DOM tasks, we conduct comprehensive experiments over representative DOM methods, ranging from planning to imitation learning and reinforcement learning. In addition, we provide careful empirical studies of existing decision-making algorithms based on differentiable physics, and discuss their limitations, as well as potential future directions.Comment: ICLR 2023 (Oral

Combining spectral and texture feature of UAV image with plant height to improve LAI estimation of winter wheat at jointing stage

IntroductionLeaf area index (LAI) is a critical physiological and biochemical parameter that profoundly affects vegetation growth. Accurately estimating the LAI for winter wheat during jointing stage is particularly important for monitoring wheat growth status and optimizing variable fertilization decisions. Recently, unmanned aerial vehicle (UAV) data and machine/depth learning methods are widely used in crop growth parameter estimation. In traditional methods, vegetation indices (VI) and texture are usually to estimate LAI. Plant Height (PH) unlike them, contains information about the vertical structure of plants, which should be consider.MethodsTaking Xixingdian Township, Cangzhou City, Hebei Province, China as the research area in this paper, and four machine learning algorithms, namely, support vector machine(SVM), back propagation neural network (BPNN), random forest (RF), extreme gradient boosting (XGBoost), and two deep learning algorithms, namely, convolutional neural network (CNN) and long short-term memory neural network (LSTM), were applied to estimate LAI of winter wheat at jointing stage by integrating the spectral and texture features as well as the plant height information from UAV multispectral images. Initially, Digital Surface Model (DSM) and Digital Orthophoto Map (DOM) were generated. Subsequently, the PH, VI and texture features were extracted, and the texture indices (TI) was further constructed. The measured LAI on the ground were collected for the same period and calculated its Pearson correlation coefficient with PH, VI and TI to pick the feature variables with high correlation. The VI, TI, PH and fusion were considered as the independent features, and the sample set partitioning based on joint x-y distance (SPXY) method was used to divide the calibration set and validation set of samples.ResultsThe ability of different inputs and algorithms to estimate winter wheat LAI were evaluated. The results showed that (1) The addition of PH as a feature variable significantly improved the accuracy of the LAI estimation, indicating that wheat plant height played a vital role as a supplementary parameter for LAI inversion modeling based on traditional indices; (2) The combination of texture features, including normalized difference texture indices (NDTI), difference texture indices (DTI), and ratio texture indices (RTI), substantially improved the correlation between texture features and LAI; Furthermore, multi-feature combinations of VI, TI, and PH exhibited superior capability in estimating LAI for winter wheat; (3) Six regression algorithms have achieved high accuracy in estimating LAI, among which the XGBoost algorithm estimated winter wheat LAI with the highest overall accuracy and best results, achieving the highest R2 (R2 = 0.88), the lowest RMSE (RMSE=0.69), and an RPD greater than 2 (RPD=2.54).DiscussionThis study provided compelling evidence that utilizing XGBoost and integrating spectral, texture, and plant height information extracted from UAV data can accurately monitor LAI during the jointing stage of winter wheat. The research results will provide a new perspective for accurate monitoring of crop parameters through remote sensing

Assessing rice chlorophyll content with vegetation indices from hyperspectral data

Abstract. Leaf chlorophyll content is not only an important biochemical parameter for determinating the capacity of rice photosynthesis, but also a good indicator of crop stress, nutritional state. Due to the reliable, operational and non-destructive advantages, hyperspectral remote sensing plays a significant role for assessing and monitoring chlorophyll content. In the study, a few of typical vegetation indices (VI) with the combination of 670nm and 800nm band reflectance, Normalized Difference Vegetation Index (NDVI), Modified Simple Ratio index (MSR), Modified Chlorophyll Absorption Ratio Index (MCARI), Transformed Chlorophyll Absorption Ratio Index (TCARI), and Optimized Soil-Adjusted Vegetation Index (OSAVI) are modified by using 705nm and 750nm band reflectance so as to reduce the effect of spectral saturation in 660-680nm absorptive band region, and then used to assess the rice chlorophyll content. The result shows that the five mentioned VIs have better correlation with rice chlorophyll content while using 705nm and 750nm. In addition, in the study the Weight optimization combination (WOC) principle is utilized to further assess the capacity of the five modified VIs for estimating rice chlorophyll content, it is proved that OSAVI and MSR display the better performance

Vision-Language Foundation Models as Effective Robot Imitators

Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.Comment: Fix typos. Project page: https://roboflamingo.github.i