GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration

We introduce a pipeline that enhances a general-purpose Vision Language Model, GPT-4V(ision), by integrating observations of human actions to facilitate robotic manipulation. This system analyzes videos of humans performing tasks and creates executable robot programs that incorporate affordance insights. The computation starts by analyzing the videos with GPT-4V to convert environmental and action details into text, followed by a GPT-4-empowered task planner. In the following analyses, vision systems reanalyze the video with the task plan. Object names are grounded using an open-vocabulary object detector, while focus on the hand-object relation helps to detect the moment of grasping and releasing. This spatiotemporal grounding allows the vision systems to further gather affordance data (e.g., grasp type, way points, and body postures). Experiments across various scenarios demonstrate this method's efficacy in achieving real robots' operations from human demonstrations in a zero-shot manner. The prompts of GPT-4V/GPT-4 are available at this project page: https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/Comment: 8 pages, 10 figures, 1 table. Last updated on November 20th, 202

Interactive Task Encoding System for Learning-from-Observation

We introduce a practical pipeline that interactively encodes multimodal human demonstrations for robot teaching. This pipeline is designed as an input system for a framework called Learning-from-Observation (LfO), which aims to program household robots with manipulative tasks through few-shots human demonstration without coding. While most previous LfO systems run with visual demonstration, recent research on robot teaching has shown the effectiveness of verbal instruction in making recognition robust and teaching interactive. To the best of our knowledge, however, no LfO system has yet been proposed that utilizes both verbal instruction and interaction, namely \textit{multimodal LfO}. This paper proposes the interactive task encoding system (ITES) as an input pipeline for multimodal LfO. ITES assumes that the user teaches step-by-step, pausing hand movements in order to match the granularity of human instructions with the granularity of robot execution. ITES recognizes tasks based on step-by-step verbal instructions that accompany the hand movements. Additionally, the recognition is made robust through interactions with the user. We test ITES on a real robot and show that the user can successfully teach multiple operations through multimodal demonstrations. The results suggest the usefulness of ITES for multimodal LfO. The source code is available at https://github.com/microsoft/symbolic-robot-teaching-interface.Comment: 7 pages, 10 figures. Last updated January 24st, 202

GPT Models Meet Robotic Applications: Co-Speech Gesturing Chat System

This technical paper introduces a chatting robot system that utilizes recent advancements in large-scale language models (LLMs) such as GPT-3 and ChatGPT. The system is integrated with a co-speech gesture generation system, which selects appropriate gestures based on the conceptual meaning of speech. Our motivation is to explore ways of utilizing the recent progress in LLMs for practical robotic applications, which benefits the development of both chatbots and LLMs. Specifically, it enables the development of highly responsive chatbot systems by leveraging LLMs and adds visual effects to the user interface of LLMs as an additional value. The source code for the system is available on GitHub for our in-house robot (https://github.com/microsoft/LabanotationSuite/tree/master/MSRAbotChatSimulation) and GitHub for Toyota HSR (https://github.com/microsoft/GPT-Enabled-HSR-CoSpeechGestures)

ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application

This paper demonstrates how OpenAI's ChatGPT can be used in a few-shot setting to convert natural language instructions into an executable robot action sequence. The paper proposes easy-to-customize input prompts for ChatGPT that meet common requirements in practical applications, such as easy integration with robot execution systems and applicability to various environments while minimizing the impact of ChatGPT's token limit. The prompts encourage ChatGPT to output a sequence of predefined robot actions, represent the operating environment in a formalized style, and infer the updated state of the operating environment. Experiments confirmed that the proposed prompts enable ChatGPT to act according to requirements in various environments, and users can adjust ChatGPT's output with natural language feedback for safe and robust operation. The proposed prompts and source code are open-source and publicly available at https://github.com/microsoft/ChatGPT-Robot-Manipulation-PromptsComment: 17 figures. Last updated April 11th, 202

Constraint-aware Policy for Compliant Manipulation

Robot manipulation in a physically-constrained environment requires compliant manipulation. Compliant manipulation is a manipulation skill to adjust hand motion based on the force imposed by the environment. Recently, reinforcement learning (RL) has been applied to solve household operations involving compliant manipulation. However, previous RL methods have primarily focused on designing a policy for a specific operation that limits their applicability and requires separate training for every new operation. We propose a constraint-aware policy that is applicable to various unseen manipulations by grouping several manipulations together based on the type of physical constraint involved. The type of physical constraint determines the characteristic of the imposed force direction; thus, a generalized policy is trained in the environment and reward designed on the basis of this characteristic. This paper focuses on two types of physical constraints: prismatic and revolute joints. Experiments demonstrated that the same policy could successfully execute various compliant-manipulation operations, both in the simulation and reality. We believe this study is the first step toward realizing a generalized household-robot

Usefulness of Cardiac Computed Tomography in the Diagnosis of Prosthetic Coronary Artery Graft with Interposition Procedure

An 80-year-old Japanese man was admitted with orthopnea and pitting edema of both lower legs. We diagnosed congestive heart failure (CHF) on the basis of a chest X-ray and an echocardiogram. An electrocardiogram showed a heart rate of 120 beats/min with atrial fibrillation rhythm (Af). The patient developed aortic valve failure and destruction of the base of right coronary artery (RCA) due to infectious endocarditis at 71 years of age. The patient underwent aortic valve replacement and coronary artery bypass grafting with an interposed graft with polyester vascular graft to RCA. The patient recovered from CHF after the 6 days of treatment with diuretics and verapamil. We confirmed the patency of coronary arteries and bypass grafts using a 64-slice cardiac computed tomography scan (CT) and diagnosed CHF due to Af. Here we describe the estimation of the prosthetic coronary artery graft patency with the interposition procedure using 64-slice cardiac CT

The Gravitation of the Moon Plays Pivotal Roles in the Occurrence of the Acute Myocardial Infarction

Acute myocardial infarction (AMI) is a social burden. However, being able to predict AMI could lead to prevention. A previous study showed only the relation between the lunar phase and the occurrence of AMI, but the period it takes for the moon to orbit around the earth and the period of the lunar phase differ. This study investigated the effect of the gravitation of the moon on AMI. Data was comprised of 1369 consecutive patients with first AMI at 5 hospitals from October, 1984 to December, 1997. The universal gravitation of the moon was calculated and compared to the earth onset time of AMI. Universal gravitation of the moon was derived by G*m/d2 (G: universal gravitation constant, m: the mass of the moon, d: the distance between the center of the moon and the center of the earth). The relationship between m/d2 and the cases of AMI was determined. There was an increase in cases, when there is a distance of more than 399864 km from the center of the earth to the center of the moon. The gravitation of more than 399864 km was determined to be weaker gravitation. It is confirmed that the number of AMI patients significantly increases at weaker gravitation periods in this multicenter trial. In conclusion, these results suggest that the gravitation of the moon may have an influence on the occurrence of AMI

Personalized prediction of overall survival in patients with AML in non‐complete remission undergoing allo‐HCT

Allogenic hematopoietic stem cell transplantation (allo-HCT) is the standard treatment for acute myeloid leukemia (AML) in non-complete remission (non-CR); however, the prognosis is inconsistent. This study aimed to develop and validate nomograms and a web application to predict the overall survival (OS) of patients with non-CR AML undergoing allo-HCT (cord blood transplantation [CBT], bone marrow transplantation [BMT], and peripheral blood stem cell transplantation [PBSCT]). Data from 3052 patients were analyzed to construct and validate the prognostic models. The common significant prognostic factors among patients undergoing allo-HCT were age, performance status, percentage of peripheral blasts, cytogenetic risk, chemotherapy response, and number of transplantations. The conditioning regimen was a significant prognostic factor only in patients undergoing CBT. Compared with cyclophosphamide/total body irradiation, a conditioning regimen of ≥3 drugs, including fludarabine, with CBT exhibited the lowest hazard ratio for mortality (0.384; 95% CI, 0.266-0.554; p < 0.0001). A conditioning regimen of ≥3 drugs with CBT also showed the best leukemia-free survival among all conditioning regimens. Based on the results of the multivariable analysis, we developed prognostic models showing adequate calibration and discrimination (the c-indices for CBT, BMT, and PBSCT were 0.648, 0.600, and 0.658, respectively). Our prognostic models can help in assessing individual risks and designing future clinical studies. Furthermore, our study indicates the effectiveness of multi-drug conditioning regimens in patients undergoing CBT