A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models

Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. However, observing this grounding in contemporary models is complex, even if it is generally expected to take place if the task is addressed in a way that is conductive to generalization. We propose a framework to jointly study task performance and phrase grounding, and propose three benchmarks to study the relation between the two. Our results show that contemporary models demonstrate inconsistency between their ability to ground phrases and solve tasks. We show how this can be addressed through brute-force training on ground phrasing annotations, and analyze the dynamics it creates. Code and at available at https://github.com/lil-lab/phrase_grounding

lilGym: Natural Language Visual Reasoning with Reinforcement Learning

We present lilGym, a new benchmark for language-conditioned reinforcement learning in visual environments. lilGym is based on 2,661 highly-compositional human-written natural language statements grounded in an interactive visual environment. We introduce a new approach for exact reward computation in every possible world state by annotating all statements with executable Python programs. Each statement is paired with multiple start states and reward functions to form thousands of distinct Markov Decision Processes of varying difficulty. We experiment with lilGym with different models and learning regimes. Our results and analysis show that while existing methods are able to achieve non-trivial performance, lilGym forms a challenging open problem. lilGym is available at https://lil.nlp.cornell.edu/lilgym/.Comment: ACL 2023 Long Pape

OASIS: A Large-Scale Dataset for Single Image 3D in the Wild

Single-view 3D is the task of recovering 3D properties such as depth and surface normals from a single image. We hypothesize that a major obstacle to single-image 3D is data. We address this issue by presenting Open Annotations of Single Image Surfaces (OASIS), a dataset for single-image 3D in the wild consisting of annotations of detailed 3D geometry for 140,000 images. We train and evaluate leading models on a variety of single-image 3D tasks. We expect OASIS to be a useful resource for 3D vision research. Project site: https://pvl.cs.princeton.edu/OASIS.Comment: Accepted to CVPR 202

Pulmonary resection for metachronous metastatic gastric cancer diagnosed using multi-detector computed tomography: Report of five cases

Introduction As pulmonary resection for metastatic gastric cancer has been rarely reported on, the role of metastasectomy remains unclear in such settings. We reviewed the clinicopathological characteristics and surgical outcomes of patients with metachronous pulmonary metastasis from gastric cancer (MPMGC) diagnosed using multi-detector computed tomography (MDCT) who underwent pulmonary resection. Presentation of case From September 2002 to May 2018, five patients underwent pulmonary resection for MPMGC at Shizuoka Cancer Center. All patients received curative resection for initial gastric cancer. Three patients received adjuvant chemotherapy. The median age at pulmonary resection was 70 years. The median disease-free interval between initial gastrectomy and MPMGC diagnosis was 41 months. The first site of recurrence was the lung in all patients. All patients were diagnosed as having primary lung cancer using MDCT before pulmonary resection and fit the surgical indication for primary lung cancer. Lobectomy was performed in three patients, while wedge resection was performed in two. The median overall survival following pulmonary resection was 79 (range, 18–89) months. Two patients experienced recurrence. While one showed recurrence in the mediastinal lymph node, in the other it was observed in the remnant lung; the latter underwent repeated pulmonary resection followed by systemic chemotherapy. Four patients survived for longer than 4 years after pulmonary resection. Conclusions Of the five patients with MPMGC diagnosed using MDCT who underwent pulmonary resection, long-term survival was achieved after pulmonary resection in four. Thus, pulmonary resection may be considered for those diagnosed with lung nodules after surgery for gastric cancer, and who fit the surgical indication for primary lung cancer