Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data

Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose a comprehensive benchmark to gauge the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our benchmark includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks. Our benchmark will be made publicly available at https://vleo.danielz.ch/ and on Hugging Face at https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70 for easy model evaluation.Comment: 62 pages; work in progres

Geometric Multi-Model Fitting by Deep Reinforcement Learning

This paper deals with the geometric multi-model fitting from noisy, unstructured point set data (e.g., laser scanned point clouds). We formulate multi-model fitting problem as a sequential decision making process. We then use a deep reinforcement learning algorithm to learn the optimal decisions towards the best fitting result. In this paper, we have compared our method against the state-of-the-art on simulated data. The results demonstrated that our approach significantly reduced the number of fitting iterations

Wannier solitons in spin-orbit-coupled Bose-Einstein condensates in optical lattices with a flat-band

We investigate families of soliton solutions in a spin-orbit coupled Bose-Einstein condensate embedded in an optical lattice, which bifurcate from the nearly flat lowest band. Unlike the conventional gap solitons the obtained solutions have the shape well approximated by a Wannier function (or a few Wannier functions) of the underlying linear Hamiltonian with amplitudes varying along the family and with nearly constant widths. The Wannier solitons (WSs) sharing all symmetries of the system Hamiltonian are found to be stable. Such solutions allow for the construction of Wannier breathers, that can be viewed as nonlinearly coupled one-hump solitons. The breathers are well described by a few-mode model and manifest stable behavior either in an oscillatory regime with balanced average populations or in a self-trapping regime characterized by unbalanced atomic populations of the local potential minima (similarly to the conventional boson Josephson junction), with the frequencies controlled by the inter-atomic interactions.Comment: Accepted for publication in Physical Review