Visual-Linguistic Semantic Alignment: Fusing Human Gaze and Spoken Narratives for Image Region Annotation

Advanced image-based application systems such as image retrieval and visual question answering depend heavily on semantic image region annotation. However, improvements in image region annotation are limited because of our inability to understand how humans, the end users, process these images and image regions. In this work, we expand a framework for capturing image region annotations where interpreting an image is influenced by the end user\u27s visual perception skills, conceptual knowledge, and task-oriented goals. Human image understanding is reflected by individuals\u27 visual and linguistic behaviors, but the meaningful computational integration and interpretation of their multimodal representations (e.g. gaze, text) remain a challenge. Our work explores the hypothesis that eye movements can help us understand experts\u27 perceptual processes and that spoken language descriptions can reveal conceptual elements of image inspection tasks. We propose that there exists a meaningful relation between gaze, spoken narratives, and image content. Using unsupervised bitext alignment, we create meaningful mappings between participants\u27 eye movements (which reveal key areas of images) and spoken descriptions of those images. The resulting alignments are then used to annotate image regions with concept labels. Our alignment accuracy exceeds baseline alignments that are obtained using both simultaneous and a fixed-delay temporal correspondence. Additionally, comparison of alignment accuracy between a method that identifies clusters in the images based on eye movements and a method that identifies clusters using image features shows that the two approaches perform well on different types of images and concept labels. This suggests that an image annotation framework could integrate information from more than one technique to handle heterogeneous images. The resulting alignments can be used to create a database of low-level image features and high-level semantic annotations corresponding to perceptually important image regions. We demonstrate the applicability of the proposed framework with two datasets: one consisting of general-domain images and another with images from the domain of medicine. This work is an important contribution toward the highly challenging problem of fusing human-elicited multimodal data sources, a problem that will become increasingly important as low-resource scenarios become more common

Pilot investigation of remote sensing for intertidal oyster mapping in coastal South Carolina: a methods comparison

South Carolina’s oyster reefs are a major component of the coastal landscape. Eastern oysters Crassostrea virginica are an important economic resource to the state and serve many essential functions in the environment, including water filtration, creek bank stabilization and habitat for other plants and animals. Effective conservation and management of oyster reefs is dependent on an understanding of their abundance, distribution, condition, and change over time. In South Carolina, over 95% of the state’s oyster habitat is intertidal. The current intertidal oyster reef database for South Carolina was developed by field assessment over several years. This database was completed in the early 1980s and is in need of an update to assess resource/habitat status and trends across the state. Anthropogenic factors such as coastal development and associated waterway usage (e.g., boat wakes) are suspected of significantly altering the extent and health of the state’s oyster resources. In 2002 the NOAA Coastal Services Center’s (Center) Coastal Remote Sensing Program (CRS) worked with the Marine Resources Division of the South Carolina Department of Natural Resources (SCDNR) to develop methods for mapping intertidal oyster reefs along the South Carolina coast using remote sensing technology. The objective of this project was to provide SCDNR with potential methodologies and approaches for assessing oyster resources in a more efficiently than could be accomplished through field digitizing. The project focused on the utility of high-resolution aerial imagery and on documenting the effectiveness of various analysis techniques for accomplishing the update. (PDF contains 32 pages

The role of earth observation in an integrated deprived area mapping “system” for low-to-middle income countries

Urbanization in the global South has been accompanied by the proliferation of vast informal and marginalized urban areas that lack access to essential services and infrastructure. UN-Habitat estimates that close to a billion people currently live in these deprived and informal urban settlements, generally grouped under the term of urban slums. Two major knowledge gaps undermine the efforts to monitor progress towards the corresponding sustainable development goal (i.e., SDG 11—Sustainable Cities and Communities). First, the data available for cities worldwide is patchy and insufficient to differentiate between the diversity of urban areas with respect to their access to essential services and their specific infrastructure needs. Second, existing approaches used to map deprived areas (i.e., aggregated household data, Earth observation (EO), and community-driven data collection) are mostly siloed, and, individually, they often lack transferability and scalability and fail to include the opinions of different interest groups. In particular, EO-based-deprived area mapping approaches are mostly top-down, with very little attention given to ground information and interaction with urban communities and stakeholders. Existing top-down methods should be complemented with bottom-up approaches to produce routinely updated, accurate, and timely deprived area maps. In this review, we first assess the strengths and limitations of existing deprived area mapping methods. We then propose an Integrated Deprived Area Mapping System (IDeAMapS) framework that leverages the strengths of EO- and community-based approaches. The proposed framework offers a way forward to map deprived areas globally, routinely, and with maximum accuracy to support SDG 11 monitoring and the needs of different interest groups