A New Fine-grained Alignment Method for Image-text Matching

Image-text retrieval is a widely studied topic in the field of computer vision due to the exponential growth of multimedia data, whose core concept is to measure the similarity between images and text. However, most existing retrieval methods heavily rely on cross-attention mechanisms for cross-modal fine-grained alignment, which takes into account excessive irrelevant regions and treats prominent and non-significant words equally, thereby limiting retrieval accuracy. This paper aims to investigate an alignment approach that reduces the involvement of non-significant fragments in images and text while enhancing the alignment of prominent segments. For this purpose, we introduce the Cross-Modal Prominent Fragments Enhancement Aligning Network(CPFEAN), which achieves improved retrieval accuracy by diminishing the participation of irrelevant regions during alignment and relatively increasing the alignment similarity of prominent words. Additionally, we incorporate prior textual information into image regions to reduce misalignment occurrences. In practice, we first design a novel intra-modal fragments relationship reasoning method, and subsequently employ our proposed alignment mechanism to compute the similarity between images and text. Extensive quantitative comparative experiments on MS-COCO and Flickr30K datasets demonstrate that our approach outperforms state-of-the-art methods by about 5% to 10% in the rSum metric

Retrieval and monitoring processes during visual working memory: An ERP study of the benefit of visual semantics

In this study we examined electrophysiological indices of episodic remembering whilst participants recalled novel shapes, with and without semantic content, within a visual working memory paradigm. The components of interest were the parietal episodic (PE; 400-800ms) and late posterior negativity (LPN; 500-900ms), as these have previously been identified as reliable markers of recollection and post retrieval monitoring, respectively. Fifteen young adults completed a visual matrix patterns task, assessing memory for low and high semantic visual representations. Matrices with either low semantic or high semantic content (containing familiar visual forms) were briefly presented to participants for study (1500ms), followed by a retention interval (6000ms) and finally a same/different recognition phase. The event-related potentials of interest were tracked from the onset of the recognition test stimuli. Analyses revealed equivalent amplitude for the earlier PE effect for the processing of both low and high semantic stimulus types. However, the LPN was more negative-going for the processing of the low semantic stimuli. These data are discussed in terms of relatively ‘pure’ retrieval of high semantic items, where support can readily be recruited from semantic memory. However, for the low semantic items additional executive resources, as indexed by the LPN, are recruited when memory monitoring and uncertainty exist in order to recall previously studied items more effectively

Automatic detection of salient objects and spatial relations in videos for a video database system

Cataloged from PDF version of article.Multimedia databases have gained popularity due to rapidly growing quantities of multimedia data and the need to perform efficient indexing, retrieval and analysis of this data. One downside of multimedia databases is the necessity to process the data for feature extraction and labeling prior to storage and querying. Huge amount of data makes it impossible to complete this task manually. We propose a tool for the automatic detection and tracking of salient objects, and derivation of spatio-temporal relations between them in video. Our system aims to reduce the work for manual selection and labeling of objects significantly by detecting and tracking the salient objects, and hence, requiring to enter the label for each object only once within each shot instead of specifying the labels for each object in every frame they appear. This is also required as a first step in a fully-automatic video database management system in which the labeling should also be done automatically. The proposed framework covers a scalable architecture for video processing and stages of shot boundary detection, salient object detection and tracking, and knowledge-base construction for effective spatio-temporal object querying. (c) 2008 Elsevier B.V. All rights reserved

A new approach for estimating northern peatland gross primary productivity using a satellite-sensor-derived chlorophyll index

Carbon flux models that are largely driven by remotely sensed data can be used to estimate gross primary productivity (GPP) over large areas, but despite the importance of peatland ecosystems in the global carbon cycle, relatively little attention has been given to determining their success in these ecosystems. This paper is the first to explore the potential of chlorophyll-based vegetation index models for estimating peatland GPP from satellite data. Using several years of carbon flux data from contrasting peatlands, we explored the relationships between the MERIS terrestrial chlorophyll index (MTCI) and GPP, and determined whether the inclusion of environmental variables such as PAR and temperature, thought to be important determinants of peatland carbon flux, improved upon direct relationships. To place our results in context, we compared the newly developed GPP models with the MODIS (Moderate Resolution Imaging Spectrometer) GPP product. Our results show that simple MTCI-based models can be used for estimates of interannual and intra-annual variability in peatland GPP. The MTCI is a good indicator of GPP and compares favorably with more complex products derived from the MODIS sensor on a site-specific basis. The incorporation of MTCI into a light use efficiency type model, by means of partitioning the fraction of photosynthetic material within a plant canopy, shows most promise for peatland GPP estimation, outperforming all other models. Our results demonstrate that satellite data specifically related to vegetation chlorophyll content may ultimately facilitate improved quantification of peatland carbon flux dynamics

Effect of the ingestion in the WRF model of different Sentinel-derived and GNSS-derived products: analysis of the forecasts of a high impact weather event

This paper presents the first experimental results of a study on the ingestion in the Weather Research and Forecasting (WRF) model, of Sentinel satellites and Global Navigation Satellite Systems (GNSS) derived products. The experiments concern a flash-floodevent occurred in Tuscany (Central Italy) in September 2017. The rationale is that numerical weather prediction (NWP) models are presently able to produce forecasts with a km scale spatial resolution, but the poor knowledge of the initial state of the atmosphere may imply an inaccurate simulation of the weather phenomena. Hence, to fully exploit the advances in numerical weather modelling, it is necessary to feed them with high spatiotemporal resolution information over the surface boundary and the atmospheric column. In this context, the Copernicus Sentinel satellites represent an important source of data, because they can provide a set of high-resolution observations of physical variables (e.g. soil moisture, land/sea surface temperature, wind speed) used in NWP models runs. The possible availability of a spatially dense network of GNSS stations is also exploited to assimilate water vapour content. Results show that the assimilation of Sentinel-1 derived wind field and GNSS-derivedwater vapour data produce the most positive effects on the performance of the forecast

Effect of the ingestion in the WRF model of different Sentinel-derived and GNSS-derived products: analysis of the forecasts of a high impact weather event

This paper presents the first experimental results of a study on the ingestion in the Weather Research and Forecasting (WRF) model, of Sentinel satellites and Global Navigation Satellite Systems (GNSS) derived products. The experiments concern a flash-floodevent occurred in Tuscany (Central Italy) in September 2017. The rationale is that numerical weather prediction (NWP) models are presently able to produce forecasts with a km scale  spatial resolution, but the poor knowledge of the initial state of the atmosphere may imply an inaccurate simulation of the weather phenomena. Hence, to fully exploit the advances in numerical weather modelling, it is necessary to feed them with high spatiotemporal resolution information over the surface boundary and the atmospheric column. In this context, the Copernicus Sentinel satellites represent an important source of data, because they can provide a set of high-resolution observations of physical variables (e.g. soil moisture, land/sea surface temperature, wind speed) used in NWP models runs. The possible availability of a spatially dense network of GNSS stations is also exploited to assimilate water vapour content. Results show that the assimilation of Sentinel-1 derived wind field and GNSS-derivedwater vapour data produce the most positive effects on the performance of the forecast

Saliency maps on image hierarchies

© 2015 Elsevier B.V. All rights reserved. In this paper we propose two saliency models for salient object segmentation based on a hierarchical image segmentation, a tree-like structure that represents regions at different scales from the details to the whole image (e.g. gPb-UCM, BPT). The first model is based on a hierarchy of image partitions. The saliency at each level is computed on a region basis, taking into account the contrast between regions. The maps obtained for the different partitions are then integrated into a final saliency map. The second model directly works on the structure created by the segmentation algorithm, computing saliency at each node and integrating these cues in a straightforward manner into a single saliency map. We show that the proposed models produce high quality saliency maps. Objective evaluation demonstrates that the two methods achieve state-of-the-art performance in several benchmark datasets.Peer ReviewedPostprint (author's final draft

Seasonal Arctic sea ice forecasting with probabilistic deep learning.

Anthropogenic warming has led to an unprecedented year-round reduction in Arctic sea ice extent. This has far-reaching consequences for indigenous and local communities, polar ecosystems, and global climate, motivating the need for accurate seasonal sea ice forecasts. While physics-based dynamical models can successfully forecast sea ice concentration several weeks ahead, they struggle to outperform simple statistical benchmarks at longer lead times. We present a probabilistic, deep learning sea ice forecasting system, IceNet. The system has been trained on climate simulations and observational data to forecast the next 6 months of monthly-averaged sea ice concentration maps. We show that IceNet advances the range of accurate sea ice forecasts, outperforming a state-of-the-art dynamical model in seasonal forecasts of summer sea ice, particularly for extreme sea ice events. This step-change in sea ice forecasting ability brings us closer to conservation tools that mitigate risks associated with rapid sea ice loss

Seasonal Arctic sea ice forecasting with probabilistic deep learning

Anthropogenic warming has led to an unprecedented year-round reduction in Arctic sea ice extent. This has far-reaching consequences for indigenous and local communities, polar ecosystems, and global climate, motivating the need for accurate seasonal sea ice forecasts. While physics-based dynamical models can successfully forecast sea ice concentration several weeks ahead, they struggle to outperform simple statistical benchmarks at longer lead times. We present a probabilistic, deep learning sea ice forecasting system, IceNet. The system has been trained on climate simulations and observational data to forecast the next 6 months of monthly-averaged sea ice concentration maps. We show that IceNet advances the range of accurate sea ice forecasts, outperforming a state-of-the-art dynamical model in seasonal forecasts of summer sea ice, particularly for extreme sea ice events. This step-change in sea ice forecasting ability brings us closer to conservation tools that mitigate risks associated with rapid sea ice loss