Multimodal Earth observation data fusion: Graph-based approach in shared latent space

Multiple and heterogenous Earth observation (EO) platforms are broadly used for a wide array of applications, and the integration of these diverse modalities facilitates better extraction of information than using them individually. The detection capability of the multispectral unmanned aerial vehicle (UAV) and satellite imagery can be significantly improved by fusing with ground hyperspectral data. However, variability in spatial and spectral resolution can affect the efficiency of such dataset's fusion. In this study, to address the modality bias, the input data was projected to a shared latent space using cross-modal generative approaches or guided unsupervised transformation. The proposed adversarial networks and variational encoder-based strategies used bi-directional transformations to model the cross-domain correlation without using cross-domain correspondence. It may be noted that an interpolation-based convolution was adopted instead of the normal convolution for learning the features of the point spectral data (ground spectra). The proposed generative adversarial network-based approach employed dynamic time wrapping based layers along with a cyclic consistency constraint to use the minimal number of unlabeled samples, having cross-domain correlation, to compute a cross-modal generative latent space. The proposed variational encoder-based transformation also addressed the cross-modal resolution differences and limited availability of cross-domain samples by using a mixture of expert-based strategy, cross-domain constraints, and adversarial learning. In addition, the latent space was modelled to be composed of modality independent and modality dependent spaces, thereby further reducing the requirement of training samples and addressing the cross-modality biases. An unsupervised covariance guided transformation was also proposed to transform the labelled samples without using cross-domain correlation prior. The proposed latent space transformation approaches resolved the requirement of cross-domain samples which has been a critical issue with the fusion of multi-modal Earth observation data. This study also proposed a latent graph generation and graph convolutional approach to predict the labels resolving the domain discrepancy and cross-modality biases. Based on the experiments over different standard benchmark airborne datasets and real-world UAV datasets, the developed approaches outperformed the prominent hyperspectral panchromatic sharpening, image fusion, and domain adaptation approaches. By using specific constraints and regularizations, the network developed was less sensitive to network parameters, unlike in similar implementations. The proposed approach illustrated improved generalizability in comparison with the prominent existing approaches. In addition to the fusion-based classification of the multispectral and hyperspectral datasets, the proposed approach was extended to the classification of hyperspectral airborne datasets where the latent graph generation and convolution were employed to resolve the domain bias with a small number of training samples. Overall, the developed transformations and architectures will be useful for the semantic interpretation and analysis of multimodal data and are applicable to signal processing, manifold learning, video analysis, data mining, and time series analysis, to name a few.This research was partly supported by the Hebrew University of Jerusalem Intramural Research Found Career Development, Association of Field Crop Farmers in Israel and the Chief Scientist of the Israeli Ministry of Agriculture and Rural Development (projects 20-02-0087 and 12-01-0041)

Perceptual Anchoring in Preschool Children: Not Adultlike, but There

BACKGROUND: Recent studies suggest that human auditory perception follows a prolonged developmental trajectory, sometimes continuing well into adolescence. Whereas both sensory and cognitive accounts have been proposed, the development of the ability to base current perceptual decisions on prior information, an ability that strongly benefits adult perception, has not been directly explored. Here we ask whether the auditory frequency discrimination of preschool children also improves when given the opportunity to use previously presented standard stimuli as perceptual anchors, and whether the magnitude of this anchoring effect undergoes developmental changes. METHODOLOGY/PRINCIPAL FINDINGS: Frequency discrimination was tested using two adaptive same/different protocols. In one protocol (with-reference), a repeated 1-kHz standard tone was presented repeatedly across trials. In the other (no-reference), no such repetitions occurred. Verbal memory and early reading skills were also evaluated to determine if the pattern of correlations between frequency discrimination, memory and literacy is similar to that previously reported in older children and adults. Preschool children were significantly more sensitive in the with-reference than in the no-reference condition, but the magnitude of this anchoring effect was smaller than that observed in adults. The pattern of correlations among discrimination thresholds, memory and literacy replicated previous reports in older children. CONCLUSIONS/SIGNIFICANCE: The processes allowing the use of context to form perceptual anchors are already functional among preschool children, albeit to a lesser extent than in adults. Nevertheless, immature anchoring cannot fully account for the poorer frequency discrimination abilities of young children. That anchoring is present among the majority of typically developing preschool children suggests that the anchoring deficits observed among individuals with dyslexia represent a true deficit rather than a developmental delay