An Overview of Multimodal Remote Sensing Data Fusion: From Image to Feature, from Shallow to Deep

With the ever-growing availability of different remote sens-ing (RS) products from both satellite and airborne platforms,simultaneous processing and interpretation of multimodal RSdata have shown increasing significance in the RS field. Dif-ferent resolutions, contexts, and sensors of multimodal RSdata enable the identification and recognition of the materialslying on the earth’s surface at a more accurate level by de-scribing the same object from different points of the view. Asa result, the topic on multimodal RS data fusion has graduallyemerged as a hotspot research direction in recent years.This paper aims at presenting an overview of multimodalRS data fusion in several mainstream applications, which canbe roughly categorized by 1) image pansharpening, 2) hyper-spectral and multispectral image fusion, 3) multimodal fea-ture learning, and (4) crossmodal feature learning. For eachtopic, we will briefly describe what is the to-be-addressed re-search problem related to multimodal RS data fusion and givethe representative and state-of-the-art models from shallow todeep perspectives

A Deep Learning Framework in Selected Remote Sensing Applications

The main research topic is designing and implementing a deep learning framework applied to remote sensing. Remote sensing techniques and applications play a crucial role in observing the Earth evolution, especially nowadays, where the effects of climate change on our life is more and more evident. A considerable amount of data are daily acquired all over the Earth. Effective exploitation of this information requires the robustness, velocity and accuracy of deep learning. This emerging need inspired the choice of this topic. The conducted studies mainly focus on two European Space Agency (ESA) missions: Sentinel 1 and Sentinel 2. Images provided by the ESA Sentinel-2 mission are rapidly becoming the main source of information for the entire remote sensing community, thanks to their unprecedented combination of spatial, spectral and temporal resolution, as well as their open access policy. The increasing interest gained by these satellites in the research laboratory and applicative scenarios pushed us to utilize them in the considered framework. The combined use of Sentinel 1 and Sentinel 2 is crucial and very prominent in different contexts and different kinds of monitoring when the growing (or changing) dynamics are very rapid. Starting from this general framework, two specific research activities were identified and investigated, leading to the results presented in this dissertation. Both these studies can be placed in the context of data fusion. The first activity deals with a super-resolution framework to improve Sentinel 2 bands supplied at 20 meters up to 10 meters. Increasing the spatial resolution of these bands is of great interest in many remote sensing applications, particularly in monitoring vegetation, rivers, forests, and so on. The second topic of the deep learning framework has been applied to the multispectral Normalized Difference Vegetation Index (NDVI) extraction, and the semantic segmentation obtained fusing Sentinel 1 and S2 data. The S1 SAR data is of great importance for the quantity of information extracted in the context of monitoring wetlands, rivers and forests, and many other contexts. In both cases, the problem was addressed with deep learning techniques, and in both cases, very lean architectures were used, demonstrating that even without the availability of computing power, it is possible to obtain high-level results. The core of this framework is a Convolutional Neural Network (CNN). {CNNs have been successfully applied to many image processing problems, like super-resolution, pansharpening, classification, and others, because of several advantages such as (i) the capability to approximate complex non-linear functions, (ii) the ease of training that allows to avoid time-consuming handcraft filter design, (iii) the parallel computational architecture. Even if a large amount of "labelled" data is required for training, the CNN performances pushed me to this architectural choice.} In our S1 and S2 integration task, we have faced and overcome the problem of manually labelled data with an approach based on integrating these two different sensors. Therefore, apart from the investigation in Sentinel-1 and Sentinel-2 integration, the main contribution in both cases of these works is, in particular, the possibility of designing a CNN-based solution that can be distinguished by its lightness from a computational point of view and consequent substantial saving of time compared to more complex deep learning state-of-the-art solutions

Mechanisms and Consequences of Microtubule-Based Symmetry Breaking in Plant Roots

Directional growth in plants is primarily determined by the axis of cell expansion, which is specified by the net orientation of cortical microtubules. Microtubules guide the deposition of cellulose and other cell wall materials. In rapidly elongating cells, transversely oriented microtubules create material anisotropy in the cell wall that prevents radial cell expansion, channeling cell expansion in the longitudinal direction. Mutations perturbing microtubule organization frequently lead to aberrant cell growth in land plants, with some mutations leading to helical growth patterns (called ‘twisted mutants’), often in roots. This phenotype manifests as right-handed or left-handed twisting of cell files along the long axis of plant organs, which correlates with rightward or leftward organ growth, respectively. Helical growth is a common occurrence in the plant kingdom and serves a variety of purposes, but the molecular mechanisms that produce helical growth and define handedness are not well understood. Furthermore, how molecular-level processes propagate across spatial scales to control organ-level growth is undefined. Here, I used the model plant Arabidopsis thaliana as an experimentally tractable system, focusing on the root organ to study the mechanisms underlying helical growth in plants. In this work, I used roots as a model plant organ to investigate the molecular mechanisms that control symmetry maintenance and symmetry breaking in plants. Arabidopsis roots are ideally suited for this work because of their simple, concentric ring-like cellular anatomy and well-defined process of development. I selected two Arabidopsis twisted mutants with opposite chirality to study whether the emergence of right-handed and left-handed helical growth involves conserved or distinct mechanisms. Cortical microtubules are skewed in the right-handed spr1 mutant, which lacks a microtubule plus end-associated protein that regulates polymerization dynamics. In contrast, cortical microtubules tend to be laterally displaced in the left-handed cmu1 mutant, which lacks a protein that contributes to the attachment of cortical microtubules to the plasma membrane. Using a cell-type specific complementation approach, I showed that both SPR1 and CMU1 gene expression in the epidermis alone is sufficient to maintain wild-type-like straight cell files and root growth. In addition, epidermal expression of SPR1 restores both the morphology and skew of the cortical cell file to wild-type-like. By genetically disrupting cell-cell adhesion in the spr1 mutant, I found that a physical connection between epidermal and cortical cells is required for the epidermis to cause organ-level skewed growth. Together, these data demonstrate that the epidermis plays a central role in maintaining straight root growth, suggesting that twisted plant growth in nature could arise by altering microtubule behavior in the epidermis alone and does not require null alleles in all cells. To examine whether cortical microtubule defects in the spr1-3 mutant affect cell growth, I conducted morphometry analysis. I found that while skewed cortical microtubule orientation correlates with asymmetric epidermal cell morphology and growth in the spr1-3 mutant root meristem, cell file twisting is not manifested until the differentiation zone of the root where cell growth slows down and root hairs emerge. Furthermore, I demonstrated that cell file twisting is not sufficient to generate skewed growth at the organ level, which requires that the root is grown on an agar medium, a mechanically heterogeneous environment. Increasing the stiffness of the agar medium caused the spr1-3 and cmu1 mutant roots to grow straight, indicating that mechanical stimuli influence twisted root growth. Despite their important role in root anchorage, root hairs on the epidermis are not required for skewed root growth, nor for reorienting root skewing in response to changes in the mechanical environment. Overall, this work provides new insights into how symmetry breaking affects root mechanoresponse. Spatial heterogeneity in the composition and organization of the plant cell wall affects its mechanics to control cell shape and directional growth. In the last chapter of this work, I describe a new methodology for imaging plant primary cell walls at the nanoscale using atomic force microscopy coupled with infrared spectroscopy (AFM-IR). I contributed to generating a novel sample preparation technique and employed AFM-IR and spectral deconvolution to generate high-resolution spatial maps of the mechanochemical signatures of the Arabidopsis epidermal cell wall. Cross-correlation analysis of the spatial distribution of chemical and mechanical properties suggested that the carbohydrate composition of cell wall junctions correlates with increased local stiffness. In developing this methodology, this chapter provides an essential foundation for applying AFM-IR to understand the complex mechanochemistry of intact plant cell walls at nanometer resolution

Imaging studies of peripheral nerve regeneration induced by porous collagen biomaterials

Thesis: Ph. D., Massachusetts Institute of Technology, Department of Mechanical Engineering, 2013.Cataloged from PDF version of thesis.Includes bibliographical references.There is urgent need to develop treatments for inducing regeneration in injured organs. Porous collagen-based scaffolds have been utilized clinically to induce regeneration in skin and peripheral nerves, however still there is no complete explanation about the underlying mechanism. This thesis utilizes advanced microscopy to study the expression of contractile cell phenotypes during wound healing, a phenotype believed to affect significantly the final outcome. The first part develops an efficient pipeline for processing challenging spectral fluorescence microscopy images. Images are segmented into regions of objects by refining the outcome of a pixel-wide model selection classifier by an efficient Markov Random Field model. The methods of this part are utilized by the following parts. The second part extends the image informatics methodology in studying signal transduction networks in cells interacting with 3D matrices. The methodology is applied in a pilot study of TGFP signal transduction by the SMAD pathway in fibroblasts seeded in porous collagen scaffolds. Preliminary analysis suggests that the differential effect of TGFP1 and TGFP3 to cells could be attributed to the "non-canonical" SMADI and SMAD5. The third part is an ex vivo imaging study of peripheral nerve regeneration, which focuses on the formation of a capsule of contractile cells around transected rat sciatic nerves grafted with collagen scaffolds, 1 or 2 weeks post-injury. It follows a recent study that highlights an inverse relationship between the quality of the newly formed nerve tissue and the size of the contractile cell capsule 9 weeks post-injury. Results suggest that "active" biomaterials result in significantly thinner capsule already 1 week post-injury. The fourth part describes a novel method for quantifying the surface chemistry of 3D matrices. The method is an in situ binding assay that utilizes fluorescently labeled recombinant proteins that emulate the receptor of , and is applied to quantify the density of ligands for integrins a113, a2p1 on the surface of porous collagen scaffolds. Results provide estimates for the density of ligands on "active" and "inactive" scaffolds and demonstrate that chemical crosslinking can affect the surface chemistry of biomaterials, therefore can affect the way cells sense and respond to the material.by Dimitrios S. Tzeranis.Ph. D

Advances and Applications of DSmT for Information Fusion. Collected Works, Volume 5

This fifth volume on Advances and Applications of DSmT for Information Fusion collects theoretical and applied contributions of researchers working in different fields of applications and in mathematics, and is available in open-access. The collected contributions of this volume have either been published or presented after disseminating the fourth volume in 2015 in international conferences, seminars, workshops and journals, or they are new. The contributions of each part of this volume are chronologically ordered. First Part of this book presents some theoretical advances on DSmT, dealing mainly with modified Proportional Conflict Redistribution Rules (PCR) of combination with degree of intersection, coarsening techniques, interval calculus for PCR thanks to set inversion via interval analysis (SIVIA), rough set classifiers, canonical decomposition of dichotomous belief functions, fast PCR fusion, fast inter-criteria analysis with PCR, and improved PCR5 and PCR6 rules preserving the (quasi-)neutrality of (quasi-)vacuous belief assignment in the fusion of sources of evidence with their Matlab codes. Because more applications of DSmT have emerged in the past years since the apparition of the fourth book of DSmT in 2015, the second part of this volume is about selected applications of DSmT mainly in building change detection, object recognition, quality of data association in tracking, perception in robotics, risk assessment for torrent protection and multi-criteria decision-making, multi-modal image fusion, coarsening techniques, recommender system, levee characterization and assessment, human heading perception, trust assessment, robotics, biometrics, failure detection, GPS systems, inter-criteria analysis, group decision, human activity recognition, storm prediction, data association for autonomous vehicles, identification of maritime vessels, fusion of support vector machines (SVM), Silx-Furtif RUST code library for information fusion including PCR rules, and network for ship classification. Finally, the third part presents interesting contributions related to belief functions in general published or presented along the years since 2015. These contributions are related with decision-making under uncertainty, belief approximations, probability transformations, new distances between belief functions, non-classical multi-criteria decision-making problems with belief functions, generalization of Bayes theorem, image processing, data association, entropy and cross-entropy measures, fuzzy evidence numbers, negator of belief mass, human activity recognition, information fusion for breast cancer therapy, imbalanced data classification, and hybrid techniques mixing deep learning with belief functions as well

Gaze-Based Human-Robot Interaction by the Brunswick Model

We present a new paradigm for human-robot interaction based on social signal processing, and in particular on the Brunswick model. Originally, the Brunswick model copes with face-to-face dyadic interaction, assuming that the interactants are communicating through a continuous exchange of non verbal social signals, in addition to the spoken messages. Social signals have to be interpreted, thanks to a proper recognition phase that considers visual and audio information. The Brunswick model allows to quantitatively evaluate the quality of the interaction using statistical tools which measure how effective is the recognition phase. In this paper we cast this theory when one of the interactants is a robot; in this case, the recognition phase performed by the robot and the human have to be revised w.r.t. the original model. The model is applied to Berrick, a recent open-source low-cost robotic head platform, where the gazing is the social signal to be considered