Multimodal Data Analytics and Fusion for Data Science

Advances in technologies have rapidly accumulated a zettabyte of “new” data every two years. The huge amount of data have a powerful impact on various areas in science and engineering and generates enormous research opportunities, which calls for the design and development of advanced approaches in data analytics. Given such demands, data science has become an emerging hot topic in both industry and academia, ranging from basic business solutions, technological innovations, and multidisciplinary research to political decisions, urban planning, and policymaking. Within the scope of this dissertation, a multimodal data analytics and fusion framework is proposed for data-driven knowledge discovery and cross-modality semantic concept detection. The proposed framework can explore useful knowledge hidden in different formats of data and incorporate representation learning from data in multimodalities, especial for disaster information management. First, a Feature Affinity-based Multiple Correspondence Analysis (FA-MCA) method is presented to analyze the correlations between low-level features from different features, and an MCA-based Neural Network (MCA-NN) ispro- posedto capture the high-level features from individual FA-MCA models and seamlessly integrate the semantic data representations for video concept detection. Next, a genetic algorithm-based approach is presented for deep neural network selection. Furthermore, the improved genetic algorithm is integrated with deep neural networks to generate populations for producing optimal deep representation learning models. Then, the multimodal deep representation learning framework is proposed to incorporate the semantic representations from data in multiple modalities efficiently. At last, fusion strategies are applied to accommodate multiple modalities. In this framework, cross-modal mapping strategies are also proposed to organize the features in a better structure to improve the overall performance

Knowledge-based design support and inductive learning

Designing and learning are closely related activities in that design as an ill-structure problem involves identifying the problem of the design as well as finding its solutions. A knowledge-based design support system should support learning by capturing and reusing design knowledge. This thesis addresses two fundamental problems in computational support to design activities: the development of an intelligent design support system architecture and the integration of inductive learning techniques in this architecture.This research is motivated by the belief that (1) the early stage of the design process can be modelled as an incremental learning process in which the structure of a design problem or the product data model of an artefact is developed using inductive learning techniques, and (2) the capability of a knowledge-based design support system can be enhanced by accumulating and storing reusable design product and process information.In order to incorporate inductive learning techniques into a knowledge-based design model and an integrated knowledge-based design support system architecture, the computational techniques for developing a knowledge-based design support system architecture and the role of inductive learning in Al-based design are investigated. This investigation gives a background to the development of an incremental learning model for design suitable for a class of design tasks whose structures are not well known initially.This incremental learning model for design is used as a basis to develop a knowledge-based design support system architecture that can be used as a kernel for knowledge-based design applications. This architecture integrates a number of computational techniques to support the representation and reasoning of design knowledge. In particular, it integrates a blackboard control system with an assumption-based truth maintenance system in an object-oriented environment to support the exploration of multiple design solutions by supporting the exploration and management of design contexts.As an integral part of this knowledge-based design support architecture, a design concept learning system utilising a number of unsupervised inductive learning techniques is developed. This design concept learning system combines concept formation techniques with design heuristics as background knowledge to build a design concept tree from raw data or past design examples. The design concept tree is used as a conceptual structure for the exploration of new designs.The effectiveness of this knowledge-based design support architecture and the design concept learning system is demonstrated through a realistic design domain, the design of small-molecule drugs one of the key tasks of which is to identify a pharmacophore description (the structure of a design problem) from known molecule examples.In this thesis, knowledge-based design and inductive learning techniques are first reviewed. Based on this review, an incremental learning model and an integrated architecture for intelligent design support are presented. The implementation of this architecture and a design concept learning system is then described. The application of the architecture and the design concept learning system in the domain of small-molecule drug design is then discussed. The evaluation of the architecture and the design concept learning system within and beyond this particular domain, and future research directions are finally discussed

A role for consolidation in cross-modal category learning

The ability to categorize objects and events is a fundamental human skill that depends upon the representation of multimodal conceptual knowledge. This study investigated the acquisition and consolidation of categorical information that required participants to integrate information across visual and auditory dimensions. The impact of wake- and sleep-dependent consolidation were investigated using a paradigm in which training and testing were separated by a delay spanning either an evening of sleep or daytime wakefulness, with a paired-associate episodic memory task used as a measure of classic sleep-dependent consolidation. Participants displayed good evidence of category learning, but did not show any wake- or sleep-dependent changes in memory for category information immediately following the delay. This is in contrast to paired-associate learning, where a sleep-dependent benefit was observed in memory recall. To replicate real-world concept learning, in which knowledge is acquired across multiple distinct episodes, participants were given a second opportunity for category learning following the consolidation delay. Here we found an interaction between consolidation and learning; with greater improvements in category knowledge as a result of the second session learning for those participants who had a sleep filled delay. These results suggest a role for sleep in the consolidation of recently acquired categorical knowledge; however this benefit does not emerge as an immediate benefit in memory recall, but by enhancing the effectiveness of future learning. This study therefore provides insights into the processes responsible for the formation and development of conceptual representations

Mental Representation and the Construction of Conceptual Understanding in Electronics Education

Learning about abstract electronics concepts can be difficult due to the hidden nature of the phenomena of interest. Developing understanding about electronics is therefore challenging because voltage cannot be readily observed; only the outcomes of the behaviour of voltage can be observed. Consequently modelling the phenomena of interest becomes a crucial factor in supporting learners in their development of knowledge and understanding. Visualisation skills have been promoted as important when modelling knowledge in different forms, supporting learners in their development of knowledge and understanding. Current research about electronics education, however, has tended to focus on learners’ misconceptions, experimental methods and interventions focusing on theoretical aspects of knowledge. Perspectives on learners’ actual constructions of knowledge in practice are not common. The aim of this research study, therefore, was to explore the use of external visual representations in support of learning about electronics concepts, within the context of Secondary Design and Technology education. The study adopts a case study approach and uses an interpretative cross-case synthesis methodology to explore a specific case of representation use among one class of Year 10 students. The analytical framework is designed to focus on the translation of and transition between multiple representations, including computer program code, and the representation of phenomena at three levels of representation: observable, symbolic and abstract. Data collection involved the observation of learners engaged with learning activities, documents collected from these activities, individual semi-structured interviews and participant characteristics data collected from course records. The findings show that common processes of learning are accompanied by individual developments in meaning and understanding. Individual understanding was characterised with the creation of four cognitive profiles representing key learner constructs. Understanding about abstract concepts was shown to benefit from representations where concrete referents linked with practical experience. Electronics understanding was also shown to benefit from the explanatory use of program code as a supporting method with which to model and simulate circuit behaviour. The research approach involving the close observation of learners engaging with learning activities was found to provide a greater understanding of learners’ approaches to learning in practice. The outcomes are applied to the practice of teaching electronics and modifications to the research are suggested for future researchers interested in the issues of teaching, learning and concept development in electronics education