Spatio-Temporal Multimedia Big Data Analytics Using Deep Neural Networks

With the proliferation of online services and mobile technologies, the world has stepped into a multimedia big data era, where new opportunities and challenges appear with the high diversity multimedia data together with the huge amount of social data. Nowadays, multimedia data consisting of audio, text, image, and video has grown tremendously. With such an increase in the amount of multimedia data, the main question raised is how one can analyze this high volume and variety of data in an efficient and effective way. A vast amount of research work has been done in the multimedia area, targeting different aspects of big data analytics, such as the capture, storage, indexing, mining, and retrieval of multimedia big data. However, there is insufficient research that provides a comprehensive framework for multimedia big data analytics and management. To address the major challenges in this area, a new framework is proposed based on deep neural networks for multimedia semantic concept detection with a focus on spatio-temporal information analysis and rare event detection. The proposed framework is able to discover the pattern and knowledge of multimedia data using both static deep data representation and temporal semantics. Specifically, it is designed to handle data with skewed distributions. The proposed framework includes the following components: (1) a synthetic data generation component based on simulation and adversarial networks for data augmentation and deep learning training, (2) an automatic sampling model to overcome the imbalanced data issue in multimedia data, (3) a deep representation learning model leveraging novel deep learning techniques to generate the most discriminative static features from multimedia data, (4) an automatic hyper-parameter learning component for faster training and convergence of the learning models, (5) a spatio-temporal deep learning model to analyze dynamic features from multimedia data, and finally (6) a multimodal deep learning fusion model to integrate different data modalities. The whole framework has been evaluated using various large-scale multimedia datasets that include the newly collected disaster-events video dataset and other public datasets

Exploring Hidden Coherent Feature Groups and Temporal Semantics for Multimedia Big Data Analysis

Thanks to the advanced technologies and social networks that allow the data to be widely shared among the Internet, there is an explosion of pervasive multimedia data, generating high demands of multimedia services and applications in various areas for people to easily access and manage multimedia data. Towards such demands, multimedia big data analysis has become an emerging hot topic in both industry and academia, which ranges from basic infrastructure, management, search, and mining to security, privacy, and applications. Within the scope of this dissertation, a multimedia big data analysis framework is proposed for semantic information management and retrieval with a focus on rare event detection in videos. The proposed framework is able to explore hidden semantic feature groups in multimedia data and incorporate temporal semantics, especially for video event detection. First, a hierarchical semantic data representation is presented to alleviate the semantic gap issue, and the Hidden Coherent Feature Group (HCFG) analysis method is proposed to capture the correlation between features and separate the original feature set into semantic groups, seamlessly integrating multimedia data in multiple modalities. Next, an Importance Factor based Temporal Multiple Correspondence Analysis (i.e., IF-TMCA) approach is presented for effective event detection. Specifically, the HCFG algorithm is integrated with the Hierarchical Information Gain Analysis (HIGA) method to generate the Importance Factor (IF) for producing the initial detection results. Then, the TMCA algorithm is proposed to efficiently incorporate temporal semantics for re-ranking and improving the final performance. At last, a sampling-based ensemble learning mechanism is applied to further accommodate the imbalanced datasets. In addition to the multimedia semantic representation and class imbalance problems, lack of organization is another critical issue for multimedia big data analysis. In this framework, an affinity propagation-based summarization method is also proposed to transform the unorganized data into a better structure with clean and well-organized information. The whole framework has been thoroughly evaluated across multiple domains, such as soccer goal event detection and disaster information management

Using Smartpens to Examine and Influence the Relationship between Homework Habits and Academic Achievement in Introductory Engineering Courses

This dissertation examines students’ homework behaviors and their relationship to academic achievement in introductory engineering courses. Much of the prior work examining the relationship between homework and achievement has relied on student self-reports of homework effort. Our results demonstrate that such self-reports are problematic. Instead, we avoid this methodological shortcoming by using smartpens to objectively measure students’ learning activities in an unobtrusive manner and with a high level of fidelity. This dissertation examines how much, how frequently, and when students work on their homework assignments, and if these factors are related to achievement. This dissertation also examines if informing students of their homework behavior influences them to change that behavior and improve achievement. This work makes four major contributions. First, we developed quantitative measures of student homework behavior that are related to academic achievement. Second, we demonstrate that self-reported measures of student homework effort are problematic. Third, we show that measures of homework effort early in a course are nearly as effective at predicting achievement as measures from the entire course. This result suggests that student behavior does not change significantly over a course. Finally, we show that informing students of their homework behaviors, and providing suggestions for improving those behaviors, is an insufficient motivator to change behaviors and improve achievement. This result suggests a two-stage model of metacognition for study behaviors, requiring both monitoring (i.e., being aware of how one is studying) and regulation (i.e., adjusting how one studies based on feedback) to affect changes in behavior.This work makes both applied and methodological contributions to educational research. In contrast to existing research, our results demonstrate a strong and consistent relationship between students’ homework behaviors and academic achievement. Additionally, this work shows that students’ homework behaviors are established early in a course, and tend to remain relatively constant throughout a course.This work highlights the potential of educational data mining and smartpen technology for educational research. Our results confirm the unreliability of studies employing self-reports. Our studies also speak to the value of replication in education research