Integrating Deep Learning with Correlation-based Multimedia Semantic Concept Detection

The rapid advances in technologies make the explosive growth of multimedia data possible and available to the public. Multimedia data can be defined as data collection, which is composed of various data types and different representations. Due to the fact that multimedia data carries knowledgeable information, it has been widely adopted to different genera, like surveillance event detection, medical abnormality detection, and many others. To fulfil various requirements for different applications, it is important to effectively classify multimedia data into semantic concepts across multiple domains. In this dissertation, a correlation-based multimedia semantic concept detection framework is seamlessly integrated with the deep learning technique. The framework aims to explore implicit and explicit correlations among features and concepts while adopting different Convolutional Neural Network (CNN) architectures accordingly. First, the Feature Correlation Maximum Spanning Tree (FC-MST) is proposed to remove the redundant and irrelevant features based on the correlations between the features and positive concepts. FC-MST identifies the effective features and decides the initial layer\u27s dimension in CNNs. Second, the Negative-based Sampling method is proposed to alleviate the data imbalance issue by keeping only the representative negative instances in the training process. To adjust dierent sizes of training data, the number of iterations for the CNN is determined adaptively and automatically. Finally, an Indirect Association Rule Mining (IARM) approach and a correlation-based re-ranking method are proposed to reveal the implicit relationships from the correlations among concepts, which are further utilized together with the classification scores to enhance the re-ranking process. The framework is evaluated using two benchmark multimedia data sets, TRECVID and NUS-WIDE, which contain large amounts of multimedia data and various semantic concepts

Negative-Based Sampling for Multimedia Retrieval

Nowadays, in such a high-tech living lifestyle, profusion of multimedia data are produced and propagated around the world. To identify meaningful semantic concepts from the large amount of data, one of the major challenges is called the data imbalance problem. Data imbalance occurs when the number of positive instances (i.e., instances which contain the target concept) is greatly less than the number of negative instances (i.e., instances which do not contain the target concept). In other words, the ratio between positive and negative instances is extremely low. Rebalancing the dataset is usually proposed to resolve the problem by sampling or data pruning. In this paper, we propose a sampling method which consists of three stages, namely selecting features to identify the negative instances, producing negative ranking scores, and performing sampling. The method is compared with some other existing methods on the TRECVID dataset and is demonstrated to have better performance