Query-controllable Video Summarization

When video collections become huge, how to explore both within and across videos efficiently is challenging. Video summarization is one of the ways to tackle this issue. Traditional summarization approaches limit the effectiveness of video exploration because they only generate one fixed video summary for a given input video independent of the information need of the user. In this work, we introduce a method which takes a text-based query as input and generates a video summary corresponding to it. We do so by modeling video summarization as a supervised learning problem and propose an end-to-end deep learning based method for query-controllable video summarization to generate a query-dependent video summary. Our proposed method consists of a video summary controller, video summary generator, and video summary output module. To foster the research of query-controllable video summarization and conduct our experiments, we introduce a dataset that contains frame-based relevance score labels. Based on our experimental result, it shows that the text-based query helps control the video summary. It also shows the text-based query improves our model performance. Our code and dataset: https://github.com/Jhhuangkay/Query-controllable-Video-Summarization.Comment: This paper is accepted by ACM International Conference on Multimedia Retrieval (ICMR), 202

A Framework for Augmenting Building Performance Models Using Machine Learning and Immersive Virtual Environment

Building performance models (BPMs), such as building energy simulation models, have been widely used in building design. Existing BPMs are mainly derived using data from existing buildings. They may not be able to effectively address human-building interactions and lack the capability to address specific contextual factors in buildings under design. The lack of such capability often contributes to the existence of building performance discrepancies, i.e., differences between predicted performance during design and the actual performance. To improve the prediction accuracy of existing BPMs, a computational framework is developed in this dissertation. It combines an existing BPM with context-aware design-specific data involving human-building interactions in new designs by using a machine learning approach. Immersive virtual environments (IVEs) are used to acquire data describing design-specific human-building interactions, a machine learning technique is used to combine data obtained from an existing BPM, and IVEs are used to generate an augmented BPM. The potential of the framework is investigated and evaluated. An artificial neural network (ANN)-based greedy algorithm combines context-aware design-specific data obtained from IVEs with an existing BPM to enhance the simulations of human-building interactions in new designs. The results of the application show the potential of the framework to improve the prediction accuracy of an existing BPM evaluated against data obtained from the physical environment. However, it lacks the ability to determine the appropriate combination between context-aware design-specific data and data of the existing BPM. Consequently, the framework is improved to have ability to determine an appropriate combination based on a specified performance target. A generative adversarial network (GAN) is used to combine context-aware design-specific data and data of an existing BPM using the performance target as guide to generate an augmented BPM. The results confirm the effectiveness of this new framework. The performance of the augmented BPMs generated using the GAN-based framework is significantly better than the updated BPMs generated using the ANN-based greedy algorithm. The framework is completed by incorporating a robustness analysis to assist investigations of robustness of the GAN regarding the uncertainty involved in the input parameters (i.e., an existing BPM and context-aware design-specific data). Overall, this dissertation shows the promising potential of the framework in enhancing performance of BPMs and reducing performance discrepancies between estimations made during design and in performance in actual buildings