Rethinking Multi-Modal Alignment in Video Question Answering from Feature and Sample Perspectives

Reasoning about causal and temporal event relations in videos is a new destination of Video Question Answering (VideoQA).The major stumbling block to achieve this purpose is the semantic gap between language and video since they are at different levels of abstraction. Existing efforts mainly focus on designing sophisticated architectures while utilizing frame- or object-level visual representations. In this paper, we reconsider the multi-modal alignment problem in VideoQA from feature and sample perspectives to achieve better performance. From the view of feature,we break down the video into trajectories and first leverage trajectory feature in VideoQA to enhance the alignment between two modalities. Moreover, we adopt a heterogeneous graph architecture and design a hierarchical framework to align both trajectory-level and frame-level visual feature with language feature. In addition, we found that VideoQA models are largely dependent on language priors and always neglect visual-language interactions. Thus, two effective yet portable training augmentation strategies are designed to strengthen the cross-modal correspondence ability of our model from the view of sample. Extensive results show that our method outperforms all the state-of-the-art models on the challenging NExT-QA benchmark, which demonstrates the effectiveness of the proposed method

FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration

To accomplish punctuation restoration, most existing methods focus on introducing extra information (e.g., part-of-speech) or addressing the class imbalance problem. Recently, large-scale transformer-based pre-trained language models (PLMS) have been utilized widely and obtained remarkable success. However, the PLMS are trained on the large dataset with marks, which may not fit well with the small dataset without marks, causing the convergence to be not ideal. In this study, we propose a Feature Fusion two-stream framework (FF2) to bridge the gap. Specifically, one stream leverages a pre-trained language model to capture the semantic feature, while another auxiliary module captures the feature at hand. We also modify the computation of multi-head attention to encourage communication among heads. Then, two features with different perspectives are aggregated to fuse information and enhance context awareness. Without additional data, the experimental results on the popular benchmark IWSLT demonstrate that FF2 achieves new SOTA performance, which verifies that our approach is effective.Comment: 5pages. arXiv admin note: substantial text overlap with arXiv:2203.1248

Reparation in evolutionary algorithms for multi-objective feature selection in large software product lines

Software Product Lines Engineering is the area of software engineering that aims to systematise the modelling, creation and improvement of groups of interconnected software systems by formally expressing possible alternative products in the form of Feature Models. Deriving a software product/system from a feature model is called Feature Configuration. Engineers select the subset of features (software components) from a feature model that suits their needs, while respecting the underlying relationships/constraints of the system–which is challenging on its own. Since there exist several (and often antagonistic) perspectives on which the quality of software could be assessed, the problem is even more challenging as it becomes a multi-objective optimisation problem. Current multi-objective feature selection in software product line approaches (e.g., SATIBEA) combine the scalability of a genetic algorithm (IBEA) with a solution reparation approach based on a SAT solver or one of its derivatives. In this paper, we propose MILPIBEA, a novel hybrid algorithm which combines IBEA with the accuracy of a mixed-integer linear programming (MILP) reparation. We show that the MILP reparation modifies fewer features from the original infeasible solutions than the SAT reparation and in a shorter time. We also demonstrate that MILPIBEA outperforms SATIBEA on average on various multi-objective performance metrics, especially on the largest feature models. The other major challenge in software engineering in general and in software product lines, in particular, is evolution. While the change in software components is common in the software engineering industry, the particular case of multi-objective optimisation of evolving software product lines is not well-tackled yet. We show that MILPIBEA is not only able to better take advantage of the evolution than SATIBEA, but it is also the one that continues to improve the quality of the solutions when SATIBEA stagnates. Overall, IBEA performs better when combined with MILP instead of SAT reparation when optimising the multi-objective feature selection in large and evolving software product lines

Deep Learning based Recommender System: A Survey and New Perspectives

With the ever-growing volume of online information, recommender systems have been an effective strategy to overcome such information overload. The utility of recommender systems cannot be overstated, given its widespread adoption in many web applications, along with its potential impact to ameliorate many problems related to over-choice. In recent years, deep learning has garnered considerable interest in many research fields such as computer vision and natural language processing, owing not only to stellar performance but also the attractive property of learning feature representations from scratch. The influence of deep learning is also pervasive, recently demonstrating its effectiveness when applied to information retrieval and recommender systems research. Evidently, the field of deep learning in recommender system is flourishing. This article aims to provide a comprehensive review of recent research efforts on deep learning based recommender systems. More concretely, we provide and devise a taxonomy of deep learning based recommendation models, along with providing a comprehensive summary of the state-of-the-art. Finally, we expand on current trends and provide new perspectives pertaining to this new exciting development of the field.Comment: The paper has been accepted by ACM Computing Surveys. https://doi.acm.org/10.1145/328502

Incremental Consistency Checking in Delta-oriented UML-Models for Automation Systems

Automation systems exist in many variants and may evolve over time in order to deal with different environment contexts or to fulfill changing customer requirements. This induces an increased complexity during design-time as well as tedious maintenance efforts. We already proposed a multi-perspective modeling approach to improve the development of such systems. It operates on different levels of abstraction by using well-known UML-models with activity, composite structure and state chart models. Each perspective was enriched with delta modeling to manage variability and evolution. As an extension, we now focus on the development of an efficient consistency checking method at several levels to ensure valid variants of the automation system. Consistency checking must be provided for each perspective in isolation, in-between the perspectives as well as after the application of a delta.Comment: In Proceedings FMSPLE 2016, arXiv:1603.0857