Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation

Video scene graph generation (VidSGG) aims to identify objects in visual scenes and infer their relationships for a given video. It requires not only a comprehensive understanding of each object scattered on the whole scene but also a deep dive into their temporal motions and interactions. Inherently, object pairs and their relationships enjoy spatial co-occurrence correlations within each image and temporal consistency/transition correlations across different images, which can serve as prior knowledge to facilitate VidSGG model learning and inference. In this work, we propose a spatial-temporal knowledge-embedded transformer (STKET) that incorporates the prior spatial-temporal knowledge into the multi-head cross-attention mechanism to learn more representative relationship representations. Specifically, we first learn spatial co-occurrence and temporal transition correlations in a statistical manner. Then, we design spatial and temporal knowledge-embedded layers that introduce the multi-head cross-attention mechanism to fully explore the interaction between visual representation and the knowledge to generate spatial- and temporal-embedded representations, respectively. Finally, we aggregate these representations for each subject-object pair to predict the final semantic labels and their relationships. Extensive experiments show that STKET outperforms current competing algorithms by a large margin, e.g., improving the mR@50 by 8.1%, 4.7%, and 2.1% on different settings over current algorithms.Comment: Technical Repor

Spatial Thinking in the Engineering Curriculum: an Investigation of the Relationship Between Problem Solving and Spatial Skills Among Engineering Students.

Long considered a primary factor of intelligence, spatial ability has been shown to correlate strongly with success in engineering education, yet is rarely included as a learning outcome in engineering programmes. A clearer understanding of how and why spatial ability impacts on performance in science, technology, engineering and mathematics (STEM) subjects would allow educators to determine if spatial skills development merits greater priority in STEM curricula. The aim of this study is to help inform that debate by shedding new light on the role of spatial thinking in STEM learning and allow teaching practice and curriculum design to be informed by evidence based research. A cross cutting theme in STEM education – problem solving – is examined with respect to its relationship with spatial ability. Several research questions were addressed that related to the role and relevance of spatial ability to first year engineering education and, more specifically, the manner in which spatial ability is manifest in the representation and solution of word story problems in mathematics. Working with samples of engineering students in Ireland and the United States, data were collected in the form of responses to spatial ability tests and problem solving exercises in the areas of mathematics and electric circuits. Following a pilot study to select and refine a set of mathematical story problems a mixed methods design was followed in which data were first analysed using quantitative methods to highlight phenomena that were then explored using an interpretive approach. With regard to engineering education in general, it was found that spatial ability cannot be assumed to improve as students progress through an engineering programme and that spatial ability is highly relevant to assessments that require reasoning about concepts, novel scenarios and problems but can remain hidden in overall course grades possibly due to an emphasis on assessing rote learning. With regard to problem solving, spatial ability was found to have a significant relationship with the problem representation step but not with the solution step. Those with high levels of spatial ability were more able to apply linguistic and schematic knowledge to the problem representation phase which led to higher success rates in translating word statements to mathematical form

Estimating the Topological Structure of Driver Spatial Knowledge

Spatial knowledge has long been recognised as playing an important role in influencing transportation flows in urban areas. The limits imposed by bounded knowledge of space restrict an individual’s decision-making ability, a factor that naturally influences the evolution of collective patterns of behaviour. In this paper, a new methodology for the estimation of the extents of the bounded spatial knowledge of vehicular drivers is outlined, an approach that models the relationship between road transportation activity and human cognition of space. In describing the methodology, the paper begins in outlining a topological representation of urban space that aims to capture the role of salient locations in the construction of spatial knowledge. In the second stage, a spatial interaction model is specified that estimates the relationship between home locations and nearby areas of leisure activity. In the final stage, the models of space and activity are integrated within a framework for spatial learning, enabling the estimation of the growth of spatial knowledge over time. The approach is applied to London, United Kingdom, and the spatial and temporal processes of extension in spatial knowledge are discussed. The paper concludes by outlining the potential for development and application of the model, as well as the natural limitations inherent in this approach

Pareto Adversarial Robustness: Balancing Spatial Robustness and Sensitivity-based Robustness

Adversarial robustness, which mainly contains sensitivity-based robustness and spatial robustness, plays an integral part in the robust generalization. In this paper, we endeavor to design strategies to achieve universal adversarial robustness. To hit this target, we firstly investigate the less-studied spatial robustness and then integrate existing spatial robustness methods by incorporating both local and global spatial vulnerability into one spatial attack and adversarial training. Based on this exploration, we further present a comprehensive relationship between natural accuracy, sensitivity-based and different spatial robustness, supported by the strong evidence from the perspective of robust representation. More importantly, in order to balance these mutual impacts of different robustness into one unified framework, we incorporate \textit{Pareto criterion} into the adversarial robustness analysis, yielding a novel strategy called \textit{Pareto Adversarial Training} towards universal robustness. The resulting Pareto front, the set of optimal solutions, provides the set of optimal balance among natural accuracy and different adversarial robustness, shedding light on solutions towards universal robustness in the future. To the best of our knowledge, we are the first to consider the universal adversarial robustness via multi-objective optimization

From representation to emergence: complexity's challenge to the epistemology of schooling

In modern,Western societies the purpose of schooling is to ensure that school-goers acquire knowledge of pre-existing practices, events, entities and so on.The knowledge that is learned is then tested to see if the learner has acquired a correct or adequate understanding of it. For this reason, it can be argued that schooling is organised around a representational epistemology: one which holds that knowledge is an accurate representation of something that is separate from knowledge itself. Since the object of knowledge is assumed to exist separately from the knowledge itself, this epistemology can also be considered ‘spatial.’ In this paper we show how ideas from complexity have challenged the ‘spatial epistemology’ of representation and we explore possibilities for an alternative ‘temporal’ understanding of knowledge in its relationship to reality. In addition to complexity, our alternative takes its inspiration from Deweyan ‘transactional realism’ and deconstruction. We suggest that ‘knowledge’ and ‘reality’ should not be understood as separate systems which somehow have to be brought into alignment with each other, but that they are part of the same emerging complex system which is never fully ‘present’ in any (discrete) moment in time. This not only introduces the notion of time into our understanding of the relationship between knowledge and reality, but also points to the importance of acknowledging the role of the ‘unrepresentable’ or ‘incalculable’. With this understanding knowledge reaches us not as something we receive but as a response, which brings forth new worlds because it necessarily adds something (which was not present anywhere before it appeared) to what came before. This understanding of knowledge suggests that the acquisition of curricular content should not be considered an end in itself. Rather, curricular content should be used to bring forth that which is incalculable from the perspective of the present. The epistemology of emergence therefore calls for a switch in focus for curricular thinking, away from questions about presentation and representation and towards questions about engagement and response