Semantics-Space-Time Cube. A Conceptual Framework for Systematic Analysis of Texts in Space and Time

We propose an approach to analyzing data in which texts are associated with spatial and temporal references with the aim to understand how the text semantics vary over space and time. To represent the semantics, we apply probabilistic topic modeling. After extracting a set of topics and representing the texts by vectors of topic weights, we aggregate the data into a data cube with the dimensions corresponding to the set of topics, the set of spatial locations (e.g., regions), and the time divided into suitable intervals according to the scale of the planned analysis. Each cube cell corresponds to a combination (topic, location, time interval) and contains aggregate measures characterizing the subset of the texts concerning this topic and having the spatial and temporal references within these location and interval. Based on this structure, we systematically describe the space of analysis tasks on exploring the interrelationships among the three heterogeneous information facets, semantics, space, and time. We introduce the operations of projecting and slicing the cube, which are used to decompose complex tasks into simpler subtasks. We then present a design of a visual analytics system intended to support these subtasks. To reduce the complexity of the user interface, we apply the principles of structural, visual, and operational uniformity while respecting the specific properties of each facet. The aggregated data are represented in three parallel views corresponding to the three facets and providing different complementary perspectives on the data. The views have similar look-and-feel to the extent allowed by the facet specifics. Uniform interactive operations applicable to any view support establishing links between the facets. The uniformity principle is also applied in supporting the projecting and slicing operations on the data cube. We evaluate the feasibility and utility of the approach by applying it in two analysis scenarios using geolocated social media data for studying people's reactions to social and natural events of different spatial and temporal scales

Resonant Neural Dynamics of Speech Perception

What is the neural representation of a speech code as it evolves in time? How do listeners integrate temporally distributed phonemic information across hundreds of milliseconds, even backwards in time, into coherent representations of syllables and words? What sorts of brain mechanisms encode the correct temporal order, despite such backwards effects, during speech perception? How does the brain extract rate-invariant properties of variable-rate speech? This article describes an emerging neural model that suggests answers to these questions, while quantitatively simulating challenging data about audition, speech and word recognition. This model includes bottom-up filtering, horizontal competitive, and top-down attentional interactions between a working memory for short-term storage of phonetic items and a list categorization network for grouping sequences of items. The conscious speech and word recognition code is suggested to be a resonant wave of activation across such a network, and a percept of silence is proposed to be a temporal discontinuity in the rate with which such a resonant wave evolves. Properties of these resonant waves can be traced to the brain mechanisms whereby auditory, speech, and language representations are learned in a stable way through time. Because resonances are proposed to control stable learning, the model is called an Adaptive Resonance Theory, or ART, model.Air Force Office of Scientific Research (F49620-01-1-0397); National Science Foundation (IRI-97-20333); Office of Naval Research (N00014-01-1-0624)

Detecting multineuronal temporal patterns in parallel spike trains

We present a non-parametric and computationally efficient method that detects spatiotemporal firing patterns and pattern sequences in parallel spike trains and tests whether the observed numbers of repeating patterns and sequences on a given timescale are significantly different from those expected by chance. The method is generally applicable and uncovers coordinated activity with arbitrary precision by comparing it to appropriate surrogate data. The analysis of coherent patterns of spatially and temporally distributed spiking activity on various timescales enables the immediate tracking of diverse qualities of coordinated firing related to neuronal state changes and information processing. We apply the method to simulated data and multineuronal recordings from rat visual cortex and show that it reliably discriminates between data sets with random pattern occurrences and with additional exactly repeating spatiotemporal patterns and pattern sequences. Multineuronal cortical spiking activity appears to be precisely coordinated and exhibits a sequential organization beyond the cell assembly concept

Visualisation of Parallel Data Streams with Temporal Mosaics

Despite its popularity and widespread use, timeline visualisation suffers from shortcomings which limit its use for displaying multiple data streams when the number of streams increases to more than a handful. This paper presents the TemporalMosaic technique for visualisation of parallel time-based streams which addresses some of these shortcomings. Temporal mosaics provide a compact way of representing parallel streams of events by allocating a fixed drawing area to time intervals and partitioning that area according to the number of concurrent events. A user study is presented which compares this technique to a standard timeline representation technique in which events are depicted as horizontal bars and multiple streams are drawn in parallel along a vertical axis. Results of this user study show that users of the temporal mosaic visualisation perform significantly better at detecting concurrency, interval overlaps and inactivity than users of standard timelines

Occupational Fraud Detection Through Visualization

Occupational fraud affects many companies worldwide causing them economic loss and liability issues towards their customers and other involved entities. Detecting internal fraud in a company requires significant effort and, unfortunately cannot be entirely prevented. The internal auditors have to process a huge amount of data produced by diverse systems, which are in most cases in textual form, with little automated support. In this paper, we exploit the advantages of information visualization and present a system that aims to detect occupational fraud in systems which involve a pair of entities (e.g., an employee and a client) and periodic activity. The main visualization is based on a spiral system on which the events are drawn appropriately according to their time-stamp. Suspicious events are considered those which appear along the same radius or on close radii of the spiral. Before producing the visualization, the system ranks both involved entities according to the specifications of the internal auditor and generates a video file of the activity such that events with strong evidence of fraud appear first in the video. The system is also equipped with several different visualizations and mechanisms in order to meet the requirements of an internal fraud detection system

Audio-visual foreground extraction for event characterization

This paper presents a new method able to integrate audio and visual information for scene analysis in a typical surveillance scenario, using only one camera and one monaural microphone. Visual information is analyzed by a standard visual background/foreground (BG/FG) modelling module, enhanced with a novelty detection stage, and coupled with an audio BG/FG modelling scheme. The audiovisual association is performed on-line, by exploiting the concept of synchrony. Experimental tests carrying out classification and clustering of events show all the potentialities of the proposed approach, also in comparison with the results obtained by using the single modalities

Content-based Video Retrieval by Integrating Spatio-Temporal and Stochastic Recognition of Events

As amounts of publicly available video data grow the need to query this data efficiently becomes significant. Consequently content-based retrieval of video data turns out to be a challenging and important problem. We address the specific aspect of inferring semantics automatically from raw video data. In particular, we introduce a new video data model that supports the integrated use of two different approaches for mapping low-level features to high-level concepts. Firstly, the model is extended with a rule-based approach that supports spatio-temporal formalization of high-level concepts, and then with a stochastic approach. Furthermore, results on real tennis video data are presented, demonstrating the validity of both approaches, as well us advantages of their integrated us

AMPTE/CCE‐SCATHA simultaneous observations of substorm‐associated magnetic fluctuations

This study examines substorm-associated magnetic field fluctuations observed by the AMPTE/CCE and SCATHA satellites in the near-Earth tail. Three tail reconfiguration events are selected, one event on August 28, 1986, and two consecutive events on August 30, 1986. The fractal analysis was applied to magnetic field measurements of each satellite. The result indicates that (1) the amplitude of the fluctuation of the north-south magnetic component is larger, though not overwhelmingly, than the amplitudes of the other two components and (2) the magnetic fluctuations do have a characteristic timescale, which is several times the proton gyroperiod. In the examined events the satellite separation was less than 10 times the proton gyroradius. Nevertheless, the comparison between the AMPTE/CCE and SCATHA observations indicates that (3) there was a noticeable time delay between the onsets of the magnetic fluctuations at the two satellite positions, which is too long to ascribe to the propagation of a fast magnetosonic wave, and (4) the coherence of the magnetic fluctuations was low in the August 28, 1986, event and the fluctuations had different characteristic timescales in the first event of August 30, 1986, whereas some similarities can be found for the second event of August 30, 1986. Result 1 indicates that perturbation electric currents associated with the magnetic fluctuations tend to flow parallel to the tail current sheet and are presumably related to the reduction of the tail current intensity. Results 2 and 3 suggest that the excitation of the magnetic fluctuations and therefore the trigger of the tail current disruption is a kinetic process in which ions play an important role. It is inferred from results 3 and 4 that the characteristic spatial scale of the associated instability is of the order of the proton gyroradius or even shorter, and therefore the tail current disruption is described as a system of chaotic filamentary electric currents. However, result 4 suggests that the nature of the tail current disruption can vary from event to event