Big data and the SP theory of intelligence

This article is about how the "SP theory of intelligence" and its realisation in the "SP machine" may, with advantage, be applied to the management and analysis of big data. The SP system -- introduced in the article and fully described elsewhere -- may help to overcome the problem of variety in big data: it has potential as "a universal framework for the representation and processing of diverse kinds of knowledge" (UFK), helping to reduce the diversity of formalisms and formats for knowledge and the different ways in which they are processed. It has strengths in the unsupervised learning or discovery of structure in data, in pattern recognition, in the parsing and production of natural language, in several kinds of reasoning, and more. It lends itself to the analysis of streaming data, helping to overcome the problem of velocity in big data. Central in the workings of the system is lossless compression of information: making big data smaller and reducing problems of storage and management. There is potential for substantial economies in the transmission of data, for big cuts in the use of energy in computing, for faster processing, and for smaller and lighter computers. The system provides a handle on the problem of veracity in big data, with potential to assist in the management of errors and uncertainties in data. It lends itself to the visualisation of knowledge structures and inferential processes. A high-parallel, open-source version of the SP machine would provide a means for researchers everywhere to explore what can be done with the system and to create new versions of it.Comment: Accepted for publication in IEEE Acces

Bayesian Compositional Hierarchies - A Probabilistic Structure for Scene Interpretation

In high-level vision, it is often useful to organize conceptual models in compositional hierarchies. For example, models of building facades (which are used here as examples) can be described in terms of constituent parts such as balconies or window arrays which in turn may be further decomposed. While compositional hierarchies are widely used in scene interpretation, it is not clear how to model and exploit probabilistic dependencies which may exist within and between aggregates. In this contribution I present Bayesian Aggregate Hierarchies as a means to capture probabilistic dependencies in a compositional hierarchy. The formalism integrates well with object-centered representations and extends Bayesian Networks by allowing arbitrary probabilistic dependencies within aggregates. To obtain efficient inference procedures, the aggregate structure must possess abstraction properties which ensure that internal aggregate properties are only affected in accordance with the hierarchical structure. Using examples from the building domain, it is shown that probabilistic aggregate information can thus be integrated into a logic-based scene interpretation system and provide a preference measure for interpretation steps