Multi-Modal Spatial Querying

This project investigates the use of two concurrent communication channels, graphics and speech, to achieve a successful interaction between a person and a geographic information system (GIS). The objective is to construct a multi-modal spatial query language in which users interact with a geographic database by drawing sketches of the desired configuration, while simultaneously talking about the spatial objects and the spatial relations drawn. This study will increase our understanding of multi-modal spatial interactions, and will lead to improved strategies for intelligent integration and processing of such multi-modal spatial queries in a GIS. The key to this interaction is the exploitation of complementary or redundant information present in both graphical and verbal descriptions of the same spatial scenes. A multiple-resolution model of spatial relations is used to capture the essential aspects of a sketch and its corresponding verbal description. The model stresses topological properties, such as containment and neighborhood, and considers metrical properties, such as distance and directions, as refinements where necessary. This model enables the retrieval of similar, not only exact, matches between a spatial query and a geographic database. Such new methods of multi-modal spatial querying and spatial similarity retrieval will empower experts as well as novice users to perform easier spatial searches, ultimately providing new user communities access to spatial databases

Maine Perspective, v 12, i 1

The Maine Perspective, a publication for the University of Maine, was a campus newsletter produced by the Department of Public Affairs which eventually transformed into the Division of Marketing and Communication. Regular columns included the UM Calendar, Ongoing Events, People in Perspective, Look Who\u27s on Campus, In Focus, and Along the Mall. The weekly newsletter also included position openings on campus as well as classified ads. Reporting in this issue includes coverage of a 15% increase in grant funding at UMaine; an announced increase in graduate student stipends, though the allotment remains among the lowest of New England\u27s land grant institutions; research into the foundation of friendship relationships in childhood; and an inventory of major capital improvement projects slated for campus

Arrow Symbols: Theory for Interpretation

People often sketch diagrams when they communicate successfully among each other. Such an intuitive collaboration would also be possible with computers if the machines understood the meanings of the sketches. Arrow symbols are a frequent ingredient of such sketched diagrams. Due to the arrows’ versatility, however, it remains a challenging problem to make computers distinguish the various semantic roles of arrow symbols. The solution to this problem is highly desirable for more effective and user-friendly pen-based systems. This thesis, therefore, develops an algorithm for deducing the semantic roles of arrow symbols, called the arrow semantic interpreter (ASI). The ASI emphasizes the structural patterns of arrow-containing diagrams, which have a strong influence on their semantics. Since the semantic roles of arrow symbols are assigned to individual arrow symbols and sometimes to the groups of arrow symbols, two types of the corresponding structures are introduced: the individual structure models the spatial arrangement of components around each arrow symbol and the inter-arrow structure captures the spatial arrangement of multiple arrow symbols. The semantic roles assigned to individual arrow symbols are classified into orientation, behavioral description, annotation, and association, and the formats of individual structures that correspond to these four classes are identified. The result enables the derivation of the possible semantic roles of individual arrow symbols from their individual structures. In addition, for the diagrams with multiple arrow symbols, the patterns of their inter-arrow structures are exploited to detect the groups of arrow symbols that jointly have certain semantic roles, as well as the nesting relations between the arrow symbols. The assessment shows that for 79% of sample arrow symbols the ASI successfully detects their correct semantic roles, even though the average number of the ASI’s interpretations is only 1.31 per arrow symbol. This result indicates that the structural information is highly useful for deriving the reliable interpretations of arrow symbols

A Qualitative Representation of Spatial Scenes in R2 with Regions and Lines

Regions and lines are common geographic abstractions for geographic objects. Collections of regions, lines, and other representations of spatial objects form a spatial scene, along with their relations. For instance, the states of Maine and New Hampshire can be represented by a pair of regions and related based on their topological properties. These two states are adjacent (i.e., they meet along their shared boundary), whereas Maine and Florida are not adjacent (i.e., they are disjoint). A detailed model for qualitatively describing spatial scenes should capture the essential properties of a configuration such that a description of the represented objects and their relations can be generated. Such a description should then be able to reproduce a scene in a way that preserves all topological relationships, but without regards to metric details. Coarse approaches to qualitative spatial reasoning may underspecify certain relations. For example, if two objects meet, it is unclear if they meet along an edge, at a single point, or multiple times along their boundaries. Where the boundaries of spatial objects converge, this is called a spatial intersection. This thesis develops a model for spatial scene descriptions primarily through sequences of detailed spatial intersections and object containment, capturing how complex spatial objects relate. With a theory of complex spatial scenes developed, a tool that will automatically generate a formal description of a spatial scene is prototyped, enabling the described objects to be analyzed. The strengths and weaknesses of the provided model will be discussed relative to other models of spatial scene description, along with further refinements

Qualitative Spatial Query Processing : Towards Cognitive Geographic Information Systems

For a long time, Geographic Information Systems (GISs) have been used by GIS-experts to perform numerous tasks including way finding, mapping, and querying geo-spatial databases. The advancement of Web 2.0 technologies and the development of mobile-based device applications present an excellent opportunity to allow the public -non-expert users- to access information of GISs. However, the interfaces of GISs were mainly designed and developed based on quantitative values of spatial databases to serve GIS-experts, whereas non-expert users usually prefer a qualitative approach to interacting with GISs. For example, humans typically resort to expressions such as the building is near a riverbank or there is a restaurant inside a park which qualitatively locate the spatial entity with respect to another. In other words, the users' interaction with current GISs is still not intuitive and not efficient. This dissertation thusly aims at enabling users to intuitively and efficiently search spatial databases of GISs by means of qualitative relations or terms such as left, north of, or inside. We use these qualitative relations to formalise so-called Qualitative Spatial Queries (QSQs). Aside from existing topological models, we integrate distance and directional qualitative models into Spatial Data-Base Management Systems (SDBMSs) to allow the qualitative and intuitive formalism of queries in GISs. Furthermore, we abstract binary Qualitative Spatial Relations (QSRs) covering the aforementioned aspects of space from the database objects. We store the abstracted QSRs in a Qualitative Spatial Layer (QSL) that we extend into current SDBMSs to avoid the additional cost of the abstraction process when dealing with every single query. Nevertheless, abstracting the QSRs of QSL results in a high space complexity in terms of qualitative representations

Semantic Similarity of Spatial Scenes

The formalization of similarity in spatial information systems can unleash their functionality and contribute technology not only useful, but also desirable by broad groups of users. As a paradigm for information retrieval, similarity supersedes tedious querying techniques and unveils novel ways for user-system interaction by naturally supporting modalities such as speech and sketching. As a tool within the scope of a broader objective, it can facilitate such diverse tasks as data integration, landmark determination, and prediction making. This potential motivated the development of several similarity models within the geospatial and computer science communities. Despite the merit of these studies, their cognitive plausibility can be limited due to neglect of well-established psychological principles about properties and behaviors of similarity. Moreover, such approaches are typically guided by experience, intuition, and observation, thereby often relying on more narrow perspectives or restrictive assumptions that produce inflexible and incompatible measures. This thesis consolidates such fragmentary efforts and integrates them along with novel formalisms into a scalable, comprehensive, and cognitively-sensitive framework for similarity queries in spatial information systems. Three conceptually different similarity queries at the levels of attributes, objects, and scenes are distinguished. An analysis of the relationship between similarity and change provides a unifying basis for the approach and a theoretical foundation for measures satisfying important similarity properties such as asymmetry and context dependence. The classification of attributes into categories with common structural and cognitive characteristics drives the implementation of a small core of generic functions, able to perform any type of attribute value assessment. Appropriate techniques combine such atomic assessments to compute similarities at the object level and to handle more complex inquiries with multiple constraints. These techniques, along with a solid graph-theoretical methodology adapted to the particularities of the geospatial domain, provide the foundation for reasoning about scene similarity queries. Provisions are made so that all methods comply with major psychological findings about people’s perceptions of similarity. An experimental evaluation supplies the main result of this thesis, which separates psychological findings with a major impact on the results from those that can be safely incorporated into the framework through computationally simpler alternatives

Um arcabouço multimodal para geocodificação de objetos digitais

Orientador: Ricardo da Silva TorresTese (doutorado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Informação geográfica é usualmente encontrada em objetos digitais (como documentos, imagens e vídeos), sendo de grande interesse utilizá-la na implementação de diferentes serviços. Por exemplo, serviços de navegação baseados em mapas e buscas geográficas podem se beneficiar das localizações geográficas associadas a objetos digitais. A implementação destes serviços, no entanto, demanda o uso de coleções de dados geocodificados. Este trabalho estuda a combinação de conteúdo textual e visual para geocodificar objetos digitais e propõe um arcabouço de agregação de listas para geocodificação multimodal. A informação textual e visual de vídeos e imagens é usada para definir listas ordenadas. Em seguida, elas são combinadas e a nova lista ordenada resultante é usada para definir a localização geográfica de vídeos e imagens. Uma arquitetura que implementa essa proposta foi projetada de modo que módulos específicos para cada modalidade (e.g., textual ou visual) possam ser aperfeiçoados independentemente. Outro componente é o módulo de fusão responsável pela combinação das listas ordenadas definidas por cada modalidade. Outra contribuição deste trabalho é a proposta de uma nova medida de avaliação da efetividade de métodos de geocodificação chamada Weighted Average Score (WAS). Ela é baseada em ponderações de distâncias que permitem avaliar a efetividade de uma abordagem, considerando todos os resultados de geocodificação das amostras de teste. O arcabouço proposto foi validado em dois contextos: desafio Placing Task da iniciativa MediaEval 2012, que consiste em atribuir, automaticamente, coordenadas geográficas a vídeos; e geocodificação de fotos de prédios da Virginia Tech (VT) nos EUA. No contexto do desafio Placing Task, os resultados mostram como nossa abordagem melhora a geocodificação em comparação a métodos que apenas contam com uma modalidade (sejam descritores textuais ou visuais). Nós mostramos ainda que a proposta multimodal produziu resultados comparáveis às melhores submissões que também não usavam informações adicionais além daquelas disponibilizadas na base de treinamento. Em relação à geocodificação das fotos de prédios da VT, os experimentos demostraram que alguns dos descritores visuais locais produziram resultados efetivos. A seleção desses descritores e sua combinação melhoraram esses resultados quando a base de conhecimento tinha as mesmas características da base de testeAbstract: Geographical information is often enclosed in digital objects (like documents, images, and videos) and its use to support the implementation of different services is of great interest. For example, the implementation of map-based browser services and geographic searches may take advantage of geographic locations associated with digital objects. The implementation of such services, however, demands the use of geocoded data collections. This work investigates the combination of textual and visual content to geocode digital objects and proposes a rank aggregation framework for multimodal geocoding. Textual and visual information associated with videos and images are used to define ranked lists. These lists are later combined, and the new resulting ranked list is used to define appropriate locations. An architecture that implements the proposed framework is designed in such a way that specific modules for each modality (e.g., textual and visual) can be developed and evolved independently. Another component is a data fusion module responsible for combining seamlessly the ranked lists defined for each modality. Another contribution of this work is related to the proposal of a new effectiveness evaluation measure named Weighted Average Score (WAS). The proposed measure is based on distance scores that are combined to assess how effective a designed/tested approach is, considering its overall geocoding results for a given test dataset. We validate the proposed framework in two contexts: the MediaEval 2012 Placing Task, whose objective is to automatically assign geographical coordinates to videos; and the task of geocoding photos of buildings from Virginia Tech (VT), USA. In the context of Placing Task, obtained results show how our multimodal approach improves the geocoding results when compared to methods that rely on a single modality (either textual or visual descriptors). We also show that the proposed multimodal approach yields comparable results to the best submissions to the Placing Task in 2012 using no additional information besides the available development/training data. In the context of the task of geocoding VT building photos, performed experiments demonstrate that some of the evaluated local descriptors yield effective results. The descriptor selection criteria and their combination improved the results when the used knowledge base has the same characteristics of the test setDoutoradoCiência da ComputaçãoDoutora em Ciência da Computaçã

Semi-supervised image classification based on a multi-feature image query language

The area of Content-Based Image Retrieval (CBIR) deals with a wide range of research disciplines. Being closely related to text retrieval and pattern recognition, the probably most serious issue to be solved is the so-called \semantic gap". Except for very restricted use-cases, machines are not able to recognize the semantic content of digital images as well as humans. This thesis identifies the requirements for a crucial part of CBIR user interfaces, a multimedia-enabled query language. Such a language must be able to capture the user's intentions and translate them into a machine-understandable format. An approach to tackle this translation problem is to express high-level semantics by merging low-level image features. Two related methods are improved for either fast (retrieval) or accurate(categorization) merging. A query language has previously been developed by the author of this thesis. It allows the formation of nested Boolean queries. Each query term may be text- or content-based and the system merges them into a single result set. The language is extensible by arbitrary new feature vector plug-ins and thus use-case independent. This query language should be capable of mapping semantics to features by applying machine learning techniques; this capability is explored. A supervised learning algorithm based on decision trees is used to build category descriptors from a training set. Each resulting \query descriptor" is a feature-based description of a concept which is comprehensible and modifiable. These descriptors could be used as a normal query and return a result set with a high CBIR based precision/recall of the desired category. Additionally, a method for normalizing the similarity profiles of feature vectors has been developed which is essential to perform categorization tasks. To prove the capabilities of such queries, the outcome of a semi-supervised training session with \leave-one-object-out" cross validation is compared to a reference system. Recent work indicates that the discriminative power of the query-based descriptors is similar and is likely to be improved further by implementing more recent feature vectors.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

A Visual Tool for Querying Geographic Databases

To support users in querying geographic databases we have developed a system that lets people sketch what they are looking for. It closes the gap between user and information system because the translation of a users question into a processable query statement is delegated to the information system so that a user can focus on the actual query rather than spending time with its formulation. This system paper highlights a set of interaction methods and sketch interpretation algorithms that are necessary for pen-based querying of geographic information systems. They are part of a comprehensive prototype implementation of Spatial-Query-by-Sketch, which provides feature-based and relation-based spatial similarity retrieval