Interpreting Spatial Language in Image Captions

The map as a tool for accessing data has become very popular in recent years, but a lot of data do not have the necessary spatial meta-data to allow for that. Some data such as photographs however have spatial information in their captions and if this could be extracted, then they could be made available via map-based interfaces. Towards this goal, we introduce a model and spatio-linguistic reasoner for interpreting the spatial information in image captions that is based upon quantitative data about spatial language use acquired directly from people. Spatial language is inherently vague, and both the model and reasoner have been designed to incorporate this vagueness at the quantitative level and not only qualitatively

Cultural and Language Influences on the Interpretation of Spatial Prepositions

Abstract Culture and language can influence the generation and interpretation of spatial language, which would impact the quality of computational spatial language processing. This paper presents three human-subject experiments aimed at investigating these potential influences on the quantitative interpretations of five spatial prepositions. We show that for the languages (English and German) and cultures investigated (Europe and United States) neither language nor culture have a significant influence

Spatial Natural Language Generation for Location Description in Photo Captions

We present a spatial natural language generation system to create captions that describe the geographical context of geo-referenced photos. An analysis of existing photo captions was used to design templates representing typical caption language patterns, while the results of human subject experiments were used to create field-based spatial models of the applicability of some commonly used spatial prepositions. The language templates are instantiated with geo-data retrieved from the vicinity of the photo locations. A human subject evaluation was used to validate and to improve the spatial language generation procedure, examples of the results of which are presented in the paper

Spatiotemporal information extraction from a historic expedition gazetteer

Historic expeditions are events that are flavored by exploratory, scientific, military or geographic characteristics. Such events are often documented in literature, journey notes or personal diaries. A typical historic expedition involves multiple site visits and their descriptions contain spatiotemporal and attributive contexts. Expeditions involve movements in space that can be represented by triplet features (location, time and description). However, such features are implicit and innate parts of textual documents. Extracting the geospatial information from these documents requires understanding the contextualized entities in the text. To this end, we developed a semi-automated framework that has multiple Information Retrieval and Natural Language Processing components to extract the spatiotemporal information from a two-volume historic expedition gazetteer. Our framework has three basic components, namely, the Text Preprocessor, the Gazetteer Processing Machine and the JAPE (Java Annotation Pattern Engine) Transducer. We used the Brazilian Ornithological Gazetteer as an experimental dataset and extracted the spatial and temporal entities from entries that refer to three expeditioners’ site visits (which took place between 1910 and 1926) and mapped the trajectory of each expedition using the extracted information. Finally, one of the mapped trajectories was manually compared with a historical reference map of that expedition to assess the reliability of our framework

Detecting geospatial location descriptions in natural language text

References to geographic locations are common in text data sources including social media and web pages. They take different forms from simple place names to relative expressions that describe location through a spatial relationship to a reference object (e.g. the house beside the Waikato River). Often complex, multi-word phrases are employed (e.g. the road and railway cross at right angles; the road in line with the canal) where spatial relationships are communicated with various parts of speech including prepositions, verbs, adverbs and adjectives. We address the problem of automatically detecting relative geospatial location descriptions, which we define as those that include spatial relation terms referencing geographic objects, and distinguishing them from non-geographical descriptions of location (e.g. the book on the table). We experiment with several methods for automated classification of text expressions, using features for machine learning that include bag of words that detect distinctive words, word embeddings that encode meanings of words and manually identified language patterns that characterise geospatial expressions. Using three data sets created for this study, we find that ensemble and meta-classifier approaches, that variously combine predictions from several other classifiers with data features, provide the best F-measure of 0.90 for detecting geospatial expressions

Spatial Relations and Natural-Language Semantics for Indoor Scenes

Over the past 15 years, there have been increased efforts to represent and communicate spatial information about entities within indoor environments. Automated annotation of information about indoor environments is needed for natural-language processing tasks, such as spatially anchoring events, tracking objects in motion, scene descriptions, and interpretation of thematic places in relationship to confirmed locations. Descriptions of indoor scenes often require a fine granularity of spatial information about the meaning of natural-language spatial utterances to improve human-computer interactions and applications for the retrieval of spatial information. The development needs of these systems provide a rationale as to why—despite an extensive body of research in spatial cognition and spatial linguistics—it is still necessary to investigate basic understandings of how humans conceptualize and communicate about objects and structures in indoor space. This thesis investigates the alignment of conceptual spatial relations and naturallanguage (NL) semantics in the representation of indoor space. The foundation of this work is grounded in spatial information theory as well as spatial cognition and spatial linguistics. In order to better understand how to align computational models and NL expressions about indoor space, this dissertation used an existing dataset of indoor scene descriptions to investigate patterns in entity identification, spatial relations, and spatial preposition use within vista-scale indoor settings. Three human-subject experiments were designed and conducted within virtual indoor environments. These experiments investigate alignment of human-subject NL expressions for a sub-set of conceptual spatial relations (contact, disjoint, and partof) within a controlled virtual environment. Each scene was designed to focus participant attention on a single relation depicted in the scene and elicit a spatial preposition term(s) to describe the focal relationship. The major results of this study are the identification of object and structure categories, spatial relationships, and patterns of spatial preposition use in the indoor scene descriptions that were consistent across both open response, closed response and ranking type items. There appeared to be a strong preference for describing scene objects in relation to the structural objects that bound the room depicted in the indoor scenes. Furthermore, for each of the three relations (contact, disjoint, and partof), a small set of spatial prepositions emerged that were strongly preferred by participants at statistically significant levels based on the overall frequency of response, image sorting, and ranking judgments. The use of certain spatial prepositions to describe relations between room structures suggests there may be differences in how indoor vista-scale space is understood in relation to tabletop and geographic scales. Finally, an indoor scene description corpus was developed as a product of this work, which should provide researchers with new human-subject based datasets for training NL algorithms used to generate more accurate and intuitive NL descriptions of indoor scenes

Context-sensitive interpretation of natural language location descriptions : a thesis submitted in partial fulfilment of the requirements for the award of Doctor of Philosophy in Information Technology at Massey University, Auckland, New Zealand

People frequently describe the locations of objects using natural language. Location descriptions may be either structured, such as 26 Victoria Street, Auckland, or unstructured. Relative location descriptions (e.g., building near Sky Tower) are a common form of unstructured location description, and use qualitative terms to describe the location of one object relative to another (e.g., near, close to, in, next to). Understanding the meaning of these terms is easy for humans, but much more difficult for machines since the terms are inherently vague and context sensitive. In this thesis, we study the semantics (or meaning) of qualitative, geospatial relation terms, specifically geospatial prepositions. Prepositions are one of the most common forms of geospatial relation term, and they are commonly used to describe the location of objects in the geographic (geospatial) environment, such as rivers, mountains, buildings, and towns. A thorough understanding of the semantics of geospatial relation terms is important because it enables more accurate automated georeferencing of text location descriptions than use of place names only. Location descriptions that use geospatial prepositions are found in social media, web sites, blogs, and academic reports, and georeferencing can allow mapping of health, disaster and biological data that is currently inaccessible to the public. Such descriptions have unstructured format, so, their analysis is not straightforward. The specific research questions that we address are: RQ1. Which geospatial prepositions (or groups of prepositions) and senses are semantically similar? RQ2. Is the role of context important in the interpretation of location descriptions? RQ3. Is the object distance associated with geospatial prepositions across a range of geospatial scenes and scales accurately predictable using machine learning methods? RQ4. Is human annotation a reliable form of annotation for the analysis of location descriptions? To address RQ1, we determine the nature and degree of similarity among geospatial prepositions by analysing data collected with a human subjects experiment, using clustering, extensional mapping and t-stochastic neighbour embedding (t-SNE) plots to form a semantic similarity matrix. In addition to calculating similarity scores among prepositions, we identify the senses of three groups of geospatial prepositions using Venn diagrams, t-sne plots and density-based clustering, and define the relationships between the senses. Furthermore, we use two text mining approaches to identify the degree of similarity among geospatial prepositions: bag of words and GloVe embeddings. By using these methods and further analysis, we identify semantically similar groups of geospatial prepositions including: 1- beside, close to, near, next to, outside and adjacent to; 2- across, over and through and 3- beyond, past, by and off. The prepositions within these groups also share senses. Through is recognised as a specialisation of both across and over. Proximity and adjacency prepositions also have similar senses that express orientation and overlapping relations. Past, off and by share a proximal sense but beyond has a different sense from these, representing on the other side. Another finding is the more frequent use of the preposition close to for pairs of linear objects than near, which is used more frequently for non-linear ones. Also, next to is used to describe proximity more than touching (in contrast to other prepositions like adjacent to). Our application of text mining to identify semantically similar prepositions confirms that a geospatial corpus (NCGL) provides a better representation of the semantics of geospatial prepositions than a general corpus. Also, we found that GloVe embeddings provide adequate semantic similarity measures for more specialised geospatial prepositions, but less so for those that have more generalised applications and multiple senses. We explore the role of context (RQ2) by studying three sites that vary in size, nature, and context in London: Trafalgar Square, Buckingham Palace, and Hyde Park. We use the Google search engine to extract location descriptions that contain these three sites with 9 different geospatial prepositions (in, on, at, next to, close to, adjacent to, near, beside, outside) and calculate their acceptance profiles (the profile of the use of a preposition at different distances from the reference object) and acceptance thresholds (maximum distance from a reference object at which a preposition can acceptably be used). We use these to compare prepositions, and to explore the influence of different contexts. Our results show that near, in and outside are used for larger distances, while beside, adjacent to and at are used for smaller distances. Also, the acceptance threshold for close to is higher than for other proximity/adjacency prepositions such as next to, adjacent to and beside. The acceptance threshold of next to is larger than adjacent to, which confirms the findings in ‎Chapter 2 which identifies next to describing a proximity rather than touching spatial relation. We also found that relatum characteristics such as image schema affect the use of prepositions such as in, on and at. We address RQ3 by developing a machine learning regression model (using the SMOReg algorithm) to predict the distance associated with use of geospatial prepositions in specific expressions. We incorporate a wide range of input variables including the similarity matrix of geospatial prepositions (RQ1); preposition senses; semantic information in the form of embeddings; characteristics of the located and reference objects in the expression including their liquidity/solidity, scale and geometry type and contextual factors such as the density of features of different types in the surrounding area. We evaluate the model on two different datasets with 25% improvement against the best baseline respectively. Finally, we consider the importance of annotation of geospatial location descriptions (RQ4). As annotated data is essential for the successful study of automated interpretation of natural language descriptions, we study the impact and accuracy of human annotation on different geospatial elements. Agreement scores show that human annotators can annotate geospatial relation terms (e.g., geospatial prepositions) with higher agreement than other geospatial elements. This thesis advances understanding of the semantics of geospatial prepositions, particularly considering their semantic similarity and the impact of context on their interpretation. We quantify the semantic similarity of a set of 24 geospatial prepositions; identify senses and the relationships among them for 13 geospatial prepositions; compare the acceptance thresholds of 9 geospatial prepositions and describe the influence of context on them; and demonstrate that richer semantic and contextual information can be incorporated in predictive models to interpret relative geospatial location descriptions more accurately

Proposição de um conjunto de relações espaciais para tarefas de descrições espaciais

Orientadora: Profª. Drª Luciene Stamato DelazariTese (doutorado) - Universidade Federal do Paraná, Setor de Ciências da Terra, Programa de Pós-Graduação em Ciências Geodésicas. Defesa : Curitiba, 26/02/2019Inclui referências: p. 146-154Resumo: Descrever espacialmente objetos ou ambientes é uma tarefa cotidiana e natural presente na rotina dos indivíduos. Frequentemente, tais descrições são realizadas através da Linguagem Natural (LN), tanto a falada como a escrita. A descrição espacial é composta por expressões como "a padaria fica ao lado do supermercado" ou "a padaria fica perto do supermercado". Esse tipo de descrição é a forma predominante de comunicação espacial entre os indivíduos é conhecido como expressão locativa ou locacionais. As expressões locativas são compostas por três elementos fundamentais: o elemento a ser localizado (locatum), ponto de referência (relatum) e a relação espacial. Os pontos de referência são elementos do ambiente que se destacam dentre os outros. A característica que torna um elemento uma referência pode ser visual (cor, tamanho), estrutural (localização proeminente) ou semântica (significado cultural, histórico ou atividade relacionada). Os vocábulos empregados para fazer associações entre o elemento a ser localizado e o ponto de referência são chamados de relações espaciais. A existência de uma constante interação do homem com o ambiente torna a LN rica em vocábulos espaciais que caracterizam o espaço, resultando em uma diversidade de tais vocábulos. A quantidade substancial e a versatilidade dos vocábulos utilizados como relações espaciais dificultam sua implementação em sistemas que buscam interpretar, processar e representar a linguagem natural. O projeto "Where Am I?", no qual esta pesquisa está inserida, tem como objetivo implementar um desses sistemas, em que seja possível converter a descrição espacial do indivíduo em uma localização geográfica. Dessa forma, o objetivo geral desta tese foi verificar se a escolha das relações espaciais é afetada pelos contextos em que se encontra o indivíduo. Os dados que possibilitaram este trabalho foram obtidos a partir de três fontes: 1) teste exploratório em ambientes desconhecidos, 2) testes exploratórios em ambiente conhecido e 3) ocorrências registradas pelo Corpo de Bombeiros Militar de Santa Catarina (CBM-SC). Os resultados alcançados permitiram a realização do experimento final em dois contextos: cotidiano e estresse. O experimento final mostrou que o emprego das relações espaciais se deu de maneiras diferentes em ambos os contextos, uma vez que a quantidade e a variabilidade dos vocábulos empregados foram distintas. Apesar disso, foi perceptível o uso de um conjunto de vocábulos representativo dos demais. Isso presume, que apesar do contexto influenciar na escolha das relações espaciais, o conjunto obtido de relações espaciais é satisfatório para serem empregados nos contextos experimentados nesta tese. Sendo assim, os experimentos realizados possibilitaram a comprovação da hipótese levantada nesta tese, e ainda que o conjunto de relações espaciais representativo das demais pode auxiliar em pesquisas futuras que objetivam a implementação da LN em sistemas de localização geográfica. Palavras-chaves: Linguagem Natural; Pontos de Referência; Relações espaciais; Descrição espacial.Abstract: Spatially describing objects or environments is a daily and natural task present in the routine of individuals. Frequently, such descriptions are performed through the Natural Language (NL), both spoken and written. The spatial description is composed of expressions such as "the bakery is next to the supermarket" or "the bakery is near the supermarket". This type of description is the predominant form of spatial communication between individuals is known as a locative or locational expression. Locative expressions are composed of three fundamental elements: the feature to be located (locatum), the reference point (relatum) and the spatial relation. Landmarks are elements of the environment that stand out from others. A characteristic that makes an element a reference can be visual (structural, visual), structural (structural) or semantic (cultural meaning). The terms used to make associations between the element to be located and the reference point are called spatial relations. The existence of a constant interaction of the individuals with the environment makes NL rich in spatial terms that characterize space, resulting in a diversity of such words. The substantial quantity and versatility of the terms used as spatial relations make it difficult to implement them in systems that seek to interpret, process and represent natural language. The project called "Where am I?", in which this research is inserted, aims to implement one of these systems that possible to convert the spatial description of the individual into a geographic location. Thus, the general objective of this thesis was to verify if the choice of spatial relations is affected by the contexts in which the individual is. The data that enabled this work were obtained from three sources: 1) exploratory test in unknown environments, 2) exploratory tests in a known environment, and 3) occurrences recorded by the Santa Catarina Military Fire Brigade (CBM-SC). The results allowed the realization of the final experiment in two contexts: daily and stress. The final experiment showed that the use of spatial relations occurred in different ways in both contexts, since the quantity and variability of the terms used were different. Despite this, the use of a set of words representative of the others was perceptible. This assumes that, although the context influences the choice of spatial relations, the set of spatial relations is adequate to be used in the contexts experienced in this thesis. Thus, the experiments carried out allowed to prove the hypothesis raised in this thesis, and even though the set of spatial relations representative of the others can help in future research that aim at the implementation of NL in geographic location systems. Keywords: Natural Language; Reference points; Spatial relationships; Spatial description

Real-Time Event Analysis and Spatial Information Extraction From Text Using Social Media Data

Since the advent of websites that enable users to participate and interact with each other by sharing content in different forms, a plethora of possibly relevant information is at scientists\u27 fingertips. Consequently, this thesis elaborates on two distinct approaches to extract valuable information from social media data and sketches out the potential joint use case in the domain of natural disasters