Intelligent visual media processing: when graphics meets vision

The computer graphics and computer vision communities have been working closely together in recent years, and a variety of algorithms and applications have been developed to analyze and manipulate the visual media around us. There are three major driving forces behind this phenomenon: i) the availability of big data from the Internet has created a demand for dealing with the ever increasing, vast amount of resources; ii) powerful processing tools, such as deep neural networks, provide e�ective ways for learning how to deal with heterogeneous visual data; iii) new data capture devices, such as the Kinect, bridge between algorithms for 2D image understanding and 3D model analysis. These driving forces have emerged only recently, and we believe that the computer graphics and computer vision communities are still in the beginning of their honeymoon phase. In this work we survey recent research on how computer vision techniques bene�t computer graphics techniques and vice versa, and cover research on analysis, manipulation, synthesis, and interaction. We also discuss existing problems and suggest possible further research directions

Multi-Sensory Interaction for Blind and Visually Impaired People

This book conveyed the visual elements of artwork to the visually impaired through various sensory elements to open a new perspective for appreciating visual artwork. In addition, the technique of expressing a color code by integrating patterns, temperatures, scents, music, and vibrations was explored, and future research topics were presented. A holistic experience using multi-sensory interaction acquired by people with visual impairment was provided to convey the meaning and contents of the work through rich multi-sensory appreciation. A method that allows people with visual impairments to engage in artwork using a variety of senses, including touch, temperature, tactile pattern, and sound, helps them to appreciate artwork at a deeper level than can be achieved with hearing or touch alone. The development of such art appreciation aids for the visually impaired will ultimately improve their cultural enjoyment and strengthen their access to culture and the arts. The development of this new concept aids ultimately expands opportunities for the non-visually impaired as well as the visually impaired to enjoy works of art and breaks down the boundaries between the disabled and the non-disabled in the field of culture and arts through continuous efforts to enhance accessibility. In addition, the developed multi-sensory expression and delivery tool can be used as an educational tool to increase product and artwork accessibility and usability through multi-modal interaction. Training the multi-sensory experiences introduced in this book may lead to more vivid visual imageries or seeing with the mind’s eye

Touch- and Walkable Virtual Reality to Support Blind and Visually Impaired Peoples‘ Building Exploration in the Context of Orientation and Mobility

Der Zugang zu digitalen Inhalten und Informationen wird immer wichtiger für eine erfolgreiche Teilnahme an der heutigen, zunehmend digitalisierten Zivilgesellschaft. Solche Informationen werden meist visuell präsentiert, was den Zugang für blinde und sehbehinderte Menschen einschränkt. Die grundlegendste Barriere ist oft die elementare Orientierung und Mobilität (und folglich die soziale Mobilität), einschließlich der Erlangung von Kenntnissen über unbekannte Gebäude vor deren Besuch. Um solche Barrieren zu überbrücken, sollten technische Hilfsmittel entwickelt und eingesetzt werden. Es ist ein Kompromiss zwischen technologisch niedrigschwellig zugänglichen und verbreitbaren Hilfsmitteln und interaktiv-adaptiven, aber komplexen Systemen erforderlich. Die Anpassung der Technologie der virtuellen Realität (VR) umfasst ein breites Spektrum an Entwicklungs- und Entscheidungsoptionen. Die Hauptvorteile der VR-Technologie sind die erhöhte Interaktivität, die Aktualisierbarkeit und die Möglichkeit, virtuelle Räume und Modelle als Abbilder von realen Räumen zu erkunden, ohne dass reale Gefahren und die begrenzte Verfügbarkeit von sehenden Helfern auftreten. Virtuelle Objekte und Umgebungen haben jedoch keine physische Beschaffenheit. Ziel dieser Arbeit ist es daher zu erforschen, welche VR-Interaktionsformen sinnvoll sind (d.h. ein angemessenes Verbreitungspotenzial bieten), um virtuelle Repräsentationen realer Gebäude im Kontext von Orientierung und Mobilität berührbar oder begehbar zu machen. Obwohl es bereits inhaltlich und technisch disjunkte Entwicklungen und Evaluationen zur VR-Technologie gibt, fehlt es an empirischer Evidenz. Zusätzlich bietet diese Arbeit einen Überblick über die verschiedenen Interaktionen. Nach einer Betrachtung der menschlichen Physiologie, Hilfsmittel (z.B. taktile Karten) und technologischen Eigenschaften wird der aktuelle Stand der Technik von VR vorgestellt und die Anwendung für blinde und sehbehinderte Nutzer und der Weg dorthin durch die Einführung einer neuartigen Taxonomie diskutiert. Neben der Interaktion selbst werden Merkmale des Nutzers und des Geräts, der Anwendungskontext oder die nutzerzentrierte Entwicklung bzw. Evaluation als Klassifikatoren herangezogen. Begründet und motiviert werden die folgenden Kapitel durch explorative Ansätze, d.h. im Bereich 'small scale' (mit sogenannten Datenhandschuhen) und im Bereich 'large scale' (mit einer avatargesteuerten VR-Fortbewegung). Die folgenden Kapitel führen empirische Studien mit blinden und sehbehinderten Nutzern durch und geben einen formativen Einblick, wie virtuelle Objekte in Reichweite der Hände mit haptischem Feedback erfasst werden können und wie verschiedene Arten der VR-Fortbewegung zur Erkundung virtueller Umgebungen eingesetzt werden können. Daraus werden geräteunabhängige technologische Möglichkeiten und auch Herausforderungen für weitere Verbesserungen abgeleitet. Auf der Grundlage dieser Erkenntnisse kann sich die weitere Forschung auf Aspekte wie die spezifische Gestaltung interaktiver Elemente, zeitlich und räumlich kollaborative Anwendungsszenarien und die Evaluation eines gesamten Anwendungsworkflows (d.h. Scannen der realen Umgebung und virtuelle Erkundung zu Trainingszwecken sowie die Gestaltung der gesamten Anwendung in einer langfristig barrierefreien Weise) konzentrieren.Access to digital content and information is becoming increasingly important for successful participation in today's increasingly digitized civil society. Such information is mostly presented visually, which restricts access for blind and visually impaired people. The most fundamental barrier is often basic orientation and mobility (and consequently, social mobility), including gaining knowledge about unknown buildings before visiting them. To bridge such barriers, technological aids should be developed and deployed. A trade-off is needed between technologically low-threshold accessible and disseminable aids and interactive-adaptive but complex systems. The adaptation of virtual reality (VR) technology spans a wide range of development and decision options. The main benefits of VR technology are increased interactivity, updatability, and the possibility to explore virtual spaces as proxies of real ones without real-world hazards and the limited availability of sighted assistants. However, virtual objects and environments have no physicality. Therefore, this thesis aims to research which VR interaction forms are reasonable (i.e., offering a reasonable dissemination potential) to make virtual representations of real buildings touchable or walkable in the context of orientation and mobility. Although there are already content and technology disjunctive developments and evaluations on VR technology, there is a lack of empirical evidence. Additionally, this thesis provides a survey between different interactions. Having considered the human physiology, assistive media (e.g., tactile maps), and technological characteristics, the current state of the art of VR is introduced, and the application for blind and visually impaired users and the way to get there is discussed by introducing a novel taxonomy. In addition to the interaction itself, characteristics of the user and the device, the application context, or the user-centered development respectively evaluation are used as classifiers. Thus, the following chapters are justified and motivated by explorative approaches, i.e., in the group of 'small scale' (using so-called data gloves) and in the scale of 'large scale' (using an avatar-controlled VR locomotion) approaches. The following chapters conduct empirical studies with blind and visually impaired users and give formative insight into how virtual objects within hands' reach can be grasped using haptic feedback and how different kinds of VR locomotion implementation can be applied to explore virtual environments. Thus, device-independent technological possibilities and also challenges for further improvements are derived. On the basis of this knowledge, subsequent research can be focused on aspects such as the specific design of interactive elements, temporally and spatially collaborative application scenarios, and the evaluation of an entire application workflow (i.e., scanning the real environment and exploring it virtually for training purposes, as well as designing the entire application in a long-term accessible manner)

Towards a National 3D Mapping Product for Great Britain

Knowing where something happens and where people are located can be critically important to understand issues ranging from climate change to road accidents, crime, schooling, transport and much more. To analyse these spatial problems, two-dimensional representations of the world, such as paper or digital maps, have traditionally been used. Geographic information systems (GIS) are the tools that enable capture, modelling, storage, retrieval, sharing, manipulation, analysis, and presentation of geographically referenced data. Three-dimensional geographic information (3D GI) is data that can represent real-world features as objects in 3D space. 3D GI offers additional functionality not possible in 2D, including analysing and querying volume, visibility, surface and sub-surface, and shadowing. This thesis contributes to the understanding of user requirements and other data related considerations in the production of 3D geographic information at a national level. The study promotes Ordnance Survey’s efforts in developing a 3D geographic product through: (1) identifying potential applications; (2) analysing existing 3D city modelling approaches; (3) eliciting and formalising user requirements; (4) developing metrics to describe the usefulness of 3D data and; (5) evaluating the commerciality of 3D GI. A review of current applications of 3D showed that visualisation dominated as the main use, allowing for better communication, and supporting decision-making processes. Reflecting this, an examination of existing 3D city models showed that, despite the varying modelling approaches, there was a general focus towards accurate and realistic geometric representation of the urban environment. Web-based questionnaires and semi-structured interviews revealed that while some applications (e.g. subsurface, photovoltaics, air and noise quality) lead the field with a high adoption of 3D, others were laggards due to organisational inertia (e.g. insurance, facilities management). Individuals expressed positive views on the use of 3D, but still struggled to justify the value and business case. Simple building geometry coupled with non-building thematic classes was perceived to be most useful by users. Several metrics were developed to quantify and compare the characteristics of thirty-three 3D datasets. Results showed that geometry-based metrics such as minimum feature length or Euler characteristic can be used to provide additional information as part of fitness-for-purpose evaluations. The metrics can also contribute to quality control during data production. An investigation into the commercial opportunities explored the economic value of 3D, the market size of 3D data in Great Britain, as well as proposed a number of opportunities within the wider business context of Ordnance Survey

Approaching Reality: Integrating Image-based 3D Modelling and Complex Spatial Data in Archaeological Field Recording

This thesis finalises a 5+3 PhD project within the joint doctoral programme in Digital Heritage established in collaboration between History, Archaeology and Classical Studies, Graduate School, the Faculty of Arts, Aarhus University and the University of York. The thesis deals with the overarching theme of spatial data in archaeological excavation recording. Spatial data are at the core of all archaeological observations, and are expressed in numerous ways, ranging from traditional hand drawings to digital two- and three-dimensional representations in Geographic Information Systems and proprietary 3D software. Yet, despite technological advances, state-of-the art digital spatial data are almost equally detached from textual archaeological interpretation as they were using conventional tools decades ago. The thesis presents a study of how technological advances influence archaeological excavation traditions and methodologies. Special emphasis is directed at exploring how the increased use of image-based 3D documentation may contribute to increased quality of field recording and, in particular, what theoretical conceptualisations and technical developments are needed to harness its full potential. The thesis is composed of four articles, which constitute individual chapters (2-5). Each chapter covers a theme within the underlying topic of integrating spatial data in archaeology, supplemented by an introductory chapter (1), a synthesis (6) and a conclusion (7). The first article (chapter 2) provides an introduction to the overarching research questions and their methodological and historical background. It offers some rudimentary impressions of differing excavation and recording traditions in Britain and Denmark, to critically assess the use of GIS in archaeology and the negotiation between state-of-the-art technology and archaeological practice. The article discusses how the adaptation of GIS may have contributed significantly to the detrimental effect of creating stand-alone silos of spatial data that are rarely fully integrated with non-spatial, textual data, and has acted to stifle the development of digital standards of recording by perpetuating outmoded analogue recording conventions from a previous century. The chapter outlines the potential of born-digital 3D recording technologies such as Structure From Motion (SFM), GPS, and laser scanning in current practice, while advocating for a conceptualisation of new types of data and data representation in archaeological documentation. This, however, requires changes in archaeological methodologies and workflows and that we redefine more explicitly what we actually want to do with spatial data in archaeology. The second article (chapter 3) seeks to advance the conceptual framework of 3D models within archaeological excavation recording. 3D documentation advocates for a new workflow with a more three-dimensional reasoning, allowing for the utilisation of 3D as a tool for continuous progress planning and evaluation of an excavation and its results. Just like the general use of models to form hypotheses, it is possible to use 3D models as spatial hypotheses of an ongoing excavation. This allows us to visually realise or spatially conceptualise our hypotheses as a virtual reconstruction and to combine it with our observational data. The article presents first-hand experiences of working with 3D reconstruction and visualisations during the excavations at Viking Age site Jelling, and explores how the concept of authenticity may facilitate negotiations between visualising what we know, and what we think we know. The third article (chapter 4) further addresses the challenges inherent to the integration of 3D documentation: specifically its inability to convey archaeological interpretations. Image-based 3D modelling is generally considered a superior tool for generating geometrically accurate and photo-realistic recording of an excavation, but does not immediately encourage reflexive or interpretative practice. This is a direct consequence of the technical limitations of currently available tools, but also reflects an archaeological methodology and spatial conceptualisation based on two-dimensional abstractions. Using the example of the excavations at the Iron Age site Alken Enge, this article takes a more technical approach to exploring how new tools developed for segmenting field-recorded 3D geometry allow embedding archaeological interpretations directly in the 3D model, thereby augmenting its semantic value considerably. This is considered a precondition for the successful integration of 3D models as archaeological documentation. Furthermore, the article explores how web-based 3D platforms may facilitate collaborative exchange of 3D excavation content and how the integration of spatial and attribute data into one common event-based data model may be advantageous. The event-based approach is used for conceptualising how digital spatial data are created, derived and evolve throughout the documentation and post-excavation process. This effectively means building a conceptualisation of excavation recording procedures and seeing them through to the data model implementation itself. The fourth and last article (chapter 5) further explores the technologies outlined in chapters two and four. In particular, it focuses on evaluating analytical capabilities and alternative visualisation end-goals for 3D excavation recording. The chapter presents a simple case study, demonstrating the pipeline from excavating an archaeological feature, through image-based documentation and processing, to volumetric visual representation, while exploring the potential of machine learning to aid in feature recognition and classification. Chapter six acts as a synopsis, which provides added context to the results of the preceding chapters and furthermore discusses archaeological data models in general, conceptual reference models and, finally, presents the data model and implementation developed during the research project. The research introduces several novel approaches and technical developments aimed at aggregating the fragmented excavation data throughout the archaeological sector. This includes developing software for harvesting 2D GIS data from file storage at local archaeological institutions, functions for 3D semantic segmentation, automated processes for pattern recognition (SVM), machine learning and volumetric visualisation, and database mappings to web-services such as the MUD excavation database - all of which feed into the development of the Archaeo Framework. The online database \url{www.archaeo.dk} provides an implementation of the proposed data model for complex spatial field recorded data, and demonstrates the achieved data management capabilities, analytical queries, various spatial and visual representations and data interoperability functions. The Archaeo Framework acts as a data repository for excavation data, and provides long-awaited integration of spatial and textual data in Denmark. The benefits of spatial integration are clearly evident, notably having all information in one system, available online for research, dissemination and data re-use. For the first time, it is possible to perform large-scale validation of digital excavation plans against the written record, and perform complex spatial queries at a much deeper level than merely a site on a map. This research frames the basis for further developments of dynamic data management approaches to the integration of complex spatial data in field archaeology. The data model is expected to assist archaeologists in implementing better conceptualised excavation data models, and to facilitate a better understanding and use of 3D for archaeological documentation and analysis. Ultimately, the implementation provides access to the inaccessible dimensions of archaeological recording by joining hitherto isolated and fragmentary archaeological datasets - spatial and textual. Future areas of investigation should seek to advance this further in order to facilitate the persistence of complex spatial data as integrated components of archaeological data models

Spatial ontologies for architectural heritage

Informatics and artificial intelligence have generated new requirements for digital archiving, information, and documentation. Semantic interoperability has become fundamental for the management and sharing of information. The constraints to data interpretation enable both database interoperability, for data and schemas sharing and reuse, and information retrieval in large datasets. Another challenging issue is the exploitation of automated reasoning possibilities. The solution is the use of domain ontologies as a reference for data modelling in information systems. The architectural heritage (AH) domain is considered in this thesis. The documentation in this field, particularly complex and multifaceted, is well-known to be critical for the preservation, knowledge, and promotion of the monuments. For these reasons, digital inventories, also exploiting standards and new semantic technologies, are developed by international organisations (Getty Institute, ONU, European Union). Geometric and geographic information is essential part of a monument. It is composed by a number of aspects (spatial, topological, and mereological relations; accuracy; multi-scale representation; time; etc.). Currently, geomatics permits the obtaining of very accurate and dense 3D models (possibly enriched with textures) and derived products, in both raster and vector format. Many standards were published for the geographic field or in the cultural heritage domain. However, the first ones are limited in the foreseen representation scales (the maximum is achieved by OGC CityGML), and the semantic values do not consider the full semantic richness of AH. The second ones (especially the core ontology CIDOC – CRM, the Conceptual Reference Model of the Documentation Commettee of the International Council of Museums) were employed to document museums’ objects. Even if it was recently extended to standing buildings and a spatial extension was included, the integration of complex 3D models has not yet been achieved. In this thesis, the aspects (especially spatial issues) to consider in the documentation of monuments are analysed. In the light of them, the OGC CityGML is extended for the management of AH complexity. An approach ‘from the landscape to the detail’ is used, for considering the monument in a wider system, which is essential for analysis and reasoning about such complex objects. An implementation test is conducted on a case study, preferring open source applications