This paper describes an augmented reality app running on a Windows tablet that recognizes image features on a city paper map and overlays, in real-time, digital content related to the relevant buildings of the city. The system register in 3D the location of those buildings in the map, enriching the user experience with several multimedia information per building: image, text and 3D models which can be explored in detail in an included BIM viewer. Displayed buildings can be queried and filtered by associated meta-data such as decade, author, conservation, etc. The main target users of this system are tourists or users interested in architecture or history. Our usability evaluation study conducted with several users shown that our app increase the scope of applicability of a paper map.info:eu-repo/semantics/publishedVersio

Dias, M. S.

Eloy, S.

Faria, N.

Gaspar, F.

Gomes, S.

Lopes, M.

Resende, R.

English

Repositório Institucional do ISCTE-IUL

179ARch4maps: a mobile augmented reality tool to enrich paper maps Filipe Gaspar1,2, Steven Gomes1, Ricardo Resende1, Sara Eloy1, Miguel Sales Dias1,2, Mariana Lopes2 and Nuno Faria2 1Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR-IUL  Lisbon, Portugal, Portugal stevenrmgomes@gmail.com, {jrpre, sara.eloy, jmd}@iscte.pt  2Microsoft Language Development Center Lisbon, Portugal {t-filipg, t-marig, Miguel.Dias}@microsoft. com    ABSTRACT This paper describes an augmented reality app running on a Windows tablet that recognizes image features on a city paper map and overlays, in real-time, digital content related to the relevant buildings of the city. The system register in 3D the location of those buildings in the map, enriching the user experience with several multimedia information per building: image, text and 3D models which can be explored in detail in an included BIM viewer. Displayed buildings can be queried and filtered by associated meta-data such as decade, author, conservation, etc. The main target users of this system are tourists or users interested in architecture or history. Our usability evaluation study conducted with several users shown that our app increase the scope of applicability of a paper map.  Author Keywords Augmented Reality; Paper maps; Mobile; Architecture; Tourism. 1 INTRODUCTION Maps assist navigation to a pre-chosen place, as an exploration aid or to represent geographical information to support decision-making. From the onset of their use, maps convey several layers of graphical information (roads, cities, urban landscapes, orography, etc.), and are enhanced with non-spatial information through annotations, symbols defined in legends and, in some cases, even companion books. In the last decades, geo-referenced geographical information systems (GIS) have been driving the generation of maps, and have been enriched with meta-information which can be hidden or displayed, searched and correlated. Google maps, Bing maps and some other cloud based platforms are now ubiquitous on laptops and mobile devices and are extensively consumed via apps and browsers. Nevertheless, paper maps present advantages over digital versions. They allow a natural “zoom” as the user can focus in one particular area, which is especially useful in the field, thus many users are used to manipulating this traditional media (paper). Naturally, the information on paper maps contain is limited by existing space, book-maps can contain much more information, such as critical texts, fact sheets, photographs, historical and cultural features and technical drawings. They are, however, cumbersome to use and, in general, paper maps and books cannot be updated, just replaced.  This paper describes an augmented reality (AR) solution, referred to as ARch4maps, running on a Microsoft Windows tablet, that provides a hybrid paper-digital response to the problem. Using a computer vision technique developed in-house, we recognize, in real-time, features on a paper map and overlay geo-referenced digital content, such as 3D models representing the location of points of interest (POI). Using touch-based interaction on the tablet surface, a user can explore several modalities of multimedia information in each POI: images, text and detailed 3D models layered according their BIM classification (structural, architectural, infrastructural, interior design, etc). ARch4maps is able to display or hide each component, and explore the building in AR through different perspectives (by walking around the map with the tablet, or manipulating the map directly), cross-sections, plans or sections. Search and building selection by pre-defined criteria is also possible. In the implemented use case, buildings that won the Valmor award, a century old architecture award in the city of Lisbon have been integrated in our system. It is possible to search such buildings, by decade, author, conservation, possibility of visit, etc. We have carried a usability and satisfaction evaluation study of ARch4maps, with a panel of students of the Architecture master degree at ISCTE-IUL, interested in architecture history that concluded on the validity of our approach. 2 STATE OF THE ART There are several guides and maps dedicated to architectural heritage, which is a very common tourist niche. In what regards paper guides for the city of Lisbon, we have the following. The Valmor and Lisbon Council prizes map (1902-2002) 8  was used in this case study (Figure 1). With a dimension of 60x100 cm, on the front side it shows a map of the city with the buildings location identified by a number, letter and color code and, on the right side the basic information of the 115 works highlighted in the map: name of the building, prize, year, architect, location, nearest public transport and in some cases, images). On the flip-side of the map we have more information on the awards and building 180descriptions and images, organized by decades. Lisbon Architectural Guide 2  is a 374 pages guide book presenting detailed information on relevant Lisbon buildings from 1948 to 2013. The large amount of information convey book-guides to be, less appropriate to use as an exploration aid within a reasonable period of time. Argumentum Architecture Maps [1] has a synthesized fashion map per each Portuguese district, similar to the Valmor, and the content is dedicated to the architecture audience. It provides building’s location, functions, history and architects.  Figure 1. Valmor & Lisbon Council Awards 1902-2002 map 10 . Mobile apps may contain a practically limited amount of information in several formats, even without permanent web access, but are limited in their readability by the screen size. The Architectural Guide Portugal 6  iPhone/iPad app displays works from main Portuguese architects: Álvaro Siza Vieira, Souto Moura, Gonçalo Byrne, among others. Works are indicated using the users location and allows saving favorite works, search by area, author, and function. It is connected to TomTom and Google Maps. It also has an accompanying website and paper books. 2.1 Augmented Reality Augmented Reality is a multi-disciplinary area of fusion between computer vision and 3D computer graphics that allows the 3D registration of computer-generated data in real world, usually overlay on real objects. This is achieved by performing instantaneous virtual camera pose (position and orientation) calibration, i.e., by computing, in real-time (for interactive apps), the virtual camera parameters that match the position and orientation of the observer or sensor (physical camera) in the real scene. Vision-based methods estimate the real camera pose relative to visually distinctive features present in greyscale, color or even map images 9 . In general, AR technologies provide the means for “intuitive information presentation, which enhances the perceiver’s situational awareness and cognitive perception of the real world” 4 . Several guides employ nowadays AR to help users locate remarkable buildings, both with the architecture and mainstream tourist users in mind. Wikitude is a framework that has been employed by a large number of mobile AR applications since it has several tracking technologies available such as GPS, WiFi, inertial sensors and image-based recognition 14 . Rewind Cities Lisbon is a mobile app that alerts the user which can then watch multimedia content overlaid on the city image captured by the camera in pre-determined places in Lisbon 5 . Archeoguide is an Augmented Reality-based Cultural Heritage On-site Guide that displays a 3D course in Mount Olympus with reconstructed images of the buildings 7 . In this paper, we build upon an existing AR app, referred to as ARch 11 . This app shows and manipulates 3D models of buildings in AR using a tablet with a built-in camera. The major novelty brought by Arch, is that it includes operations usually available in 3D CAD software for Architecture, Engineering and Construction, such as BIM model viewing and real-time cross-sections. 3 THE ARCH4MAPS APP This paper describes an AR application running on a tablet that allows a user to superimpose location of buildings on a map, displaying additional information like digital 3D models, the building technical drawings such as their plans, sections and elevations, historical and cultural information. To this end, this app was designed to be informative, interactive and simple to use. The target audiences for this application are mainly tourists, architects and users interested in architecture. Two possible scenarios are: 1) two tourists seated in the cafe look into a paper map and want to know more information on interesting buildings nearby; 2) an architect/researcher wants to gather information from a paper map about awarded buildings in a specific decade or from a specific architect. Figure 2. Arch4maps`s user experience. a) Map identification; b) Click on building’s point; c) Overlay the 3D model on the map. 3.1 User Requirements Figure 2 depicts the user experience common to both scenarios, in three steps. From these scenarios, we were able to derive the following user requirements for our app: 1) Identify the base map, or parts of it, as a visual marker to aid our AR engine in performing virtual camera pose calibration, in real-time; 2) Overlay the locations of the buildings on the correct position on the map; 3) Filter buildings’ on the map by several criteria: type of award, date of the award, architect name, building identification, possibility of visitation, conservation status; adulteration status; 4) Display available information about a building, chosen by the user when touching on a registered virtual point painted on the tablet screen. The information that can be displayed is descriptive text; plans, pictures, photographs or drawings; 1815) Benefit from the already available ARch features (BIM viewer, interactive cutting planes, etc.), which experience relies on touch interaction. Texture Tracking  3.2 Texture Tracking We perform virtual camera calibration, to achieve AR, by means of a texture tracking technique which uses the physical map captured by the camera of the device, as a visual marker. Given that map images are usually rich in topology (lines and geometric elements) but poor in high contrast regions, we have selected Scale Invariant Feature transform (SIFT) [10] to detect, describe and match these regions and compute the virtual camera pose in the first video frame. SIFT uses a multi-resolution pyramid of greyscale images to highlight distinctive high contrast regions in several frequency ranges. These features are very appealing for AR, since they are invariant to rotation and translation changes, are tolerant to lighting changes, but are computation demanding. To achieve real-time AR we track SIFT features between contiguous video frames using Natural Ubiquitous Texture Tracking system (NUTTS) [3] which proven to be effective with planar natural scenes and recover lost features from previous frames using optical-flow, texture back-projection and image reconstruction. Once SIFT features are tracked, the 3D virtual models are registered in the real world, on top of the map, using the virtual camera pose estimated. Our method only need a few sparsely located known SIFT features (from a pool of hundreds usually available) which means that the map can be partially occluded thus allowing the user to pan and zoom in without losing tracking. 3.3 Functionalities and Interface ARch4maps was developed and evaluated on a Microsoft Surface Pro 2 tablet with the following specs: Windows 8.1 Pro 64-bit; Intel Core i5-4200U Dual Core; 4 GB RAM; Intel HD Graphics 4400; SSD disk of 64 GB. The use of a tablet was the solution given the requirement to display several multimedia information including detailed building’s 3D models. The app interface was designed to allow the user to experience AR quickly and easily through natural right hand gestures (click, swipe and zoom) in the main part of the screen (right), while the left hand holds the device and clicks on the menu located on the left hand side of the screen (Figure 3). A short demo video of the application is available at http://istar.iscte-iul.pt/index.php/ProjectsAR. After the app is started on the tablet the camera is pointed to the map. The texture tracking module computes the virtual camera pose in each video frame, and the 3D locations of the buildings in the database are registered and shown on the screen, overlaid on their actual position in the physical map. Search menu: displayed at left hand side and fulfilling the requirements defined in section 3.1, buildings can be searched and filtered by its name, architect, awards, accessibility status (entirely or partially) and conservation state (demolished, poorly preserved, acceptable or well preserved). For each search criteria, the user can select several options from a drop-down list. In the right hand side of the screen, buildings within the search criteria are represented by virtual sphere registered in building location in paper map’s coordinate system and scale (Figure 4, middle). The user can also perform compound filters. For example, one can look for buildings from the 1910-1920 decade in an acceptable state of conservation and that can be visited. Boolean logic is applicable to the filters, i.e., conditions can be operated with conjunction and disjunction. Multimedia menu: As the user selects a building on the screen by clicking on its corresponding virtual sphere, the building’s 3D model is registered in the center of the paper map via AR and a variety of multimedia content becomes available (Figure 3):  Description, which displays a descriptive text;   Drawings and Pictures, where the user browses through plans, sections, elevations, renders of the buildings or other associated imagery such as historical photos. Other ARch features, namely, the 3D BIM visualizer engine where the user can explore in depth the building 3D model, perform interactive cutting planes, etc. ARch allows user to view the virtual building from all angles including moving the tablet around the map or by rotating the map itself. It is also possible to show, enhance or hide different components of the building (defined as a BIM model), such as its structure, walls and pavements, furniture or infrastructure. Finally, it is possible to observe the interior of the buildings through vertical and horizontal cutting planes, as shown in Figure 3. 4 USUABILITY AND SATISFACTION TESTS The app was submitted to usability and satisfaction evaluation study, with eleven participants, 91% of which aged between 21 and 35. The academic formation of the participants varied between practicing architects (4), students (4) and the remainder (3) had varied occupations. Most (10 out of 11) of the subjects travel and visit buildings regularly but their interest in such visits is “some” (7) to Figure 3. ARch4Maps main features. a) Active filters on the left hand side and filtered building locations over the map on the right. b) Building’s photo navigation browser. c) 3D model of a building overlaid on the map. d) Vertical cross-section cut of a 3D model.182“high” (4). The relationship with IT was good for the majority (9) and three of the users have had prior experiences with augmented reality. After testing the app the subjects answered four questions with the answers being on a scale of one (No/Poor/Not likely) to five (Yes/Excellent/Certainly). Q1) The application makes the maps more informative? Q2) The application facilitates understanding of the information contained in the map? Q3) The application displays the buildings clearly through the 3D model? Q4) Would you use the application in the future? Analysis of the answers, displayed in Figure 4, shows 73% of the users reported that the app delivers much more information than the paper map, but only a smaller proportion reported that understanding of the conveyed information was extremely facilitated. A majority found that map utilization is improved a lot or significantly. Two in three users reported they would certainly or most likely use the application. Given that in some of the tests participants were subject to small system failures (tracking errors of the map), we think these numbers may improve when our tracking technique will be more robust to handle physical maps.  Figure 4. Questionnaire answer distribution. 5 CONCLUSIONS This work focused primarily on the definition of an app that allows users to take advantage of cities’ physical maps to increase the information provided via AR, enabling the visualization of buildings in an interactive and dynamic way. ARch4maps provides to the user the possibility to search and filter throughout several layers of multimedia information such as descriptive texts, pictures, technical drawings as well as real time interactive 3D models. Our customer satisfaction and usability evaluation on a map of Valmor and Lisbon Council awards through AR shown that most participants easily handled the application and were satisfied with its performance. This way, ARch4maps proved to be very capable in increasing the amount and type of information of the maps, while having the possibility of launching and manipulating buildings’ 3D models of buildings, which was very appreciated and considered to work well and transmit the reality of the building to the general public, which is not used to read plans and cross sections. We can thus conclude that ARch4maps has the potential to enrich both specialist and the general public with an interest in architecture and built heritage. 6 FUTURE WORK Besides the fixing of minor flaws found in ARch4maps the informal comments obtained from participants in the user study allows us to prioritize some further work: 1) Addition of more multimedia content such as videos and biographical information about the architects and awards; 2) Integration of an geo-referenced sub-system such as Bing maps to allow the user to find the buildings and organize visiting routes; 3) Add a social and feedback component to the app (e.g. Facebook) allowing users to rate and comment the building; 4) Allow users to submit building’s related multimedia. Finally, we are interested in augment outdoor map supports, usually placed in strategic point in cities, such as the popular maps made of ceramic tiles which are quite popular in Portugal. ACKNOWLEDGMENTS This research was partially supported by OLA, a project co-financed by the AAL Joint Programme (AAL JP) and FCT (in Portugal). We thank to Lisbon City Council for providing bibliographic material. REFERENCES  1. Argumentum. www.argumentum.pt. As of 1 March 2016. 2. Melo, M., Toussaint, M. Lisbon Architectural Guide 1948/2013. A+A Books, Lisbon, Portugal, 2013. 3. Bastos, R, Dias, M. Fully automated texture tracking based on natural features extraction and template matching. Proc. ACE’05, ACM Press (2005), 180-183. 4. Behringer, R., Klinker, G, Mizell, D. Augmented Reality-Placing Artificial Objects in Real Scenes. Proc. IWAR '98, ACM Press (1998), 63-72. 5. Rewind Cities Lda www.f6s.com/rewindcitieslda. As of 1 March 2016. 6. Guias de Arquitectura. guiasdearquitectura.com. As of 1 March 2016. 7. Vlahakis, V. Archeoguide:An Augmented Reality Guide for Archaeological Sites. IEEE Computer Graphics and Applications 22, 5 (2002), 55-60. 8. Lencastre, T., Alho, C. et al.  Prémios Valmor e Municipal de Arquitectura 1902-2002. Câmara Municipal de Lisboa, Portugal, 2012. 9. Lopes, M, et al. Sistema de Realidade Aumentada para Apoio ao Projeto de Arquitetura. Proc. 21th EPCG, IPL Portugal (2014), 151-158. 10. Lowe, D. Object recognition from local scale-invariant features. Proc. ICCV 2, IEEE (1999),1150-1157. 11. Wikitude. http://www.wikitude.com. As of 1 Match 2016. 

ARch4maps: A mobile augmented reality tool to enrich paper maps

https://repositorio.iscte-iul.pt/bitstream/10071/23247/1/conferenceobject_29152.pdf

ARch4maps: A mobile augmented reality tool to enrich paper maps

Abstract

Similar works

Full text

Available Versions

Repositório Institucional do ISCTE-IUL