Learning to Detect Touches on Cluttered Tables

We present a novel self-contained camera-projector tabletop system with a lamp form-factor that brings digital intelligence to our tables. We propose a real-time, on-device, learning-based touch detection algorithm that makes any tabletop interactive. The top-down configuration and learning-based algorithm makes our method robust to the presence of clutter, a main limitation of existing camera-projector tabletop systems. Our research prototype enables a set of experiences that combine hand interactions and objects present on the table. A video can be found at https://youtu.be/hElC_c25Fg8

Adversarial Examples in the Physical World: A Survey

Deep neural networks (DNNs) have demonstrated high vulnerability to adversarial examples. Besides the attacks in the digital world, the practical implications of adversarial examples in the physical world present significant challenges and safety concerns. However, current research on physical adversarial examples (PAEs) lacks a comprehensive understanding of their unique characteristics, leading to limited significance and understanding. In this paper, we address this gap by thoroughly examining the characteristics of PAEs within a practical workflow encompassing training, manufacturing, and re-sampling processes. By analyzing the links between physical adversarial attacks, we identify manufacturing and re-sampling as the primary sources of distinct attributes and particularities in PAEs. Leveraging this knowledge, we develop a comprehensive analysis and classification framework for PAEs based on their specific characteristics, covering over 100 studies on physical-world adversarial examples. Furthermore, we investigate defense strategies against PAEs and identify open challenges and opportunities for future research. We aim to provide a fresh, thorough, and systematic understanding of PAEs, thereby promoting the development of robust adversarial learning and its application in open-world scenarios.Comment: Adversarial examples, physical-world scenarios, attacks and defense

Robotic Cameraman for Augmented Reality based Broadcast and Demonstration

In recent years, a number of large enterprises have gradually begun to use vari-ous Augmented Reality technologies to prominently improve the audiences’ view oftheir products. Among them, the creation of an immersive virtual interactive scenethrough the projection has received extensive attention, and this technique refers toprojection SAR, which is short for projection spatial augmented reality. However,as the existing projection-SAR systems have immobility and limited working range,they have a huge difficulty to be accepted and used in human daily life. Therefore,this thesis research has proposed a technically feasible optimization scheme so thatit can be practically applied to AR broadcasting and demonstrations. Based on three main techniques required by state-of-art projection SAR applica-tions, this thesis has created a novel mobile projection SAR cameraman for ARbroadcasting and demonstration. Firstly, by combining the CNN scene parsingmodel and multiple contour extractors, the proposed contour extraction pipelinecan always detect the optimal contour information in non-HD or blurred images.This algorithm reduces the dependency on high quality visual sensors and solves theproblems of low contour extraction accuracy in motion blurred images. Secondly, aplane-based visual mapping algorithm is introduced to solve the difficulties of visualmapping in these low-texture scenarios. Finally, a complete process of designing theprojection SAR cameraman robot is introduced. This part has solved three mainproblems in mobile projection-SAR applications: (i) a new method for marking con-tour on projection model is proposed to replace the model rendering process. Bycombining contour features and geometric features, users can identify objects oncolourless model easily. (ii) a camera initial pose estimation method is developedbased on visual tracking algorithms, which can register the start pose of robot to thewhole scene in Unity3D. (iii) a novel data transmission approach is introduced to establishes a link between external robot and the robot in Unity3D simulation work-space. This makes the robotic cameraman can simulate its trajectory in Unity3D simulation work-space and project correct virtual content. Our proposed mobile projection SAR system has made outstanding contributionsto the academic value and practicality of the existing projection SAR technique. Itfirstly solves the problem of limited working range. When the system is running ina large indoor scene, it can follow the user and project dynamic interactive virtualcontent automatically instead of increasing the number of visual sensors. Then,it creates a more immersive experience for audience since it supports the user hasmore body gestures and richer virtual-real interactive plays. Lastly, a mobile systemdoes not require up-front frameworks and cheaper and has provided the public aninnovative choice for indoor broadcasting and exhibitions

Enhanced life-size holographic telepresence framework with real-time three-dimensional reconstruction for dynamic scene

Three-dimensional (3D) reconstruction has the ability to capture and reproduce 3D representation of a real object or scene. 3D telepresence allows the user to feel the presence of remote user that was remotely transferred in a digital representation. Holographic display is one of alternatives to discard wearable hardware restriction, it utilizes light diffraction to display 3D images to the viewers. However, to capture a real-time life-size or a full-body human is still challenging since it involves a dynamic scene. The remaining issue arises when dynamic object to be reconstructed is always moving and changes shapes and required multiple capturing views. The life-size data captured were multiplied exponentially when working with more depth cameras, it can cause the high computation time especially involving dynamic scene. To transfer high volume 3D images over network in real-time can also cause lag and latency issue. Hence, the aim of this research is to enhance life-size holographic telepresence framework with real-time 3D reconstruction for dynamic scene. There are three stages have been carried out, in the first stage the real-time 3D reconstruction with the Marching Square algorithm is combined during data acquisition of dynamic scenes captured by life-size setup of multiple Red Green Blue-Depth (RGB-D) cameras. Second stage is to transmit the data that was acquired from multiple RGB-D cameras in real-time and perform double compression for the life-size holographic telepresence. The third stage is to evaluate the life-size holographic telepresence framework that has been integrated with the real-time 3D reconstruction of dynamic scenes. The findings show that by enhancing life-size holographic telepresence framework with real-time 3D reconstruction, it has reduced the computation time and improved the 3D representation of remote user in dynamic scene. By running the double compression for the life-size holographic telepresence, 3D representations in life-size is smooth. It has proven can minimize the delay or latency during acquired frames synchronization in remote communications

자동차 사양 변경을 실시간 반영하는 데이터 기반 디자인 접근 방법

학위논문 (박사) -- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2020. 8. 곽노준.The automotive industry is entering a new phase in response to changes in the external environment through the expansion of eco-friendly electric/hydrogen vehicles and the simplification of modules during the manufacturing process. However, in the existing automotive industry, conflicts between structured production guidelines and various stake-holders, who are aligned with periodic production plans, can be problematic. For example, if there is a sudden need to change either production parts or situation-specific designs, it is often difficult for designers to reflect those requirements within the preexisting guidelines. Automotive design includes comprehensive processes that represent the philosophy and ideology of a vehicle, and seeks to derive maximum value from the vehicle specifications. In this study, a system that displays information on parts/module components necessary for real-time design was proposed. Designers will be able to use this system in automotive design processes, based on data from various sources. By applying the system, three channels of information provision were established. These channels will aid in the replacement of specific component parts if an unexpected external problem occurs during the design process, and will help in understanding and using the components in advance. The first approach is to visualize real-time data aggregation in automobile factories using Google Analytics, and to reflect these in self-growing characters to be provided to designers. Through this, it is possible to check production and quality status data in real time without the use of complicated labor resources such as command centers. The second approach is to configure the data flow to be able to recognize and analyze the surrounding situation. This is done by applying the vehicles camera to the CCTV in the inventory and distribution center, as well as the direction inside the vehicle. Therefore, it is possible to identify and record the parts resources and real-time delivery status from the internal camera function without hesitation from existing stakeholders. The final approach is to supply real-time databases of vehicle parts at the site of an accident for on-site repair, using a public API and sensor-based IoT. This allows the designer to obtain information on the behavior of parts to be replaced after accidents involving light contact, so that it can be reflected in the design of the vehicle. The advantage of using these three information channels is that designers can accurately understand and reflect the modules and components that are brought in during the automotive design process. In order to easily compose the interface for the purpose of providing information, the information coming from the three channels is displayed in their respective, case-specific color in the CAD software that designers use in the automobile development process. Its eye tracking usability evaluation makes it easy for business designers to use as well. The improved evaluation process including usability test is also included in this study. The impact of the research is both dashboard application and CAD system as well as data systems from case studies are currently reflected to the design ecosystem of the motors group.자동차 산업은 친환경 전기/수소 자동차의 확대와 제조 공정에서의 모듈 단순화를 통해서 외부 환경의 변화에 따른 새로운 국면을 맞이하고 있다. 하지만 기존의 자동차 산업에서 구조화된 생산 가이드라인과 기간 단위 생산 계획에 맞춰진 여러 이해관계자들과의 갈등은 변화에 대응하는 방안이 관성과 부딪히는 문제로 나타날 수 있다. 예를 들어, 갑작스럽게 생산에 필요한 부품을 변경해야 하거나 특정 상황에 적용되는 디자인을 변경할 경우, 주어진 가이드라인에 따라 디자이너가 직접 의견을 반영하기 어려운 경우가 많다. 자동차 디자인은 차종의 철학과 이념을 나타내고 해당 차량제원으로 최대의 가치를 끌어내고자 하는 종합적인 과정이다. 본 연구에서는 여러 원천의 데이터를 기반으로 자동차 디자인 과정에서 활용할 수 있도록 디자인에 필요한 부품/모듈 구성요소들에 대한 정보를 실시간으로 표시해주는 시스템을 고안하였다. 이를 적용하여 자동차 디자인 과정에서 예상 못한 외부 문제가 발생했을 때 선택할 구성 부품을 대체하거나 사전에 해당 부품을 이해하고 디자인에 활용할 수 있도록 세 가지 정보 제공 채널을 구성하였다. 첫 번째는 자동차 공장 내 실시간 데이터 집계를 Google Analytics를 활용하여 시각화하고, 이를 공장 자체의 자가 성장 캐릭터에 반영하여 디자이너에게 제공하는 방식이다. 이를 통해 종합상황실 등의 복잡한 인력 체계 없이도 생산 및 품질 현황 데이터를 실시간으로 확인 가능하도록 하였다. 두 번째는 차량용 주차보조 센서 카메라를 차량 부착 뿐만 아니라 인벤토리와 물류센터의 CCTV에도 적용하여 주변상황을 인식하고 분석할 수 있도록 구성하였다. 차량의 조립 생산 단계에서 부품 단위의 이동, 운송, 출하를 거쳐 완성차의 주행 단계에 이르기까지 데이터 흐름을 파악하는 것이 디자인 부문에 필요한 정보를 제공할 수 있는 방법으로 활용되었다. 이를 통해 기존 이해관계자들의 큰 반발 없이 내부의 카메라 기능으로부터 부품 리소스와 운송 상태를 실시간 파악 및 기록 가능하도록 하였다. 마지막으로 공공 API와 센서 기반의 사물인터넷을 활용해서 도로 위 차량 사고가 발생한 위치에서의 현장 수리를 위한 차량 부품 즉시 수급 및 데이터베이스화 방법도 개발 되었다. 이는 디자이너로 하여금 가벼운 접촉 사고에서의 부품 교체 행태에 대한 정보를 얻게 하여 차량의 디자인에 반영 가능하도록 하였다. 시나리오를 바탕으로 이 세 가지 정보 제공 채널을 활용할 경우, 자동차 디자인 과정에서 불러들여오는 부품 및 모듈의 구성 요소들을 디자이너가 정확히 알고 반영할 수 있다는 장점이 부각되었다. 정보 제공의 인터페이스를 쉽게 구성하기 위해서, 실제로 디자이너들이 자동차 개발 과정에서 디자인 프로세스 상에서 활용하는 CAD software에 세 가지 채널들로부터 들어오는 정보를 사례별 컬러로 표시하고, 이를 시선추적 사용성 평가를 통해 현업 디자이너들이 사용하기 쉽게 개선한 과정도 본 연구에 포함시켜 설명하였다.1 Introduction 1 1.1 Research Background 1 1.2 Objective and Scope 2 1.3 Environmental Changes 3 1.4 Research Method 3 1.4.1 Causal Inference with Graphical Model 3 1.4.2 Design Thinking Methodology with Co-Evolution 4 1.4.3 Required Resources 4 1.5 Research Flow 4 2 Data-driven Design 7 2.1 Big Data and Data Management 6 2.1.1 Artificial Intelligence and Data Economy 6 2.1.2 API (Application Programming Interface) 7 2.1.3 AI driven Data Management for Designer 7 2.2 Datatype from Automotive Industry 8 2.2.1 Data-driven Management in Automotive Industry 8 2.2.2 Automotive Parts Case Studies 8 2.2.3 Parameter for Generative Design 9 2.3 Examples of Data-driven Design 9 2.3.1 Responsive-reactive 9 2.3.2 Dynamic Document Design 9 2.3.3 Insignts from Data-driven Design 10 3 Benchmark of Data-driven Automotive Design 12 3.1 Method of Global Benchmarking 11 3.2 Automotive Design 11 3.2.1 HMI Design and UI/UX 11 3.2.2 Hardware Design 12 3.2.3 Software Design 12 3.2.4 Convergence Design Process Model 13 3.3 Component Design Management 14 4 Vehicle Specification Design in Mobility Industry 16 4.1 Definition of Vehicle Specification 16 4.2 Field Study 17 4.3 Hypothesis 18 5 Three Preliminary Practical Case Studies for Vehicle Specification to Datadriven 21 5.1 Production Level 31 5.1.1 Background and Input 31 5.1.2 Data Process from Inventory to Designer 41 5.1.3 Output to Designer 51 5.2 Delivery Level 61 5.2.1 Background and Input 61 5.2.2 Data Process from Inventory to Designer 71 5.2.3 Output to Designer 81 5.3 Consumer Level 91 5.3.1 Background and Input 91 5.3.2 Data Process from Inventory to Designer 101 5.3.3 Output to Designer 111 6 Two Applications for Vehicle Designer 86 6.1 Real-time Dashboard DB for Decision Making 123 6.1.1 Searchable Infographic as a Designer's Tool 123 6.1.2 Scope and Method 123 6.1.3 Implementation 123 6.1.4 Result 124 6.1.5 Evaluation 124 6.1.6 Summary 124 6.2 Application to CAD for vehicle designer 124 6.2.1 CAD as a Designer's Tool 124 6.2.2 Scope and Method 125 6.2.3 Implementation and the Display of the CAD Software 125 6.2.4 Result 125 6.2.5 Evaluation: Usability Test with Eyetracking 126 6.2.6 Summary 128 7 Conclusion 96 7.1 Summary of Case Studies and Application Release 129 7.2 Impact of the Research 130 7.3 Further Study 131Docto

ISAR: Ein Autorensystem für Interaktive Tische

Developing augmented reality systems involves several challenges, that prevent end users and experts from non-technical domains, such as education, to experiment with this technology. In this research we introduce ISAR, an authoring system for augmented reality tabletops targeting users from non-technical domains. ISAR allows non-technical users to create their own interactive tabletop applications and experiment with the use of this technology in domains such as educations, industrial training, and medical rehabilitation.Die Entwicklung von Augmented-Reality-Systemen ist mit mehreren Herausforderungen verbunden, die Endbenutzer und Experten aus nicht-technischen Bereichen, wie z.B. dem Bildungswesen, daran hindern, mit dieser Technologie zu experimentieren. In dieser Forschung stellen wir ISAR vor, ein Autorensystem für Augmented-Reality-Tabletops, das sich an Benutzer aus nicht-technischen Bereichen richtet. ISAR ermöglicht es nicht-technischen Anwendern, ihre eigenen interaktiven Tabletop-Anwendungen zu erstellen und mit dem Einsatz dieser Technologie in Bereichen wie Bildung, industrieller Ausbildung und medizinischer Rehabilitation zu experimentieren

Augmented reality selection through smart glasses

O mercado de óculos inteligentes está em crescimento. Este crescimento abre a possibilidade de um dia os óculos inteligentes assumirem um papel mais ativo tal como os smartphones já têm na vida quotidiana das pessoas. Vários métodos de interação com esta tecnologia têm sido estudados, mas ainda não é claro qual o método que poderá ser o melhor para interagir com objetos virtuais. Neste trabalho são mencionados diversos estudos que se focam nos diferentes métodos de interação para aplicações de realidade aumentada. É dado destaque às técnicas de interação para óculos inteligentes tal como às suas vantagens e desvantagens. No contexto deste trabalho foi desenvolvido um protótipo de Realidade Aumentada para locais fechados, implementando três métodos de interação diferentes. Foram também estudadas as preferências do utilizador e sua vontade de executar o método de interação em público. Além disso, é extraído o tempo de reação que é o tempo entre a deteção de uma marca e o utilizador interagir com ela. Um protótipo de Realidade Aumentada ao ar livre foi desenvolvido a fim compreender os desafios diferentes entre uma aplicação de Realidade Aumentada para ambientes interiores e exteriores. Na discussão é possível entender que os utilizadores se sentem mais confortáveis usando um método de interação semelhante ao que eles já usam. No entanto, a solução com dois métodos de interação, função de toque nos óculos inteligentes e movimento da cabeça, permitem obter resultados próximos aos resultados do controlador. É importante destacar que os utilizadores não passaram por uma fase de aprendizagem os resultados apresentados nos testes referem-se sempre à primeira e única vez com o método de interação. O que leva a crer que o futuro de interação com óculos inteligentes possa ser uma fusão de diferentes técnicas de interação.The smart glasses’ market continues growing. It enables the possibility of someday smart glasses to have a presence as smartphones have already nowadays in people's daily life. Several interaction methods for smart glasses have been studied, but it is not clear which method could be the best to interact with virtual objects. In this research, it is covered studies that focus on the different interaction methods for reality augmented applications. It is highlighted the interaction methods for smart glasses and the advantages and disadvantages of each interaction method. In this work, an Augmented Reality prototype for indoor was developed, implementing three different interaction methods. It was studied the users’ preferences and their willingness to perform the interaction method in public. Besides that, it is extracted the reaction time which is the time between the detection of a marker and the user interact with it. An outdoor Augmented Reality application was developed to understand the different challenges between indoor and outdoor Augmented Reality applications. In the discussion, it is possible to understand that users feel more comfortable using an interaction method similar to what they already use. However, the solution with two interaction methods, smart glass’s tap function, and head movement allows getting results close to the results of the controller. It is important to highlight that was always the first time of the users, so there was no learning before testing. This leads to believe that the future of smart glasses interaction can be the merge of different interaction methods

Novel haptic interface For viewing 3D images

In recent years there has been an explosion of devices and systems capable of displaying stereoscopic 3D images. While these systems provide an improved experience over traditional bidimensional displays they often fall short on user immersion. Usually these systems only improve depth perception by relying on the stereopsis phenomenon. We propose a system that improves the user experience and immersion by having a position dependent rendering of the scene and the ability to touch the scene. This system uses depth maps to represent the geometry of the scene. Depth maps can be easily obtained on the rendering process or can be derived from the binocular-stereo images by calculating their horizontal disparity. This geometry is then used as an input to be rendered in a 3D display, do the haptic rendering calculations and have a position depending render of the scene. The author presents two main contributions. First, since the haptic devices have a finite work space and limited resolution, we used what we call detail mapping algorithms. These algorithms compress geometry information contained in a depth map, by reducing the contrast among pixels, in such a way that it can be rendered into a limited resolution display medium without losing any detail. Second, the unique combination of a depth camera as a motion capturing system, a 3D display and haptic device to enhance user experience. While developing this system we put special attention on the cost and availability of the hardware. We decided to use only off-the-shelf, mass consumer oriented hardware so our experiments can be easily implemented and replicated. As an additional benefit the total cost of the hardware did not exceed the one thousand dollars mark making it affordable for many individuals and institutions

Enhanced Virtuality: Increasing the Usability and Productivity of Virtual Environments

Mit stetig steigender Bildschirmauflösung, genauerem Tracking und fallenden Preisen stehen Virtual Reality (VR) Systeme kurz davor sich erfolgreich am Markt zu etablieren. Verschiedene Werkzeuge helfen Entwicklern bei der Erstellung komplexer Interaktionen mit mehreren Benutzern innerhalb adaptiver virtueller Umgebungen. Allerdings entstehen mit der Verbreitung der VR-Systeme auch zusätzliche Herausforderungen: Diverse Eingabegeräte mit ungewohnten Formen und Tastenlayouts verhindern eine intuitive Interaktion. Darüber hinaus zwingt der eingeschränkte Funktionsumfang bestehender Software die Nutzer dazu, auf herkömmliche PC- oder Touch-basierte Systeme zurückzugreifen. Außerdem birgt die Zusammenarbeit mit anderen Anwendern am gleichen Standort Herausforderungen hinsichtlich der Kalibrierung unterschiedlicher Trackingsysteme und der Kollisionsvermeidung. Beim entfernten Zusammenarbeiten wird die Interaktion durch Latenzzeiten und Verbindungsverluste zusätzlich beeinflusst. Schließlich haben die Benutzer unterschiedliche Anforderungen an die Visualisierung von Inhalten, z.B. Größe, Ausrichtung, Farbe oder Kontrast, innerhalb der virtuellen Welten. Eine strikte Nachbildung von realen Umgebungen in VR verschenkt Potential und wird es nicht ermöglichen, die individuellen Bedürfnisse der Benutzer zu berücksichtigen. Um diese Probleme anzugehen, werden in der vorliegenden Arbeit Lösungen in den Bereichen Eingabe, Zusammenarbeit und Erweiterung von virtuellen Welten und Benutzern vorgestellt, die darauf abzielen, die Benutzerfreundlichkeit und Produktivität von VR zu erhöhen. Zunächst werden PC-basierte Hardware und Software in die virtuelle Welt übertragen, um die Vertrautheit und den Funktionsumfang bestehender Anwendungen in VR zu erhalten. Virtuelle Stellvertreter von physischen Geräten, z.B. Tastatur und Tablet, und ein VR-Modus für Anwendungen ermöglichen es dem Benutzer reale Fähigkeiten in die virtuelle Welt zu übertragen. Des Weiteren wird ein Algorithmus vorgestellt, der die Kalibrierung mehrerer ko-lokaler VR-Geräte mit hoher Genauigkeit und geringen Hardwareanforderungen und geringem Aufwand ermöglicht. Da VR-Headsets die reale Umgebung der Benutzer ausblenden, wird die Relevanz einer Ganzkörper-Avatar-Visualisierung für die Kollisionsvermeidung und das entfernte Zusammenarbeiten nachgewiesen. Darüber hinaus werden personalisierte räumliche oder zeitliche Modifikationen vorgestellt, die es erlauben, die Benutzerfreundlichkeit, Arbeitsleistung und soziale Präsenz von Benutzern zu erhöhen. Diskrepanzen zwischen den virtuellen Welten, die durch persönliche Anpassungen entstehen, werden durch Methoden der Avatar-Umlenkung (engl. redirection) kompensiert. Abschließend werden einige der Methoden und Erkenntnisse in eine beispielhafte Anwendung integriert, um deren praktische Anwendbarkeit zu verdeutlichen. Die vorliegende Arbeit zeigt, dass virtuelle Umgebungen auf realen Fähigkeiten und Erfahrungen aufbauen können, um eine vertraute und einfache Interaktion und Zusammenarbeit von Benutzern zu gewährleisten. Darüber hinaus ermöglichen individuelle Erweiterungen des virtuellen Inhalts und der Avatare Einschränkungen der realen Welt zu überwinden und das Erlebnis von VR-Umgebungen zu steigern