Camera-based virtual environment interaction on mobile devices

Mobile virtual environments, with real-time 3D and 2D graphics, are now possible on smart phone and other camera-enabled devices. Using computer vision, the camera sensor can be treated as an input modality in applications by analyzing the incoming live video. We present our tracking algorithm and several mobile virtual environment and gaming prototypes including: a 3D first person shooter, a 2D puzzle game and a simple action game. Camera-based interaction provides a user experience that is not possible through traditional means, and maximizes the use of the limited display size. © Springer-Verlag Berlin Heidelberg 2006

Mobile phone interaction techniques for rural economy development - a review

Rural communities, especially in developing countries, are often neglected in terms of facilities and services that aid their social and economic development. This is evident even in software development processes, in that these groups of users or potential users’ are often not taken into consideration. The resultant effect is that they may not use it or use it sparingly. The objective of this study is to identify the various researches on interaction techniques and user interface design as a first step to the design of suitable mobile interactions and user interfaces for rural users. This research project is also aimed at socio-economic development and adding value to mobile phone users in Dwesa, a rural community in South Africa. This paper presents a literature survey of interaction techniques and user-interfaces. An analysis of the interaction techniques with respect to their suitability, availability of technologies, user capabilities for implementation in a rural context is discussed. Descriptive statistics of users’ current phones interaction facilities in the rural community which briefly illustrates users’ experiences and capabilities in different interaction modes is also presented.KEY WORDS: Interaction Techniques, Mobile phone, User Interface, ICT, Rural Development

A face tracking algorithm for user interaction in mobile devices

A new face tracking algorithm, and a human-computer interaction technique based on this algorithm, are proposed for use on mobile devices. The face tracking algorithm considers the limitations of mobile use case - constrained computational resources and varying environmental conditions. The solution is based on color comparisons and works on images gathered from the front camera of a device. The face tracking system generates 2D face position as an output that can be used for controlling different applications. Two of such applications are also presented in this work; the first example uses face position to determine the viewpoint, and the second example enables an intuitive way of browsing large images. © 2009 IEEE

Are my Apps Peeking? Comparing Nudging Mechanisms to Raise Awareness of Access to Mobile Front-facing Camera

Mobile applications that are granted permission to access the device’s camera can access it at any time without necessarily showing the camera feed to the user or communicating that it is being used. This lack of transparency raises privacy concerns, which are exacerbated by the increased adoption of applications that leverage front-facing cameras. Through a focus group we identified three promising approaches for nudging the user that the camera is being accessed, namely: notification bar, frame, and camera preview. We experimented with accompanying each nudging method with vibrotactile and audio feedback. Results from a user study (N=15) show that while using frame nudges is the least annoying and interrupting, but was less understandable than the camera feed and notifications. On the other hand, participants found that indicating camera usage by showing its feed or by using notifications is easy to understand. We discuss how these nudges raise user awareness and the effects on app usage and perception

Development of computer vision algorithms using J2ME for mobile phone applications.

This thesis describes research on the use of Java to develop cross-platform computer vision applications for mobile phones with integrated cameras. The particular area of research that we are interested in is Mobile Augmented Reality (AR). Currently there is no computer vision library which can be used for mobile Augmented Reality using the J2ME platform. This thesis introduces the structure of our J2ME computer vision library and describes the implementation of algorithms in our library. We also present several sample applications on J2ME enabled mobile phones and report on experiments conducted to evaluate the compatibility, portability and efficiency of the implemented algorithms

Study of the interaction with a virtual 3D environment displayed on a smartphone

Les environnements virtuels à 3D (EV 3D) sont de plus en plus utilisés dans différentes applications telles que la CAO, les jeux ou la téléopération. L'évolution des performances matérielles des Smartphones a conduit à l'introduction des applications 3D sur les appareils mobiles. En outre, les Smartphones offrent de nouvelles capacités bien au-delà de la communication vocale traditionnelle qui sont consentis par l'intégrité d'une grande variété de capteurs et par la connectivité via Internet. En conséquence, plusieurs intéressantes applications 3D peuvent être conçues en permettant aux capacités de l'appareil d'interagir dans un EV 3D. Sachant que les Smartphones ont de petits et aplatis écrans et que EV 3D est large, dense et contenant un grand nombre de cibles de tailles différentes, les appareils mobiles présentent certaines contraintes d'interaction dans l'EV 3D comme : la densité de l'environnement, la profondeur de cibles et l'occlusion. La tâche de sélection fait face à ces trois problèmes pour sélectionner une cible. De plus, la tâche de sélection peut être décomposée en trois sous-tâches : la Navigation, le Pointage et la Validation. En conséquence, les chercheurs dans un environnement virtuel 3D ont développé de nouvelles techniques et métaphores pour l'interaction en 3D afin d'améliorer l'utilisation des applications 3D sur les appareils mobiles, de maintenir la tâche de sélection et de faire face aux problèmes ou facteurs affectant la performance de sélection. En tenant compte de ces considérations, cette thèse expose un état de l'art des techniques de sélection existantes dans un EV 3D et des techniques de sélection sur Smartphone. Il expose les techniques de sélection dans un EV 3D structurées autour des trois sous-tâches de sélection: navigation, pointage et validation. En outre, il décrit les techniques de désambiguïsation permettant de sélectionner une cible parmi un ensemble d'objets présélectionnés. Ultérieurement, il expose certaines techniques d'interaction décrites dans la littérature et conçues pour être implémenter sur un Smartphone. Ces techniques sont divisées en deux groupes : techniques effectuant des tâches de sélection bidimensionnelle sur un Smartphone et techniques exécutant des tâches de sélection tridimensionnelle sur un Smartphone. Enfin, nous exposons les techniques qui utilisaient le Smartphone comme un périphérique de saisie. Ensuite, nous discuterons la problématique de sélection dans un EV 3D affichée sur un Smartphone. Il expose les trois problèmes identifiés de sélection : la densité de l'environnement, la profondeur des cibles et l'occlusion. Ensuite, il établit l'amélioration offerte par chaque technique existante pour la résolution des problèmes de sélection. Il analyse les atouts proposés par les différentes techniques, la manière dont ils éliminent les problèmes, leurs avantages et leurs inconvénients. En outre, il illustre la classification des techniques de sélection pour un EV 3D en fonction des trois problèmes discutés (densité, profondeur et occlusion) affectant les performances de sélection dans un environnement dense à 3D. Hormis pour les jeux vidéo, l'utilisation d'environnement virtuel 3D sur Smartphone n'est pas encore démocratisée. Ceci est dû au manque de techniques d'interaction proposées pour interagir avec un dense EV 3D composé de nombreux objets proches les uns des autres et affichés sur un petit écran aplati et les problèmes de sélection pour afficher l' EV 3D sur un petit écran plutôt sur un grand écran. En conséquence, cette thèse se concentre sur la proposition et la description du fruit de cette étude : la technique d'interaction DichotoZoom. Elle compare et évalue la technique proposée à la technique de circulation suggérée par la littérature. L'analyse comparative montre l'efficacité de la technique DichotoZoom par rapport à sa contrepartie. Ensuite, DichotoZoom a été évalué selon les différentes modalités d'interaction disponibles sur les Smartphones. Cette évaluation montre la performance de la technique de sélection proposée basée sur les quatre modalités d'interaction suivantes : utilisation de boutons physiques ou sous forme de composants graphiques, utilisation d'interactions gestuelles via l'écran tactile ou le déplacement de l'appareil lui-même. Enfin, cette thèse énumère nos contributions dans le domaine des techniques d'interaction 3D utilisées dans un environnement virtuel 3D dense affiché sur de petits écrans et propose des travaux futurs.3D Virtual Environments (3D VE) are more and more used in different applications such as CAD, games, or teleoperation. Due to the improvement of smartphones hardware performance, 3D applications were also introduced to mobile devices. In addition, smartphones provide new computing capabilities far beyond the traditional voice communication. They are permitted by the variety of built-in sensors and the internet connectivity. In consequence, interesting 3D applications can be designed by enabling the device capabilities to interact in a 3D VE. Due to the fact that smartphones have small and flat screens and that a 3D VE is wide and dense with a large number of targets of various sizes, mobile devices present some constraints in interacting on the 3D VE like: the environment density, the depth of targets and the occlusion. The selection task faces these three problems to select a target. In addition, the selection task can be decomposed into three subtasks: Navigation, Pointing and Validation. In consequence, researchers in 3D virtual environment have developed new techniques and metaphors for 3D interaction to improve 3D application usability on mobile devices, to support the selection task and to face the problems or factors affecting selection performance. In light of these considerations, this thesis exposes a state of the art of the existing selection techniques in 3D VE and the selection techniques on smartphones. It exposes the selection techniques in 3D VE structured around the selection subtasks: navigation, pointing and validation. Moreover, it describes disambiguation techniques providing the selection of a target from a set of pre-selected objects. Afterward, it exposes some interaction techniques described in literature and designed for implementation on Smartphone. These techniques are divided into two groups: techniques performing two-dimensional selection tasks on smartphones, and techniques performing three-dimensional selection tasks on smartphones. Finally, we expose techniques that used the smartphone as an input device. Then, we will discuss the problematic of selecting in 3D VE displayed on a Smartphone. It exposes the three identified selection problems: the environment density, the depth of targets and the occlusion. Afterward, it establishes the enhancement offered by each existing technique in solving the selection problems. It analysis the assets proposed by different techniques, the way they eliminates the problems, their advantages and their inconvenient. Furthermore, it illustrates the classification of the selection techniques for 3D VE according to the three discussed problems (density, depth and occlusion) affecting the selection performance in a dense 3D VE. Except for video games, the use of 3D virtual environment (3D VE) on Smartphone has not yet been popularized. This is due to the lack of interaction techniques to interact with a dense 3D VE composed of many objects close to each other and displayed on a small and flat screen and the selection problems to display the 3D VE on a small screen rather on a large screen. Accordingly, this thesis focuses on defining and describing the fruit of this study: DichotoZoom interaction technique. It compares and evaluates the proposed technique to the Circulation technique, suggested by the literature. The comparative analysis shows the effectiveness of DichotoZoom technique compared to its counterpart. Then, DichotoZoom was evaluated in different modalities of interaction available on Smartphones. It reports on the performance of the proposed selection technique based on the following four interaction modalities: using physical buttons, using graphical buttons, using gestural interactions via touchscreen or moving the device itself. Finally, this thesis lists our contributions to the field of 3D interaction techniques used in a dense 3D virtual environment displayed on small screens and proposes some future works

Visual based finger interactions for mobile phones

Vision based technology such as motion detection has long been limited to the domain of powerful processor intensive systems such as desktop PCs and specialist hardware solutions. With the advent of much faster mobile phone processors and memory, a plethora of feature rich software and hardware is being deployed onto the mobile platform, most notably onto high powered devices called smart phones. Interaction interfaces such as touchscreens allow for improved usability but obscure the phone’s screen. Since the majority of smart phones are equipped with cameras, it has become feasible to combine their powerful processors, large memory capacity and the camera to support new ways of interacting with the phone which do not obscure the screen. However, it is not clear whether or not these processor intensive visual interactions can in fact be run at an acceptable speed on current mobile handsets or whether they will offer the user a better experience than the current number pad and direction keys present on the majority of mobile phones. A vision based finger interaction technique is proposed which uses the back of device camera to track the user’s finger. This allows the user to interact with the mobile phone with mouse based movements, gestures and steering based interactions. A simple colour thresholding algorithm was implemented in Java, Python and C++. Various benchmarks and tests conducted on a Nokia N95 smart phone revealed that on current hardware and with current programming environments only native C++ yields results plausible for real time interactions (a key requirement for vision based interactions). It is also shown that different lighting levels and background environments affects the accuracy of the system with background and finger contrast playing a large role. Finally a user study was conducted to ascertain the overall user’s satisfaction between keypad interactions and the finger interaction techniques concluding that the new finger interaction technique is well suited to steering based interactions and in time, mouse style movements. Simple navigation is better suited to the directional keypad

Mobile phones interaction techniques for second economy people

Second economy people in developing countries are people living in communities that are underserved in terms of basic amenities and social services. Due to literacy challenges and user accessibility problems in rural communities, it is often difficult to design user interfaces that conform to the capabilities and cultural experiences of low-literacy rural community users. Rural community users are technologically illiterate and lack the knowledge of the potential of information and communication technologies. In order to embrace new technology, users will need to perceive the user interface and application as useful and easy to interact with. This requires proper understanding of the users and their socio-cultural environment. This will enable the interfaces and interactions to conform to their behaviours, motivations as well as cultural experiences and preferences and thus enhance usability and user experience. Mobile phones have the potential to increase access to information and provide a platform for economic development in rural communities. Rural communities have economic potential in terms of agriculture and micro-enterprises. Information technology can be used to enhance socio-economic activities and improve rural livelihood. We conducted a study to design user interfaces for a mobile commerce application for micro-entrepreneurs in a rural community in South Africa. The aim of the study was to design mobile interfaces and interaction techniques that are easy to use and meet the cultural preferences and experiences of users who have little to no previous experience of mobile commerce technology. And also to explore the potentials of information technologies rural community users, and bring mobile added value services to rural micro-entrepreneurs. We applied a user-centred design approach in Dwesa community and used qualitative and quantitative research methods to collect data for the design of the user interfaces (graphic user interface and voice user interface) and mobile commerce application. We identified and used several interface elements to design and finally evaluate the graphical user interface. The statistics analysis of the evaluation results show that the users in the community have positive perception of the usefulness of the application, the ease of use and intention to use the application. Community users with no prior experience with this technology were able to learn and understand the interface, recorded minimum errors and a high level of v precision during task performance when they interacted with the shop-owner graphic user interface. The voice user interface designed in this study consists of two flavours (dual tone multi-frequency input and voice input) for rural users. The evaluation results show that community users recorded higher tasks successes and minimum errors with the dual tone multi-frequency input interface than the voice only input interface. Also, a higher percentage of users prefer the dual tone multi-frequency input interface. The t-Test statistical analysis performed on the tasks completion times and error rate show that there was significant statistical difference between the dual tone multi-frequency input interface and the voice input interface. The interfaces were easy to learn, understand and use. Properly designed user interfaces that meet the experience and capabilities of low-literacy users in rural areas will improve usability and users‟ experiences. Adaptation of interfaces to users‟ culture and preferences will enhance information services accessibility among different user groups in different regions. This will promote technology acceptance in rural communities for socio-economic benefits. The user interfaces presented in this study can be adapted to different cultures to provide similar services for marginalised communities in developing countrie

Perceptually Optimized Visualization on Autostereoscopic 3D Displays

The family of displays, which aims to visualize a 3D scene with realistic depth, are known as "3D displays". Due to technical limitations and design decisions, such displays create visible distortions, which are interpreted by the human vision as artefacts. In absence of visual reference (e.g. the original scene is not available for comparison) one can improve the perceived quality of the representations by making the distortions less visible. This thesis proposes a number of signal processing techniques for decreasing the visibility of artefacts on 3D displays. The visual perception of depth is discussed, and the properties (depth cues) of a scene which the brain uses for assessing an image in 3D are identified. Following the physiology of vision, a taxonomy of 3D artefacts is proposed. The taxonomy classifies the artefacts based on their origin and on the way they are interpreted by the human visual system. The principles of operation of the most popular types of 3D displays are explained. Based on the display operation principles, 3D displays are modelled as a signal processing channel. The model is used to explain the process of introducing distortions. It also allows one to identify which optical properties of a display are most relevant to the creation of artefacts. A set of optical properties for dual-view and multiview 3D displays are identified, and a methodology for measuring them is introduced. The measurement methodology allows one to derive the angular visibility and crosstalk of each display element without the need for precision measurement equipment. Based on the measurements, a methodology for creating a quality profile of 3D displays is proposed. The quality profile can be either simulated using the angular brightness function or directly measured from a series of photographs. A comparative study introducing the measurement results on the visual quality and position of the sweet-spots of eleven 3D displays of different types is presented. Knowing the sweet-spot position and the quality profile allows for easy comparison between 3D displays. The shape and size of the passband allows depth and textures of a 3D content to be optimized for a given 3D display. Based on knowledge of 3D artefact visibility and an understanding of distortions introduced by 3D displays, a number of signal processing techniques for artefact mitigation are created. A methodology for creating anti-aliasing filters for 3D displays is proposed. For multiview displays, the methodology is extended towards so-called passband optimization which addresses Moiré, fixed-pattern-noise and ghosting artefacts, which are characteristic for such displays. Additionally, design of tuneable anti-aliasing filters is presented, along with a framework which allows the user to select the so-called 3d sharpness parameter according to his or her preferences. Finally, a set of real-time algorithms for view-point-based optimization are presented. These algorithms require active user-tracking, which is implemented as a combination of face and eye-tracking. Once the observer position is known, the image on a stereoscopic display is optimised for the derived observation angle and distance. For multiview displays, the combination of precise light re-direction and less-precise face-tracking is used for extending the head parallax. For some user-tracking algorithms, implementation details are given, regarding execution of the algorithm on a mobile device or on desktop computer with graphical accelerator