American Sign Language Assistant

Our implementation of a prototype computer vision system to help the deaf and mute communicate in a shopping setting. Our system uses live video feeds to recognize American Sign Language (ASL) gestures and notify shop clerks of deaf and mute patrons’ intents. It generates a video dataset in the Unity Game Engine of 3D humanoid models in a shop setting performing ASL signs. Our system uses OpenPose to detect and recognize the bone points of the human body from the live feed. The system then represents the motion sequences as high dimensional skeleton joint point trajectories followed by a time-warping technique to generate a temporal RGB image using the Seq2Im technique. This image is then fed to the image classification algorithms that classify the gesture performed to the shop clerk. We carried out experiments to analyze the performance of this methodology on the Leap Motion Controller dataset and NTU RGB+D dataset using the SVM and LeNet-5 models. We also tested 3D vs 2D bone point dataset performance and found 90% accuracy for the 2D skeleton dataset

Interfaces for human-centered production and use of computer graphics assets

L'abstract è presente nell'allegato / the abstract is in the attachmen

Machinima And Video-based Soft Skills Training

Multimedia training methods have traditionally relied heavily on video based technologies and significant research has shown these to be very effective training tools. However production of video is time and resource intensive. Machinima (pronounced \u27muh-sheen-eh-mah\u27) technologies are based on video gaming technology. Machinima technology allows video game technology to be manipulated into unique scenarios based on entertainment or training and practice applications. Machinima is the converting of these unique scenarios into video vignettes that tell a story. These vignettes can be interconnected with branching points in much the same way that education videos are interconnected as vignettes between decision points. This study addressed the effectiveness of machinima based soft-skills education using avatar actors versus the traditional video teaching application using human actors. This research also investigated the difference between presence reactions when using avatar actor produced video vignettes as compared to human actor produced video vignettes. Results indicated that the difference in training and/or practice effectiveness is statistically insignificant for presence, interactivity, quality and the skill of assertiveness. The skill of active listening presented a mixed result indicating the need for careful attention to detail in situations where body language and facial expressions are critical to communication. This study demonstrates that a significant opportunity exists for the exploitation of avatar actors in video based instruction

Structural Performance Monitoring Using a Dynamic Data-Driven BIM Environment

Structural health monitoring data has not been fully leveraged to support asset management due to a lack of effective integration with other datasets. A Building Information Modelling (BIM) approach is presented to leverage structural monitoring data in a dynamic manner. The approach allows for the automatic generation of parametric BIM models of structural monitoring systems that include time-series sensor data; and it enables data-driven and dynamic visualisation in an interactive 3D environment. The approach supports dynamic visualisation of key structural performance parameters, allows for the seamless updating and long-term management of data, and facilitates data exchange by generating Industry Foundation Classes (IFC) compliant models. A newly-constructed bridge near Stafford, UK, with an integrated fibre-optic sensor based monitoring system was used to test the capabilities of the developed approach. The case study demonstrated how the developed approach facilitates more intuitive data interpretation, provides a user-friendly interface to communicate with various stakeholders, allows for the identification of malfunctioning sensors thus contributing to the assessment of monitoring system durability, and forms the basis for a powerful data-driven asset management tool. In addition, this project highlights the potential benefits of investing in the development of data-driven and dynamic BIM environments

Dirichlet belief networks for topic structure learning

Recently, considerable research effort has been devoted to developing deep architectures for topic models to learn topic structures. Although several deep models have been proposed to learn better topic proportions of documents, how to leverage the benefits of deep structures for learning word distributions of topics has not yet been rigorously studied. Here we propose a new multi-layer generative process on word distributions of topics, where each layer consists of a set of topics and each topic is drawn from a mixture of the topics of the layer above. As the topics in all layers can be directly interpreted by words, the proposed model is able to discover interpretable topic hierarchies. As a self-contained module, our model can be flexibly adapted to different kinds of topic models to improve their modelling accuracy and interpretability. Extensive experiments on text corpora demonstrate the advantages of the proposed model.Comment: accepted in NIPS 201

Volumetric kombat:a case study on developing a VR game with Volumetric Video

This paper presents a case study on the development of a Virtual Reality (VR) game using Volumetric Video (VV) for character animation. We delve into the potential of VV, a technology that fuses video and depth sensor data, which has progressively matured since its initial introduction in 1995. Despite its potential to deliver unmatched realism and dynamic 4D sequences, VV applications are predominantly used in non-interactive scenarios. We explore the barriers to entry such as high costs associated with large-scale VV capture systems and the lack of tools optimized for VV in modern game engines. By actively using VV to develop a VR game, we examine and overcome these constraints developing a set of tools that address these challenges. Drawing lessons from past games, we propose an open-source data processing workflow for future VV games. This case study provides insights into the opportunities and challenges of VV in game development and contributes towards making VV more accessible for creators and researchers

Automatically Detecting Visual Bugs in HTML5 <canvas> Games

The HTML5 is used to display high quality graphics in web applications such as web games (i.e., games). However, automatically testing games is not possible with existing web testing techniques and tools, and manual testing is laborious. Many widely used web testing tools rely on the Document Object Model (DOM) to drive web test automation, but the contents of the are not represented in the DOM. The main alternative approach, snapshot testing, involves comparing oracle snapshot images with test-time snapshot images using an image similarity metric to catch visual bugs, i.e., bugs in the graphics of the web application. However, creating and maintaining oracle snapshot images for games is onerous, defeating the purpose of test automation. In this paper, we present a novel approach to automatically detect visual bugs in games. By leveraging an internal representation of objects on the , we decompose snapshot images into a set of object images, each of which is compared with a respective oracle asset (e.g., a sprite) using four similarity metrics: percentage overlap, mean squared error, structural similarity, and embedding similarity. We evaluate our approach by injecting 24 visual bugs into a custom game, and find that our approach achieves an accuracy of 100%, compared to an accuracy of 44.6% with traditional snapshot testing.Comment: Accepted at ASE 2022 conferenc

Virtual Cinematography: Beyond Big Studio Production

In the current production environment, the ability to previsualize shots utilizing a virtual camera system requires expensive hardware and large motion capture spaces only available to large studio environments. By leveraging consumer-level technologies such as tablets and motion gaming controllers as well as merging the cinematic techniques of film with the real-time benefits of game engines, it is possible to develop a hybrid interface that would lower the barrier of entry for virtual production. Utilizing affordable hardware, an intuitive user interface, and an intelligent camera system, the SmartVCS is a new virtual inematography platform that provides professional directors as well as a new market of amateur filmmakers the ability to previsualize their films or game cinematics with familiar and accessible technology. This system has potential applications to other areas including game level design, real-time compositing & post-production, and architectural visualization. In addition, this system has the ability to expand as a human-computer interface for video games, robotics, and medicine as a functional hybrid freespace input device.M.S., Digital Media -- Drexel University, 201