A MATCH-MOVING METHOD COMBINING AI AND SFM ALGORITHMS IN HISTORICAL FILM FOOTAGE

Searching for suitable material for photogrammetry is a key part in the documentation of Cultural Heritage. Photogrammetry can be used to produce a metrically certified 3D model. Material contained in historical film footage archives is especially useful for documentation when the heritage has been lost. In this research an innovative match-moving method is proposed that aims to exploit Artificial Intelligence and SfM algorithms to identify the frames extracted from a film footage in which the lost monument appears and that are suitable to be processed with photogrammetry for its 3D reconstruction. First of all the identification and tracking of the heritage in the videos was performed training an object detection Neural Network. Then the frames detected were automatically extracted with the coordinates of the bounding boxes that contain the monument. The camera motions were identified by selecting only the shots taken from multiple points of view of the same scene and analysing the evolution of the bounding boxes position over time. A further check of the material was necessary to select only sequences and to eliminate single frames and images from different historic periods. After this process, only the correct frames were automatically selected and processed with photogrammetry and the quality of the obtained 3D model was assessed. The method experimented in this research represents a powerful tool in the field of Cultural Heritage because it makes the selection of suitable material for photogrammetry automatic. Moreover it offers important insights that could be extended to other sectors

Markerless Motion Capture in the Crowd

This work uses crowdsourcing to obtain motion capture data from video recordings. The data is obtained by information workers who click repeatedly to indicate body configurations in the frames of a video, resulting in a model of 2D structure over time. We discuss techniques to optimize the tracking task and strategies for maximizing accuracy and efficiency. We show visualizations of a variety of motions captured with our pipeline then apply reconstruction techniques to derive 3D structure.Comment: Presented at Collective Intelligence conference, 2012 (arXiv:1204.2991

Virtual Cinematography: Beyond Big Studio Production

In the current production environment, the ability to previsualize shots utilizing a virtual camera system requires expensive hardware and large motion capture spaces only available to large studio environments. By leveraging consumer-level technologies such as tablets and motion gaming controllers as well as merging the cinematic techniques of film with the real-time benefits of game engines, it is possible to develop a hybrid interface that would lower the barrier of entry for virtual production. Utilizing affordable hardware, an intuitive user interface, and an intelligent camera system, the SmartVCS is a new virtual inematography platform that provides professional directors as well as a new market of amateur filmmakers the ability to previsualize their films or game cinematics with familiar and accessible technology. This system has potential applications to other areas including game level design, real-time compositing & post-production, and architectural visualization. In addition, this system has the ability to expand as a human-computer interface for video games, robotics, and medicine as a functional hybrid freespace input device.M.S., Digital Media -- Drexel University, 201

Teaching Visual Effects for Audiovisual Production using Digital Learning Objects

This Ph.D. research deals with the production of a digital learning object (DLO) to aid teaching activity, more specifically in the execution of practical exercises for the production of visual effects (VFX) in audiovisual in higher education courses. The study performs a literature review on the use of Information and Communication Technologies (ICTs) in education, Learning Objects (LOs), Digital Learning Objects (DLOs), and Blended Learning (BL). In the research, it seeks to understand the methods and procedures of design, implementation, evaluation, and validation of such technologies to be able to apply to the prototype to be developed. It presents the stages of development of the prototype and all the challenges faced for its realization. With the fully functional tool already developed and tested, the methodological procedures used for its use and evaluation are demonstrated. The field research is reported and documented, presenting the results achieved. Forms were used with students participating in the research, in addition to blind analysis of the practical work they developed. Such data are analyzed and the reflections and contributions from this whole experimentation process are presented. It was possible to conclude that the tool (DLO) designed managed to achieve its objectives and answer the research questions that fostered it, having been considered by the participants as useful, motivating, and functional in helping the learning process. The blind analysis undertaken did not show significant differences between the control and test groups in the quality of the work performed, however, it evidenced the possibility of using other methodologies in conjunction with those used, to improve future research. It is believed that all the discussion and experimentation carried out and documented reaches its objectives positively and demonstrates its contribution in to assist in the scientific discussion about the pedagogical effectiveness of DLOs, create an unprecedented DLO, in addition to presenting the DLOs production methodologies used.Esta pesquisa de doutoramento trata sobre a produção de um objeto de aprendizagem digital para o auxílio à atividade docente, mais especificamente na execução de exercícios práticos para a produção de efeitos visuais em audiovisual em cursos de nível superior. O estudo realiza uma revisão da literatura sobre o uso de Tecnologias da Informação e Comunicação na educação, objetos de aprendizagem, objetos de aprendizagem digitais e aprendizagem combinada. Na investigação, busca compreender os métodos e procedimentos de design, implementação, avaliação e validação de tais tecnologias para aplicar ao protótipo a ser desenvolvido. Apresenta as etapas de desenvolvimento do protótipo e todos os desafios enfrentados para sua concretização. Com a ferramenta totalmente funcional já desenvolvida e testada, são demonstrados os procedimentos metodológicos utilizados para seu uso e avaliação. A pesquisa de campo encontra-se relatada e documentada, apresentando os resultados alcançados. Foram utilizados formulários com os alunos participantes da pesquisa, além de análise cega sobre os trabalhos práticos por eles desenvolvidos. Tais dados são analisados e apresentadas as reflexões e contribuições advindas de todo este processo de experimentação. Foi possível concluir que a ferramenta (objeto de aprendizagem digital) projetada conseguiu atingir seus objetivos e responder às perguntas de pesquisa que o fomentou, tendo sido considerada pelos participantes como útil, motivadora e funcional para o auxílio do processo de aprendizagem. A análise cega empreendida não demonstrou diferenças significativas entre os grupos controle e teste na qualidade dos trabalhos executados, contudo, evidenciou a possibilidade de utilização de outras metodologias em conjunto com as utilizadas, para aprimoramento de pesquisas futuras. Acredita-se que toda a discussão e experimentação realizada e documentada atinge positivamente seus objetivos e demonstra sua contribuição ao auxiliar na discussão científica sobre a efetividade pedagógica dos objetos de aprendizagem digitais, criar um destes objetos/software inédito, além de apresentar as metodologias de produção utilizadas

A study of how the technological advancements in capturing believable facial emotion in Computer Generated (CG) characters in film has facilitated crossing the uncanny valley

A Research Report submitted in partial fulfilment of the requirement for the Degree of Masters of Arts in Digital Animation at the University of the Witwatersrand (School of Digital Arts) Johannesburg, South AfricaIn recent years, the quest for capturing authentic emotion convincingly in computer generated (CG) characters to assist exceedingly complex narrative expressions in modern cinema has intensified. Conveying human emotion in a digital human-like character is widely accepted to be the most challenging and elusive task for even the most skilled animators. Contemporary filmmakers have increasingly looked to complex digital tools that essentially manipulate the visual design of cinema through innovative techniques to reach levels of undetectable integration of CG characters. In trying to assess how modern cinema is pursuing the realistic integration of CG human-like characters in digital film with frenetic interest despite the risk of box office failure associated with the uncanny valley, this report focuses on the progress of the advances in the technique of facial motion capture. The uncanny valley hypothesis, based on a theory by Sigmund Freud, was coined in 1970 by Japanese robotics professor, Masahiro Mori. Mori suggested that people are increasingly comfortable with robots the more human-like they appear, but only up to a point. At that turning point, when the robot becomes too human-like, it arouses feelings of repulsion. When movement is added to this equation, viewers’ sense of the uncanny is heightened when the movement is deemed to be unreal. Motion capture is the technique of mimicking and capturing realistic movement by utilising technology that enables the process of translating a live actor’s performance into a digital performance. By capturing and transferring the data collected from sensors placed on a body suit or tracked from a high definition video, computer artists are able to drive the movement of a corresponding CG character in a 3-Dimensional (3D) programme. The attention of this study is narrowed to the progress of the techniques developed during a prolific decade for facial motion capture in particular. Regardless of the conflicting discourse surrounding the use of motion capture technology, these phenomenal improvements have allowed filmmakers to overcome that aspect of the uncanny valley associated with detecting realistic movement and facial expression. The progress of facial motion capture is investigated through the lens of selected films released during the period of 2001 to 2012. The two case studies, The Curious Case of Benjamin Button (2008) and Avatar (2009) were chosen for their individual achievement and innovative techniques that introduced new methods of facial capture. Digital images are said to undermine the reality status of cinematic images by challenging the foundation of long held theories of cinematic realist theory. These theories rooted in the indexical basis of photography, have proved to be the origin of contemporary viewers' notion of cinematic realism. However, the relationship between advanced digital effects and modern cinematic realism has created a perceptual complexity that warrants closer scrutiny. In addressing the paradoxical effect that photo-real cinematic realism is having on the basic comprehension of realism in film, the history of the seminal claims made by recognized realist film theorists is briefly examined

Enhancing Mesh Deformation Realism: Dynamic Mesostructure Detailing and Procedural Microstructure Synthesis

Propomos uma solução para gerar dados de mapas de relevo dinâmicos para simular deformações em superfícies macias, com foco na pele humana. A solução incorpora a simulação de rugas ao nível mesoestrutural e utiliza texturas procedurais para adicionar detalhes de microestrutura estáticos. Oferece flexibilidade além da pele humana, permitindo a geração de padrões que imitam deformações em outros materiais macios, como couro, durante a animação. As soluções existentes para simular rugas e pistas de deformação frequentemente dependem de hardware especializado, que é dispendioso e de difícil acesso. Além disso, depender exclusivamente de dados capturados limita a direção artística e dificulta a adaptação a mudanças. Em contraste, a solução proposta permite a síntese dinâmica de texturas que se adaptam às deformações subjacentes da malha de forma fisicamente plausível. Vários métodos foram explorados para sintetizar rugas diretamente na geometria, mas sofrem de limitações como auto-interseções e maiores requisitos de armazenamento. A intervenção manual de artistas na criação de mapas de rugas e mapas de tensão permite controle, mas pode ser limitada em deformações complexas ou onde maior realismo seja necessário. O nosso trabalho destaca o potencial dos métodos procedimentais para aprimorar a geração de padrões de deformação dinâmica, incluindo rugas, com maior controle criativo e sem depender de dados capturados. A incorporação de padrões procedimentais estáticos melhora o realismo, e a abordagem pode ser estendida além da pele para outros materiais macios.We propose a solution for generating dynamic heightmap data to simulate deformations for soft surfaces, with a focus on human skin. The solution incorporates mesostructure-level wrinkles and utilizes procedural textures to add static microstructure details. It offers flexibility beyond human skin, enabling the generation of patterns mimicking deformations in other soft materials, such as leater, during animation. Existing solutions for simulating wrinkles and deformation cues often rely on specialized hardware, which is costly and not easily accessible. Moreover, relying solely on captured data limits artistic direction and hinders adaptability to changes. In contrast, our proposed solution provides dynamic texture synthesis that adapts to underlying mesh deformations. Various methods have been explored to synthesize wrinkles directly to the geometry, but they suffer from limitations such as self-intersections and increased storage requirements. Manual intervention by artists using wrinkle maps and tension maps provides control but may be limited to the physics-based simulations. Our research presents the potential of procedural methods to enhance the generation of dynamic deformation patterns, including wrinkles, with greater creative control and without reliance on captured data. Incorporating static procedural patterns improves realism, and the approach can be extended to other soft-materials beyond skin

Missouri S&T Magazine Fall 2002

https://scholarsmine.mst.edu/alumni-magazine/1059/thumbnail.jp

A design-led investigation of augmented reality: a case of AR for board games

This research presents a new approach to using Augmented Reality for presenting and navigating through information in a scenario involving a board game. Learning the rules and strategies of a new game can be a challenging task for players. In particular, this can be an intimidating task for people who do not regularly play board games. Many board games are complex and new players require several sessions to learn them to be able to play at a reasonable speed while providing a challenge to experienced players. New players generally prefer not to read the manuals and instead learn to play the game on the fly and by asking experienced players. Several issues arise here for more experienced players who need to explain the game rules on the fly and when explaining strategies. This research therefore has been guided by the following question: How can an Augmented Reality experience be designed for board games to communicate information in an interactive and non-intrusive way? This has been addressed by conducting an ethnographic inspired study on public board game sessions and a research through design methodology. The board game, Settlers of Catan was selected and the Activity Theory framework was used to explore the relationship between the components in a game session. Interactive augmented information overlays were designed using design principles of visual communication and information design. Guided by Flow Theory, the experience was designed to be in-situ and non-intrusive. The research showcases my approach to use Augmented Reality as a medium for information, moving beyond the focus on enabling technology. Innovative approach for video capture and for occlusion-based navigation of augmented content has been demonstrated through the process of design and the subsequent development of a series of software prototypes. I have used Research through Design as the methodology for generating new knowledge through five design iterations of a software prototype for the game Settlers of Catan, and evaluated the prototype through peer-review and user testing

Toward a post-digital practice of architectural representation: an animated re-engagement of architecture, visual effects and the moving image

For over a decade there has been a great deal of interest in the creation of architectural forms from digital processes. Though this has undergone extensive experimentation and critique in academia, there has been significantly less attention paid to the way architects digitally represent architecture. Amid the abundance of digitally rendered images and animations of late, which have provided new opportunities for illustrating and disseminating architectural ideas, there are some concerning trends. They include: the narrowing of aesthetic outcomes through the current digital methods, leading to greater homogeneity and limiting the communicative potential of the outcomes; the complex, inappropriate and redundant techniques employed to develop imagery and animations; the privileging of a geometric description over the poetic qualities of architecture; and, perhaps unintentionally yet importantly, the re-characterising of representation as primarily an explicative practice as distinct from the equally reflective, reflexive and contemplative practice it once was. Exploring and addressing these concerns is the interest of this research. This research examines through theoretical writings the current practices of digital representation and their results, and through two projects proposes more appropriate methodologies that would enhance the outcomes. By looking to the practice of visual effects in cinema through notable examples pertinent to architecture, we may begin reviewing current techniques and our ambitions of architectural representation. This is not merely to couple cinematic visual effects with the long-established tradition of architectural drawing as a fashionable counter, but rather to recognise their historic overlap; there was a time before the medium of film when visual effects sat clearly within the domain of architecture and painting. As a Research by Project model of PhD, this exegesis explores and views all its interests, including the historical and theoretical work, through the lens of a practitioner