Fusing Frame and Event Vision for High-speed Optical Flow for Edge Application

Optical flow computation with frame-based cameras provides high accuracy but the speed is limited either by the model size of the algorithm or by the frame rate of the camera. This makes it inadequate for high-speed applications. Event cameras provide continuous asynchronous event streams overcoming the frame-rate limitation. However, the algorithms for processing the data either borrow frame like setup limiting the speed or suffer from lower accuracy. We fuse the complementary accuracy and speed advantages of the frame and event-based pipelines to provide high-speed optical flow while maintaining a low error rate. Our bio-mimetic network is validated with the MVSEC dataset showing 19% error degradation at 4x speed up. We then demonstrate the system with a high-speed drone flight scenario where a high-speed event camera computes the flow even before the optical camera sees the drone making it suited for applications like tracking and segmentation. This work shows the fundamental trade-offs in frame-based processing may be overcome by fusing data from other modalities

Discursos em torno dos vídeos verticais: a arqueologia de uma proporção "errada" de tela

This paper seeks to provide historical references for the examination of contemporary forms of vertical moving images, often considered “wrong” due to their incompatibility with the audiovisual standards established in the West. Deploying an archaeological approach, the paper identifies expressions of verticality in moving images since their first modern developments, encompassing both the birth of cinema and the emergence of video art circuits in the 1980s-90s. These cases serve to underscore the disputed mediality of audiovisual systems. This paper concludes by showing how the negotiation of medium specificities continues through networked platforms and curated events, creating possibilities for the emergence of new technological art forms. Esse artigo busca estabelecer referências históricas para a análise de imagens em movimento verticais, frequentemente consideradas “erradas” devido a sua incompatibilidade com os standards audiovisuais ocidentais modernos. A partir de uma abordagem de arqueologia de mídias, o artigo identifica expressões de verticalidade nas imagens em movimento desde seus primeiros desenvolvimentos, considerando o nascimento do cinema e a emergência dos circuitos de videoarte nos anos 1980-90. Esses casos ressaltam a midialidade disputada dos sistemas audiovisuais. O artigo conclui demonstrando como a negociação de especificidades midiáticas se estende por meio de plataformas em rede e da curadoria de eventos, criando a possibilidade para a emergência de novas formas de arte tecnológica.&nbsp

LookOut! Interactive Camera Gimbal Controller for Filming Long Takes

The job of a camera operator is more challenging, and potentially dangerous, when filming long moving camera shots. Broadly, the operator must keep the actors in-frame while safely navigating around obstacles, and while fulfilling an artistic vision. We propose a unified hardware and software system that distributes some of the camera operator's burden, freeing them up to focus on safety and aesthetics during a take. Our real-time system provides a solo operator with end-to-end control, so they can balance on-set responsiveness to action vs planned storyboards and framing, while looking where they're going. By default, we film without a field monitor. Our LookOut system is built around a lightweight commodity camera gimbal mechanism, with heavy modifications to the controller, which would normally just provide active stabilization. Our control algorithm reacts to speech commands, video, and a pre-made script. Specifically, our automatic monitoring of the live video feed saves the operator from distractions. In pre-production, an artist uses our GUI to design a sequence of high-level camera "behaviors." Those can be specific, based on a storyboard, or looser objectives, such as "frame both actors." Then during filming, a machine-readable script, exported from the GUI, ties together with the sensor readings to drive the gimbal. To validate our algorithm, we compared tracking strategies, interfaces, and hardware protocols, and collected impressions from a) film-makers who used all aspects of our system, and b) film-makers who watched footage filmed using LookOut.Comment: V2: - Fixed typos. - Cleaner supplemental. - New plot in control section with same data from a supplemental vide

Mug for Tremors

Melissa Oliver, an occupational therapist at McGuire VA Medical Center, works extensively with veterans who struggle with certain daily activities due to their hand tremors. Hand tremors caused by essential tremor, Parkinson’s Disease, or other physical or neurological impairments can greatly affect an individual’s ability to perform daily tasks. One of these tasks include the ability to handle beverages, as hand tremors can cause excessive sloshing and spilling of liquid. Melissa reached out to find a way to help veterans with hand tremors easily consume liquids. Our team, consisting of Logan Smith, Allen Tecson, Jaret Wedow, and Kai Workman, was tasked with creating a stabilizing mug capable of resisting motion due to hand tremors. After designing, prototyping, and iterating, we constructed a final prototype. Our final design features custom 3D printed components integrated with electronics to make the mug actively detect and counteract hand tremors. Two Brushless DC Motors allow the cup to rotate along two axes of rotation after receiving signals from the sensors and controller board located in the bottom of the mug. A removable mug liner allows for easy cleaning and ensures the electronic components stay dry

The Development of an Antagonistic SMA Actuation Technology for the Active Cancellation of Human Tremor.

Human Tremor is an unintentional bodily motion that affects muscle control among both healthy individuals and those with movement disorders, occasionally to severe detriment. While assistive devices avoid the risk of side effects from pharmacological or surgical treatments, most devices are impractical for daily use due to limitations inherent in conventional actuators. The goal of this research is to address these limitations by developing an antagonistic Shape Memory Alloy (SMA) actuation technology, enabling a new class of active tremor cancellation devices. This is accomplished through the construction of a model and body of empirical support that provides the necessary design insight and predictive power for an antagonistic actuator that ensures stable amplitude and high frequency motion with low power draw. Actuation frequency and power draw were improved while balancing their competing effects through the development of: 1) a method that accurately measures the convective coefficient of SMA to enhance actuator design, 2) a growth process for carbon nanotube cooling fins to enhance cooling in a fixed medium, and 3) an understanding of the antagonistic architecture to produce increased frequency in a controllable manner. To enable applications requiring predictability for positioning and complex control, a thermodynamic model for antagonistic SMA was derived to account for inertial, slack, boiling, friction, and convective effects. Using the model, a series of simulation studies provided design insight on the effect of operating environment, driving signal, and environmental conditions so that the generic actuation system can be utilized in a wide variety of applications beyond tremor cancellation. If high forces are required in such applications, stability issues can arise, which were addressed in experimental shakedown research that broadens the high-stress SMA design space. The technology enabled by this dissertation was demonstrated in a working Active Cancellation of Tremor (ACT) prototype that produced 71% RMS cancellation of human tremor. The cancellation results show significant improvement over the current state of the art by providing intuitive, lightweight, compact hand-held tremor cancellation that is a promising solution to numerous assistive applications in medical, military, and manufacturing sectors.Ph.D.Mechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/76010/1/apathak_1.pd

Multi-Sensor Methods for Mobile Radar Motion Capture and Compensation.

Ph.D. Thesis. University of Hawaiʻi at Mānoa 2017

Instrumentation and validation of a robotic cane for transportation and fall prevention in patients with affected mobility

Dissertação de mestrado integrado em Engenharia Física, (especialização em Dispositivos, Microssistemas e Nanotecnologias)O ato de andar é conhecido por ser a forma primitiva de locomoção do ser humano, sendo que este traz muitos benefícios que motivam um estilo de vida saudável e ativo. No entanto, há condições de saúde que dificultam a realização da marcha, o que por consequência pode resultar num agravamento da saúde, e adicionalmente, levar a um maior risco de quedas. Nesse sentido, o desenvolvimento de um sistema de deteção e prevenção de quedas, integrado num dispositivo auxiliar de marcha, seria essencial para reduzir estes eventos de quedas e melhorar a qualidade de vida das pessoas. Para ultrapassar estas necessidades e limitações, esta dissertação tem como objetivo validar e instrumentar uma bengala robótica, denominada Anti-fall Robotic Cane (ARCane), concebida para incorporar um sistema de deteção de quedas e um mecanismo de atuação que possibilite a prevenção de quedas, ao mesmo tempo que assiste a marcha. Para esse fim, foi realizada uma revisão do estado da arte em bengalas robóticas para adquirir um conhecimento amplo e aprofundado dos componentes, mecanismos e estratégias utilizadas, bem como os protocolos experimentais, principais resultados, limitações e desafios em dispositivos existentes. Numa primeira fase, foi estipulado o objetivo de: (i) adaptar a missão do produto; (ii) estudar as necessidades do consumidor; e (iii) atualizar as especificações alvo da ARCane, continuação do trabalho de equipa, para obter um produto com design e engenharia compatível com o mercado. Foi depois estabelecida a arquitetura de hardware e discutidos os componentes a ser instrumentados na ARCane. Em seguida foram realizados testes de interoperabilidade a fim de validar o funcionamento singular e coletivo dos componentes. Relativamente ao controlo de movimento, foi desenvolvido um sistema inovador, de baixo custo e intuitivo, capaz de detetar a intenção do movimento e de reconhecer as fases da marcha do utilizador. Esta implementação foi validada com seis voluntários saudáveis que realizaram testes de marcha com a ARCane para testar sua operabilidade num ambiente de contexto real. Obteve-se uma precisão de 97% e de 90% em relação à deteção da intenção de movimento e ao reconhecimento da fase da marcha do utilizador. Por fim, foi projetado um método de deteção de quedas e mecanismo de prevenção de quedas para futura implementação na ARCane. Foi ainda proposta uma melhoria do método de deteção de quedas, de modo a superar as limitações associadas, bem como a proposta de dispositivos de deteção a serem implementados na ARCane para obter um sistema completo de deteção de quedas.The act of walking is known to be the primitive form of the human being, and it brings many benefits that motivate a healthy and active lifestyle. However, there are health conditions that make walking difficult, which, consequently, can result in worse health and, in addition, lead to a greater risk of falls. Thus, the development of a fall detection and prevention system integrated with a walking aid would be essential to reduce these fall events and improve people quality of life. To overcome these needs and limitations, this dissertation aims to validate and instrument a cane-type robot, called Anti-fall Robotic Cane (ARCane), designed to incorporate a fall detection system and an actuation mechanism that allow the prevention of falls, while assisting the gait. Therefore, a State-of-the-Art review concerning robotic canes was carried out to acquire a broad and in-depth knowledge of the used components, mechanisms and strategies, as well as the experimental protocols, main results, limitations and challenges on existing devices. On a first stage, it was set an objective to (i) enhance the product's mission statement; (ii) study the consumer needs; and (iii) update the target specifications of the ARCane, extending teamwork, to obtain a product with a market-compatible design and engineering that meets the needs and desires of the ARCane users. It was then established the hardware architecture of the ARCane and discussed the electronic components that will instrument the control, sensory, actuator and power units, being afterwards subjected to interoperability tests to validate the singular and collective functioning of cane components altogether. Regarding the motion control of robotic canes, an innovative, cost-effective and intuitive motion control system was developed, providing user movement intention recognition, and identification of the user's gait phases. This implementation was validated with six healthy volunteers who carried out gait trials with the ARCane, in order to test its operability in a real context environment. An accuracy of 97% was achieved for user motion intention recognition and 90% for user gait phase recognition, using the proposed motion control system. Finally, it was idealized a fall detection method and fall prevention mechanism for a future implementation in the ARCane, based on methods applied to robotic canes in the literature. It was also proposed an improvement of the fall detection method in order to overcome its associated limitations, as well as detection devices to be implemented into the ARCane to achieve a complete fall detection system

Spherical tangible user interfaces in mixed reality

The popularity of virtual reality (VR) and augmented reality (AR) has grown rapidly in recent years, both in academia and commercial applications. This is rooted in technological advances and affordable head-mounted displays (HMDs). Whether in games or professional applications, HMDs allow for immersive audio-visual experiences that transport users to compelling digital worlds or convincingly augment the real world. However, as true to life as these experiences have become in a visual and auditory sense, the question remains how we can model interaction with these virtual environments in an equally natural way. Solutions providing intuitive tangible interaction would bear the potential to fundamentally make the mixed reality (MR) spectrum more accessible, especially for novice users. Research on tangible user interfaces (TUIs) has pursued this goal by coupling virtual to real-world objects. Tangible interaction has been shown to provide significant advantages for numerous use cases. Spherical tangible user interfaces (STUIs) present a special case of these devices, mainly due to their ability to fully embody any spherical virtual content. In general, spherical devices increasingly transition from mere technology demonstrators to usable multi-modal interfaces. For this dissertation, we explore the application of STUIs in MR environments primarily by comparing them to state-of-the-art input techniques in four different contexts. Thus, investigating the questions of embodiment, overall user performance, and the ability of STUIs relying on their shape alone to support complex interaction techniques. First, we examine how spherical devices can embody immersive visualizations. In an initial study, we test the practicality of a tracked sphere embodying three kinds of visualizations. We examine simulated multi-touch interaction on a spherical surface and compare two different sphere sizes to VR controllers. Results confirmed our prototype's viability and indicate improved pattern recognition and advantages for the smaller sphere. Second, to further substantiate VR as a prototyping technology, we demonstrate how a large tangible spherical display can be simulated in VR. We show how VR can fundamentally extend the capabilities of real spherical displays by adding physical rotation to a simulated multi-touch surface. After a first study evaluating the general viability of simulating such a display in VR, our second study revealed the superiority of a rotating spherical display. Third, we present a concept for a spherical input device for tangible AR (TAR). We show how such a device can provide basic object manipulation capabilities utilizing two different modes and compare it to controller techniques with increasing hardware complexity. Our results show that our button-less sphere-based technique is only outperformed by a mode-less controller variant that uses physical buttons and a touchpad. Fourth, to study the intrinsic problem of VR locomotion, we explore two opposing approaches: a continuous and a discrete technique. For the first, we demonstrate a spherical locomotion device supporting two different locomotion paradigms that propel a user's first-person avatar accordingly. We found that a position control paradigm applied to a sphere performed mostly superior in comparison to button-supported controller interaction. For discrete locomotion, we evaluate the concept of a spherical world in miniature (SWIM) used for avatar teleportation in a large virtual environment. Results showed that users subjectively preferred the sphere-based technique over regular controllers and on average, achieved lower task times and higher accuracy. To conclude the thesis, we discuss our findings, insights, and subsequent contribution to our central research questions to derive recommendations for designing techniques based on spherical input devices and an outlook on the future development of spherical devices in the mixed reality spectrum.Die Popularität von Virtual Reality (VR) und Augmented Reality (AR) hat in den letzten Jahren rasant zugenommen, sowohl im akademischen Bereich als auch bei kommerziellen Anwendungen. Dies ist in erster Linie auf technologische Fortschritte und erschwingliche Head-Mounted Displays (HMDs) zurückzuführen. Ob in Spielen oder professionellen Anwendungen, HMDs ermöglichen immersive audiovisuelle Erfahrungen, die uns in fesselnde digitale Welten versetzen oder die reale Welt überzeugend erweitern. Doch so lebensecht diese Erfahrungen in visueller und auditiver Hinsicht geworden sind, so bleibt doch die Frage, wie die Interaktion mit diesen virtuellen Umgebungen auf ebenso natürliche Weise gestaltet werden kann. Lösungen, die eine intuitive, greifbare Interaktion ermöglichen, hätten das Potenzial, das Spektrum der Mixed Reality (MR) fundamental zugänglicher zu machen, insbesondere für Unerfahrene. Die Forschung an Tangible User Interfaces (TUIs) hat dieses Ziel durch das Koppeln virtueller und realer Objekte verfolgt und so hat sich gezeigt, dass greifbare Interaktion für zahlreiche Anwendungsfälle signifikante Vorteile bietet. Spherical Tangible User Interfaces (STUIs) stellen einen Spezialfall von greifbaren Interfaces dar, insbesondere aufgrund ihrer Fähigkeit, beliebige sphärische virtuelle Inhalte vollständig verkörpern zu können. Generell entwickeln sich sphärische Geräte zunehmend von reinen Technologiedemonstratoren zu nutzbaren multimodalen Instrumenten, die auf eine breite Palette von Interaktionstechniken zurückgreifen können. Diese Dissertation untersucht primär die Anwendung von STUIs in MR-Umgebungen durch einen Vergleich mit State-of-the-Art-Eingabetechniken in vier verschiedenen Kontexten. Dies ermöglicht die Erforschung der Bedeutung der Verkörperung virtueller Objekte, der Benutzerleistung im Allgemeinen und der Fähigkeit von STUIs, die sich lediglich auf ihre Form verlassen, komplexe Interaktionstechniken zu unterstützen. Zunächst erforschen wir, wie sphärische Geräte immersive Visualisierungen verkörpern können. Eine erste Studie ergründet die Praxistauglichkeit einer einfach konstruierten, getrackten Kugel, die drei Arten von Visualisierungen verkörpert. Wir testen simulierte Multi-Touch-Interaktion auf einer sphärischen Oberfläche und vergleichen zwei Kugelgrößen mit VR-Controllern. Die Ergebnisse bestätigten die Praxistauglichkeit des Prototyps und deuten auf verbesserte Mustererkennung sowie Vorteile für die kleinere Kugel hin. Zweitens, um die Validität von VR als Prototyping-Technologie zu bekräftigen, demonstrieren wir, wie ein großes, anfassbares sphärisches Display in VR simuliert werden kann. Es zeigt sich, wie VR die Möglichkeiten realer sphärischer Displays substantiell erweitern kann, indem eine simulierte Multi-Touch-Oberfläche um die Fähigkeit der physischen Rotation ergänzt wird. Nach einer ersten Studie, die die generelle Machbarkeit der Simulation eines solchen Displays in VR evaluiert, zeigte eine zweite Studie die Überlegenheit des drehbaren sphärischen Displays. Drittens präsentiert diese Arbeit ein Konzept für ein sphärisches Eingabegerät für Tangible AR (TAR). Wir zeigen, wie ein solches Werkzeug grundlegende Fähigkeiten zur Objektmanipulation unter Verwendung von zwei verschiedenen Modi bereitstellen kann und vergleichen es mit Eingabetechniken deren Hardwarekomplexität zunehmend steigt. Unsere Ergebnisse zeigen, dass die kugelbasierte Technik, die ohne Knöpfe auskommt, nur von einer Controller-Variante übertroffen wird, die physische Knöpfe und ein Touchpad verwendet und somit nicht auf unterschiedliche Modi angewiesen ist. Viertens, um das intrinsische Problem der Fortbewegung in VR zu erforschen, untersuchen wir zwei gegensätzliche Ansätze: eine kontinuierliche und eine diskrete Technik. Für die erste präsentieren wir ein sphärisches Eingabegerät zur Fortbewegung, das zwei verschiedene Paradigmen unterstützt, die einen First-Person-Avatar entsprechend bewegen. Es zeigte sich, dass das Paradigma der direkten Positionssteuerung, angewandt auf einen Kugel-Controller, im Vergleich zu regulärer Controller-Interaktion, die zusätzlich auf physische Knöpfe zurückgreifen kann, meist besser abschneidet. Im Bereich der diskreten Fortbewegung evaluieren wir das Konzept einer kugelförmingen Miniaturwelt (Spherical World in Miniature, SWIM), die für die Avatar-Teleportation in einer großen virtuellen Umgebung verwendet werden kann. Die Ergebnisse zeigten eine subjektive Bevorzugung der kugelbasierten Technik im Vergleich zu regulären Controllern und im Durchschnitt eine schnellere Lösung der Aufgaben sowie eine höhere Genauigkeit. Zum Abschluss der Arbeit diskutieren wir unsere Ergebnisse, Erkenntnisse und die daraus resultierenden Beiträge zu unseren zentralen Forschungsfragen, um daraus Empfehlungen für die Gestaltung von Techniken auf Basis kugelförmiger Eingabegeräte und einen Ausblick auf die mögliche zukünftige Entwicklung sphärischer Eingabegräte im Mixed-Reality-Bereich abzuleiten