A Fast Geometric Regularizer to Mitigate Event Collapse in the Contrast Maximization Framework

Event cameras are emerging vision sensors and their advantages are suitable for various applications such as autonomous robots. Contrast maximization (CMax), which provides state-of-the-art accuracy on motion estimation using events, may suffer from an overfitting problem called event collapse. Prior works are computationally expensive or cannot alleviate the overfitting, which undermines the benefits of the CMax framework. We propose a novel, computationally efficient regularizer based on geometric principles to mitigate event collapse. The experiments show that the proposed regularizer achieves state-of-the-art accuracy results, while its reduced computational complexity makes it two to four times faster than previous approaches. To the best of our knowledge, our regularizer is the only effective solution for event collapse without trading off runtime. We hope our work opens the door for future applications that unlocks the advantages of event cameras.Comment: 10 pages, 7 figures, 4 tables. Project page: https://github.com/tub-rip/event collaps

Secrets of Event-Based Optical Flow

Event cameras respond to scene dynamics and offer advantages to estimate motion. Following recent image-based deep-learning achievements, optical flow estimation methods for event cameras have rushed to combine those image-based methods with event data. However, it requires several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. We investigate key elements: how to design the objective function to prevent overfitting, how to warp events to deal better with occlusions, and how to improve convergence with multi-scale raw events. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark. Moreover, our method allows us to expose the issues of the ground truth flow in those benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Our code is available at https://github.com/tub-rip/event_based_optical_flowComment: 23 pages, 11 figures, 7 tables, https://github.com/tub-rip/event_based_optical_flo

Event Collapse in Contrast Maximization Frameworks

Contrast maximization (CMax) is a framework that provides state-of-the-art results on several event-based computer vision tasks, such as ego-motion or optical flow estimation. However, it may suffer from a problem called event collapse, which is an undesired solution where events are warped into too few pixels. As prior works have largely ignored the issue or proposed workarounds, it is imperative to analyze this phenomenon in detail. Our work demonstrates event collapse in its simplest form and proposes collapse metrics by using first principles of space-time deformation based on differential geometry and physics. We experimentally show on publicly available datasets that the proposed metrics mitigate event collapse and do not harm well-posed warps. To the best of our knowledge, regularizers based on the proposed metrics are the only effective solution against event collapse in the experimental settings considered, compared with other methods. We hope that this work inspires further research to tackle more complex warp models.Comment: 19 pages, 8 figures, 3 table

Fast Event-based Optical Flow Estimation by Triplet Matching

Event cameras are novel bio-inspired sensors that offer advantages over traditional cameras (low latency, high dynamic range, low power, etc.). Optical flow estimation methods that work on packets of events trade off speed for accuracy, while event-by-event (incremental) methods have strong assumptions and have not been tested on common benchmarks that quantify progress in the field. Towards applications on resource-constrained devices, it is important to develop optical flow algorithms that are fast, light-weight and accurate. This work leverages insights from neuroscience, and proposes a novel optical flow estimation scheme based on triplet matching. The experiments on publicly available benchmarks demonstrate its capability to handle complex scenes with comparable results as prior packet-based algorithms. In addition, the proposed method achieves the fastest execution time (> 10 kHz) on standard CPUs as it requires only three events in estimation. We hope that our research opens the door to real-time, incremental motion estimation methods and applications in real-world scenarios.Comment: 5 pages, 4 figures, 2 table

Event-based Background-Oriented Schlieren

Schlieren imaging is an optical technique to observe the flow of transparent media, such as air or water, without any particle seeding. However, conventional frame-based techniques require both high spatial and temporal resolution cameras, which impose bright illumination and expensive computation limitations. Event cameras offer potential advantages (high dynamic range, high temporal resolution, and data efficiency) to overcome such limitations due to their bio-inspired sensing principle. This paper presents a novel technique for perceiving air convection using events and frames by providing the first theoretical analysis that connects event data and schlieren. We formulate the problem as a variational optimization one combining the linearized event generation model with a physically-motivated parameterization that estimates the temporal derivative of the air density. The experiments with accurately aligned frame- and event camera data reveal that the proposed method enables event cameras to obtain on par results with existing frame-based optical flow techniques. Moreover, the proposed method works under dark conditions where frame-based schlieren fails, and also enables slow-motion analysis by leveraging the event camera's advantages. Our work pioneers and opens a new stack of event camera applications, as we publish the source code as well as the first schlieren dataset with high-quality frame and event data. https://github.com/tub-rip/event_based_bosComment: Accepted at IEEE T-PAM

Calibration of AGN Reverberation Distance Measurements

In Yoshii et al. (2014), we described a new method for measuring extragalactic distances based on dust reverberation in active galactic nuclei (AGNs), and we validated our new method with Cepheid variable stars. In this paper, we validate our new method with Type Ia supernovae (SNe Ia) which occurred in two of the AGN host galaxies during our AGN monitoring program: SN 2004bd in NGC 3786 and SN 2008ec in NGC 7469. Their multicolor light curves were observed and analyzed using two widely accepted methods for measuring SN distances, and the distance moduli derived are $\mu=33.47\pm 0.15$ for SN 2004bd and $33.83\pm 0.07$ for SN 2008ec. These results are used to obtain independently the distance measurement calibration factor, $g$. The $g$ value obtained from the SN Ia discussed in this paper is $g_{\rm SN} = 10.61\pm 0.50$ which matches, within the range of 1$\sigma$ uncertainty, $g_{\rm DUST} = 10.60$, previously calculated ab initio in Yoshii et al. (2014). Having validated our new method for measuring extragalactic distances, we use our new method to calibrate reverberation distances derived from variations of H$\beta$ emission in the AGN broad line region (BLR), extending the Hubble diagram to $z\approx 0.3$ where distinguishing between cosmologies is becoming possible.Comment: Astrophysical Journal Letters accepte

The Infrared Cloud Monitor for the MAGNUM Robotic Telescope at Haleakala

We present the most successful infrared cloud monitor for a robotic telescope. This system was originally developed for the MAGNUM 2-m telescope, which has been achieving unmanned and automated monitoring observation of active galactic nuclei at Haleakala on the Hawaiian island of Maui since 2001. Using a thermal imager and two aspherical mirrors, it at once sees almost the whole sky at a wavelength of $\lambda\sim 10\mu{\rm m}$. Its outdoor part is weather-proof and is totally maintenance-free. The images obtained every one or two minutes are analysed immediately into several ranks of weather condition, from which our automated observing system not only decides to open or close the dome, but also selects what types of observations should be done. The whole-sky data accumulated over four years show that 50$-$60 % of all nights are photometric, and about 75 % are observable with respect to cloud condition at Haleakala. Many copies of this system are now used all over the world such as Mauna Kea in Hawaii, Atacama in Chile, and Okayama and Kiso in Japan.Comment: 18 pages, 15 figures, 7 tables, accepted for publication in PAS

Development of Cross-Curricular Career Education Making Use of Internship in a Middle School : Building Relevance of Students ‘Subjective Leaning, School life, and Social Life

キャリア教育として中学校の職場体験活動はほぼすべての学校で実施されている。本稿はその事前指導，事後指導を通して，職場体験と学校で学ぶ教科，学校生活，社会生活を生徒にとって意味あるつながりの構築を目指して行った実践の紹介である。具体的には，教職免許を持つ 6 名の大学院生が，それぞれの専門性を生かして教科の専門性と働くことと生活がつながっていることを，中学校 2 年生の職場体験の事後指導Ⅰとしてポスターセッションで伝えた。事後指導 2 時間目では，生徒は働くことと生活と教科のつながりを可視化するマップを作成して意見交換をした。これらの活動の成果は，生徒が記入した授業の振り返りを分類することで確認した。その結果，授業実践者がねらったつながりに気づいたと思われる記述が多く見られた。中でも多かったのは，将来に備えて今後の勉学や努力したいという趣旨の回答であった。As part of career education, work experience activities are conducted in almost all junior high schools in Japan. This paper introduces a practice that helps students build meaningful connections between subjects learned in school, school life, and social life through guidance provided both before and after work experience activities. Specifically, in the first hour of post-activity instruction for the work experience of second-year students, six graduate students with teaching licenses made use of their respective specialties, combining work with subject specialties and students’ lives through poster sessions. In the second hour, graduate student teachers showed them worksheet which made them visualizes the connection between work, life, and subjects. After working with the worksheet according to their internship work, students discussed their case each other. The results of these activities were confirmed by classifying the lessons reviews written by students. Several descriptions contained the connection that the lesson practitioner had aimed for. The most common answers by the participants included their hope they would make study more for their further and for their satisfied life