Multiplane 3D superresolution optical fluctuation imaging

By switching fluorophores on and off in either a deterministic or a stochastic manner, superresolution microscopy has enabled the imaging of biological structures at resolutions well beyond the diffraction limit. Superresolution optical fluctuation imaging (SOFI) provides an elegant way of overcoming the diffraction limit in all three spatial dimensions by computing higher-order cumulants of image sequences of blinking fluorophores acquired with a conventional widefield microscope. So far, three-dimensional (3D) SOFI has only been demonstrated by sequential imaging of multiple depth positions. Here we introduce a versatile imaging scheme which allows for the simultaneous acquisition of multiple focal planes. Using 3D cross-cumulants, we show that the depth sampling can be increased. Consequently, the simultaneous acquisition of multiple focal planes reduces the acquisition time and hence the photo-bleaching of fluorescent markers. We demonstrate multiplane 3D SOFI by imaging the mitochondria network in fixed C2C12 cells over a total volume of $65\times65\times3.5 \mu\textrm{m}^3$ without depth scanning.Comment: 7 pages, 3 figure

Sensor fusion in smart camera networks for ambient intelligence

This short report introduces the topics of PhD research that was conducted on 2008-2013 and was defended on July 2013. The PhD thesis covers sensor fusion theory, gathers it into a framework with design rules for fusion-friendly design of vision networks, and elaborates on the rules through fusion experiments performed with four distinct applications of Ambient Intelligence

Answering a Questionnaire Using Eyetracking

The beginning of eye tracking research lies far back in the past. Since eye tracking costs decreased over the past years, the usage of an eye tracker for everyday matters, like the interaction with a personal device, becomes more and more attractive. In the present work, the realization of interacting with a computer interface with only the help of an eye tracker is illustrated. The conducted study examines the acceptance and usability of such a system. Therefore, three different interaction methods have been implemented. In a study, the participants had to complete a questionnaire with those interaction methods using a Windows application and a low-cost eye tracking device. All in all, the study results imply that the number of negative aspects about this system outweigh the positive ones. The biggest issue was the restriction of mobility during the usage of the tracking device. In addition, the usage of the system turned out to be rather exhausting for the eyes. Generally speaking, among the three implemented interaction methods, the interaction method that combines gaze with a second input modality (a keyboard) scored best in terms of interaction speed and usefulness considering the completion of a questionnaire

Analytical techniques of pilot scanning behavior and their application

The state of the art of oculometric data analysis techniques and their applications in certain research areas such as pilot workload, information transfer provided by various display formats, crew role in automated systems, and pilot training are documented. These analytical techniques produce the following data: real-time viewing of the pilot's scanning behavior, average dwell times, dwell percentages, instrument transition paths, dwell histograms, and entropy rate measures. These types of data are discussed, and overviews of the experimental setup, data analysis techniques, and software are presented. A glossary of terms frequently used in pilot scanning behavior and a bibliography of reports on related research sponsored by NASA Langley Research Center are also presented

DMD: A Large-Scale Multi-Modal Driver Monitoring Dataset for Attention and Alertness Analysis

Vision is the richest and most cost-effective technology for Driver Monitoring Systems (DMS), especially after the recent success of Deep Learning (DL) methods. The lack of sufficiently large and comprehensive datasets is currently a bottleneck for the progress of DMS development, crucial for the transition of automated driving from SAE Level-2 to SAE Level-3. In this paper, we introduce the Driver Monitoring Dataset (DMD), an extensive dataset which includes real and simulated driving scenarios: distraction, gaze allocation, drowsiness, hands-wheel interaction and context data, in 41 hours of RGB, depth and IR videos from 3 cameras capturing face, body and hands of 37 drivers. A comparison with existing similar datasets is included, which shows the DMD is more extensive, diverse, and multi-purpose. The usage of the DMD is illustrated by extracting a subset of it, the dBehaviourMD dataset, containing 13 distraction activities, prepared to be used in DL training processes. Furthermore, we propose a robust and real-time driver behaviour recognition system targeting a real-world application that can run on cost-efficient CPU-only platforms, based on the dBehaviourMD. Its performance is evaluated with different types of fusion strategies, which all reach enhanced accuracy still providing real-time response.Comment: Accepted to ECCV 2020 workshop - Assistive Computer Vision and Robotic

Neuromorphic Event-Based Generalized Time-Based Stereovision

3D reconstruction from multiple viewpoints is an important problem in machine vision that allows recovering tridimensional structures from multiple two-dimensional views of a given scene. Reconstructions from multiple views are conventionally achieved through a process of pixel luminance-based matching between different views. Unlike conventional machine vision methods that solve matching ambiguities by operating only on spatial constraints and luminance, this paper introduces a fully time-based solution to stereovision using the high temporal resolution of neuromorphic asynchronous event-based cameras. These cameras output dynamic visual information in the form of what is known as “change events” that encode the time, the location and the sign of the luminance changes. A more advanced event-based camera, the Asynchronous Time-based Image Sensor (ATIS), in addition of change events, encodes absolute luminance as time differences. The stereovision problem can then be formulated solely in the time domain as a problem of events coincidences detection problem. This work is improving existing event-based stereovision techniques by adding luminance information that increases the matching reliability. It also introduces a formulation that does not require to build local frames (though it is still possible) from the luminances which can be costly to implement. Finally, this work also introduces a methodology for time based stereovision in the context of binocular and trinocular configurations using time based event matching criterion combining for the first time all together: space, time, luminance, and motion

Optimisation and comparison of dSTORM and DNA-PAINT super-resolution for quantitative cardiac protein imaging.

Fluorescence microscopy techniques, restricted by the diffraction limit of light, have seen a remarkable advancement in recent years. An approach called dSTORM (direct stochastic optical reconstruction microscopy) utilises the photoswitching capabilities of organic fluorophores when in the presence of special mounting media, the solution within which the sample is placed, to detect single molecule fluorescing events over time. The image that can be reconstructed from these events is not diffraction limited, but instead is limited by how well each event can be precisely localised. In Chapter 3 the importance of using a suitable mounting buffer in order to achieve super-resolution dSTORM is discussed in detail. A quantitative method for determining the reactivity of thiol dSTORM switching mountants was developed for use within the lab. Every fluorescent probe has different photophysical properties which can be manipulated by varying the composition of the switching buffer to enhance desirable qualities, such as; increased photon counts, faster switching rates, and longer survivability. In addition to investigating the effects of buffer composition the use of a near UV light-source was also explored as a means of manipulating the same properties to improve overall resolution and quality of the resulting images. A range of photoswitchable fluorescent dyes were tested including Alexa Fluor 660 which is a dye that to my knowledge has not been greatly tested for use in single molecule localisation microscopy by others to date. This dye performed strongly alongside the traditional Alexa Fluor 647 used for dSTORM imaging in optimal conditions. A relatively new approach to single molecule imaging which does not require the fluorophore to photoswitch, called DNA-PAINT (point accumulation for imaging in nanoscale topography), has been investigated throughout this thesis. This approach relies on the transient binding of small oligonucleotide sequences, called “Imagers”, to target docking strands anchored in positions of interest. These imagers have a photostable and bright fluorophore conjugated to the oligonucleotide. It is the transient immobilisation of the imager strand, as it binds to a fixed docking strand, which appears as stochastic blinks. The duration of these events, which can be extended by increasing the number of overlapping base pairs, is primarily responsible for improved localisation precision and therefore potentially overall resolution. At the end of Chapter 3 I compare this new pointillism microscopy approach, DNA-PAINT, with dSTORM using a set of custom-designed oligonucleotide sequences that allow both formats to be employed on the same target. The transient binding of small strands of oligonucleotides offers a far more controllable system for stochastic imaging. In Chapter 4 I use this superior approach to achieve greater resolution than other fluorescence techniques in biological samples, sufficient to visualise single ryanodine receptors (RyR). The RyR are extremely important in the contraction of muscle cells as they are capable of detecting transient changes to calcium concentration and are responsible for releasing large stores of calcium from the sarcoplasmic reticulum. With DNA-PAINT I observed that RyRs cluster into irregular arrays which contain significant gaps that are occupied by other proteins, including junctophilin (JPH). The stoichiometry of JPH with RyR varied cluster to cluster, exposing a new complexity in the regulation of RyRs. In Chapter 5, quantitative super-resolution is reliably achieved through the implementation of quantitative DNA-PAINT (qPAINT) within the Python Microscopy Environment (PYME) software. Quantitative measurements are possible because of the statistical predictability of DNA hybridisation and the near constant influx of fresh imager strands by diffusion. This results in limited photobleaching, a permanent dark state. The frequency with which a region of interest blinks is proportional to the number of binding sites available, and therefore the mean dark time between detected events is also inversely proportional. I validate my approach to qPAINT, which maintains the spatial information of individual structures, by using a DNA-origami test slide. Two distinguishable structures were present and an estimate for the ratio of available docking sites between them was satisfactorily established. I conclude that with this tool, molecule densities can be inferred and information about biological samples can be probed to new levels. The results of the full methodological approach to accomplish dual-colour super-resolution imaging of optically thick cardiac tissue, using both dSTORM and DNA PAINT techniques, is discussed in detail in Chapter 6. The current range of photoswitchable fluorophores limits the possible combination of molecular dyes for use with dSTORM and some compromise is made in their selection. For DNA-PAINT, the prospect of chromatic aberration is removed by imaging the same dye in subsequent rounds of imaging. The process, called Exchange-PAINT, allows the user to remove previously imaged imager strands, through a series of washes, and replace them with a complementary sequence for another target. I introduce the concept of using quencher strands to eliminate signal from unwanted imager sequences, accelerating their removal in samples of reduced diffusion and decreasing the risk of sample disturbance, in a process we termed Quencher Exchange-PAINT. Using this technique, I achieve superior super resolution results in optically thick samples. The results presented in this thesis are expected to (1) lead to a better understanding of the variables associated with single molecule localisation microscopy, (2) further reveal the complexity in cardiac protein distribution, (3) quantify relationships between co-localising proteins and other targets, and (4) apply DNA-PAINT to imaging in optically thick biological samples. This study shows promise for the future applications of the DNA-PAINT pointillism super-resolution method and its ability to investigate a multitude of biological questions