Detection and analysis of single event upsets in noisy digital imagers with small to medium pixels

Camera sensors are shrinking, resulting in more defects seen through image analysis. Due to cosmic radiation, camera experience both permanent defects known as hot pixels and temporal defective spikes which are Single Event Upsets (SEUs). SEUs manifest themselves as temporal random bright areas in sequential dark-frame images that are taken with long exposure times. In the past, it was difficult to separate SEUs from noise in dark-frame images taken with DSLRs at high sensitivity levels (ISO) and cell phone cameras at modest sensitivity levels. However, recent software improvements in this research have enabled the analysis of defect rates in noisy digital imagers – by leveraging local area and pixel address distribution techniques. In addition, multiple experiments were performed to understand the relationship of SEUs and elevation. This study reports data from imagers with pixels ranging from 7 μm (DSLR cameras) down to 1.2 μm (cell phone cameras)

Neutron-Induced, Single-Event Effects on Neuromorphic Event-based Vision Sensor: A First Step Towards Space Applications

This paper studies the suitability of neuromorphic event-based vision cameras for spaceflight, and the effects of neutron radiation on their performance. Neuromorphic event-based vision cameras are novel sensors that implement asynchronous, clockless data acquisition, providing information about the change in illuminance greater than 120dB with sub-millisecond temporal precision. These sensors have huge potential for space applications as they provide an extremely sparse representation of visual dynamics while removing redundant information, thereby conforming to low-resource requirements. An event-based sensor was irradiated under wide-spectrum neutrons at Los Alamos Neutron Science Center and its effects were classified. We found that the sensor had very fast recovery during radiation, showing high correlation of noise event bursts with respect to source macro-pulses. No significant differences were observed between the number of events induced at different angles of incidence but significant differences were found in the spatial structure of noise events at different angles. The results show that event-based cameras are capable of functioning in a space-like, radiative environment with a signal-to-noise ratio of 3.355. They also show that radiation-induced noise does not affect event-level computation. We also introduce the Event-based Radiation-Induced Noise Simulation Environment (Event-RINSE), a simulation environment based on the noise-modelling we conducted and capable of injecting the effects of radiation-induced noise from the collected data to any stream of events in order to ensure that developed code can operate in a radiative environment. To the best of our knowledge, this is the first time such analysis of neutron-induced noise analysis has been performed on a neuromorphic vision sensor, and this study shows the advantage of using such sensors for space applications

The Effect of Elevation on Single Event Upsets in Digital Imagers

Cosmic radiation causes transient errors in microelectronic devices, known as Single Event Upsets (SEUs). These errors are most common in space-borne electronics, however terrestrial electronics experience the same errors, to a lesser degree. SEUs can be difficult to characterize in most integrated circuits, however, in digital imagers they cause defects which appear as unexpected bright pixels that are temporarily present in a series of images. To detect them, a sequence of long exposure, dark-frame images is recorded and then during analysis pixels which appear bright in one photograph but dark in the images immediately before and after are flagged. Just as the effect of SEUs is more prevalent in space, a rise in elevation on Earth can increase the frequency of SEUs by a noticeable amount. In this thesis, I will perform a series of experiments to understand the relationship between SEUs and elevation. Using DSLR cameras, images will be recorded at elevations ranging from sea level to approximately 1200 metres above sea level. The quantity and charge distribution of the SEUs will be extracted from the photographs and compared to the elevation

Assessing the performance of Digital Micromirror Devices for use in space-based multi-object spectrometers

A current need in space-based instrumentation is a reconfigurable slit mask. Several techniques for slit masks have been employed for ground-based astronomical spectrographs. These ground-based instruments have used large discrete components, which are impractical for remote operation in space-based deployment. The Texas Instruments\u27 Digital Micromirror Device (DMD) was originally conceived purely for display purposes, but is a viable candidate to be use as a slit mask in a space-based multi-object spectrograph (MOS). The Integrated Circuit (IC) manufacturing industry has enabled the robust integration of both silicon transistors and Micro-Electrical Mechanical Systems (MEMS) optical components into a very reliable monolithic chip (the DMD). The focus of this work was in three areas that addressed the suitability of proposing DMDs for future space missions. The DMDs were optically characterized to assess their utility in a spectrograph. The DMDs were also cooled in a liquid nitrogen dewar to determine their minimum operating temperature. The low temperature tests indicated that the DMD can operate to temperatures as low as 130 K. In addition, several DMDs were irradiated with high-energy protons at the LBNL 88 Cyclotron to determine how robust the devices are to ionizing radiation (protons). The radiation testing results indicate that DMDs would survive medium to long duration space missions with full operability. Based on preliminary tests in these three areas, the DMD should be considered as an excellent candidate for deployment in future space missions

Single event upset testing of flash based field programmable gate arrays

In the last 50 years microelectronics have advanced at an exponential rate, causing microelectronic devices to shrink, have very low operating voltages and increased complexities; all this has made circuits more sensitive to various kinds of failures. These trends allowed soft errors, which up until recently was just a concern for space application, to become a major source of system failures of electronic products. The aim of this research paper was to investigate different mitigation techniques that prevent these soft errors in a Video Graphics Array (VGA) controller which is commonly used in projecting images captured by cameras. This controller was implemented on a Flash Based Field Programmable Gate array (FPGA). A test set-up was designed and implemented at NRF iThemba LABS, which was used to conduct the experiments necessary to evaluate the effectiveness of different mitigation techniques. The set-up was capable of handling multiple Device Under Tests (DUT) and had the ability to change the angle of incidence of each DUT. The DUTs were radiated with a 66MeV proton beam while the monitoring equipment observed any errors that had occurred. The results obtained indicated that all the implemented mitigation techniques tested on the VGA system improved the system’s capability of mitigating Single Event Upsets (SEU). The most effective mitigation technique was the OR-AND Multiplexer Single Event Transient (SET) filter technique. It was thus shown that mitigation techniques are viable options to prevent SEU in a VGA controller. The permanent SEU testing set-up which was designed and manufactured and was used to conduct the experiments, proved to be a practical option for further microelectronics testing at iThemba LABS

Towards the development of flexible, reliable, reconfigurable, and high-performance imaging systems

Current FPGAs can implement large systems because of the high density of reconfigurable logic resources in a single chip. FPGAs are comprehensive devices that combine flexibility and high performance in the same platform compared to other platform such as General-Purpose Processors (GPPs) and Application Specific Integrated Circuits (ASICs). The flexibility of modern FPGAs is further enhanced by introducing Dynamic Partial Reconfiguration (DPR) feature, which allows for changing the functionality of part of the system while other parts are functioning. FPGAs became an important platform for digital image processing applications because of the aforementioned features. They can fulfil the need of efficient and flexible platforms that execute imaging tasks efficiently as well as the reliably with low power, high performance and high flexibility. The use of FPGAs as accelerators for image processing outperforms most of the current solutions. Current FPGA solutions can to load part of the imaging application that needs high computational power on dedicated reconfigurable hardware accelerators while other parts are working on the traditional solution to increase the system performance. Moreover, the use of the DPR feature enhances the flexibility of image processing further by swapping accelerators in and out at run-time. The use of fault mitigation techniques in FPGAs enables imaging applications to operate in harsh environments following the fact that FPGAs are sensitive to radiation and extreme conditions. The aim of this thesis is to present a platform for efficient implementations of imaging tasks. The research uses FPGAs as the key component of this platform and uses the concept of DPR to increase the performance, flexibility, to reduce the power dissipation and to expand the cycle of possible imaging applications. In this context, it proposes the use of FPGAs to accelerate the Image Processing Pipeline (IPP) stages, the core part of most imaging devices. The thesis has a number of novel concepts. The first novel concept is the use of FPGA hardware environment and DPR feature to increase the parallelism and achieve high flexibility. The concept also increases the performance and reduces the power consumption and area utilisation. Based on this concept, the following implementations are presented in this thesis: An implementation of Adams Hamilton Demosaicing algorithm for camera colour interpolation, which exploits the FPGA parallelism to outperform other equivalents. In addition, an implementation of Automatic White Balance (AWB), another IPP stage that employs DPR feature to prove the mentioned novelty aspects. Another novel concept in this thesis is presented in chapter 6, which uses DPR feature to develop a novel flexible imaging system that requires less logic and can be implemented in small FPGAs. The system can be employed as a template for any imaging application with no limitation. Moreover, discussed in this thesis is a novel reliable version of the imaging system that adopts novel techniques including scrubbing, Built-In Self Test (BIST), and Triple Modular Redundancy (TMR) to detect and correct errors using the Internal Configuration Access Port (ICAP) primitive. These techniques exploit the datapath-based nature of the implemented imaging system to improve the system's overall reliability. The thesis presents a proposal for integrating the imaging system with the Robust Reliable Reconfigurable Real-Time Heterogeneous Operating System (R4THOS) to get the best out of the system. The proposal shows the suitability of the proposed DPR imaging system to be used as part of the core system of autonomous cars because of its unbounded flexibility. These novel works are presented in a number of publications as shown in section 1.3 later in this thesis

Radiation Effects on Semiconductor Devices in High Energy Heavy Ion Accelerators

Radiation effects on semiconductor devices in GSI Helmholtz Center for Heavy Ion Research are becoming more and more significant with the increase of beam intensity due to upgrades. Moreover a new accelerator is being constructed on the basis of GSI within the project of facility for antiproton and ion research (FAIR). Beam intensities will be increased by factor of 100 and energies by factor of 10. Radiation fields in the vicinity of beam lines will increase more than 2 orders of magnitude and so will the effects on semiconductor devices. It is necessary to carry out a study of radiation effects on semiconductor devices considering specific properties of radiation typical for high energy heavy ion accelerators. Radiation effects on electronics in accelerator environment may be divided into two categories: short-term temporary effects and long-term permanent degradation. Both may become critical for proper operation of some electronic devices. This study is focused on radiation damage to CCD cameras in radiation environment of heavy ion accelerator. Series of experiments with irradiation of devices under test (DUTs) by secondary particles produced during ion beam losses were done for this study. Monte Carlo calculations were performed to simulate the experiment conditions and conditions expected in future accelerator. Corresponding comparisons and conclusions were done. Another device typical for accelerator facilities - industrial Ethernet switch was tested in similar conditions during this study. Series of direct irradiations of CCD and MOS transistors with heavy ion beams were done as well. Typical energies of the primary ion beams were 0.5-1 GeV/u. Ion species: from Na to U. Intensities of the beam up to 1e9 ions/spill with spill length of 200-300 ns. Criteria of reliability and lifetime of DUTs in specific radiation conditions were formulated, basing on experimental results of the study. Predictions of electronic device reliability and lifetime were formulated for radiation conditions expected in future at FAIR, basing on Monte Carlo simulations. In addition to main results a new type of CCD-based beam loss monitor (BLM) was proposed and discussed

Earth imaging with microsatellites: An investigation, design, implementation and in-orbit demonstration of electronic imaging systems for earth observation on-board low-cost microsatellites.

This research programme has studied the possibilities and difficulties of using 50 kg microsatellites to perform remote imaging of the Earth. The design constraints of these missions are quite different to those encountered in larger, conventional spacecraft. While the main attractions of microsatellites are low cost and fast response times, they present the following key limitations: Payload mass under 5 kg, Continuous payload power under 5 Watts, peak power up to 15 Watts, Narrow communications bandwidths (9.6 / 38.4 kbps), Attitude control to within 5°, No moving mechanics. The most significant factor is the limited attitude stability. Without sub-degree attitude control, conventional scanning imaging systems cannot preserve scene geometry, and are therefore poorly suited to current microsatellite capabilities. The foremost conclusion of this thesis is that electronic cameras, which capture entire scenes in a single operation, must be used to overcome the effects of the satellite's motion. The potential applications of electronic cameras, including microsatellite remote sensing, have erupted with the recent availability of high sensitivity field-array CCD (charge-coupled device) image sensors. The research programme has established suitable techniques and architectures necessary for CCD sensors, cameras and entire imaging systems to fulfil scientific/commercial remote sensing despite the difficult conditions on microsatellites. The author has refined these theories by designing, building and exploiting in-orbit five generations of electronic cameras. The major objective of meteorological scale imaging was conclusively demonstrated by the Earth imaging camera flown on the UoSAT-5 spacecraft in 1991. Improved cameras have since been carried by the KITSAT-1 (1992) and PoSAT-1 (1993) microsatellites. PoSAT-1 also flies a medium resolution camera (200 metres) which (despite complete success) has highlighted certain limitations of microsatellites for high resolution remote sensing. A reworked, and extensively modularised, design has been developed for the four camera systems deployed on the FASat-Alfa mission (1995). Based on the success of these missions, this thesis presents many recommendations for the design of microsatellite imaging systems. The novelty of this research programme has been the principle of designing practical camera systems to fit on an existing, highly restrictive, satellite platform, rather than conceiving a fictitious small satellite to support a high performance scanning imager. This pragmatic approach has resulted in the first incontestable demonstrations of the feasibility of remote sensing of the Earth from inexpensive microsatellites

Enhancing Real-time Embedded Image Processing Robustness on Reconfigurable Devices for Critical Applications

Nowadays, image processing is increasingly used in several application fields, such as biomedical, aerospace, or automotive. Within these fields, image processing is used to serve both non-critical and critical tasks. As example, in automotive, cameras are becoming key sensors in increasing car safety, driving assistance and driving comfort. They have been employed for infotainment (non-critical), as well as for some driver assistance tasks (critical), such as Forward Collision Avoidance, Intelligent Speed Control, or Pedestrian Detection. The complexity of these algorithms brings a challenge in real-time image processing systems, requiring high computing capacity, usually not available in processors for embedded systems. Hardware acceleration is therefore crucial, and devices such as Field Programmable Gate Arrays (FPGAs) best fit the growing demand of computational capabilities. These devices can assist embedded processors by significantly speeding-up computationally intensive software algorithms. Moreover, critical applications introduce strict requirements not only from the real-time constraints, but also from the device reliability and algorithm robustness points of view. Technology scaling is highlighting reliability problems related to aging phenomena, and to the increasing sensitivity of digital devices to external radiation events that can cause transient or even permanent faults. These faults can lead to wrong information processed or, in the worst case, to a dangerous system failure. In this context, the reconfigurable nature of FPGA devices can be exploited to increase the system reliability and robustness by leveraging Dynamic Partial Reconfiguration features. The research work presented in this thesis focuses on the development of techniques for implementing efficient and robust real-time embedded image processing hardware accelerators and systems for mission-critical applications. Three main challenges have been faced and will be discussed, along with proposed solutions, throughout the thesis: (i) achieving real-time performances, (ii) enhancing algorithm robustness, and (iii) increasing overall system's dependability. In order to ensure real-time performances, efficient FPGA-based hardware accelerators implementing selected image processing algorithms have been developed. Functionalities offered by the target technology, and algorithm's characteristics have been constantly taken into account while designing such accelerators, in order to efficiently tailor algorithm's operations to available hardware resources. On the other hand, the key idea for increasing image processing algorithms' robustness is to introduce self-adaptivity features at algorithm level, in order to maintain constant, or improve, the quality of results for a wide range of input conditions, that are not always fully predictable at design-time (e.g., noise level variations). This has been accomplished by measuring at run-time some characteristics of the input images, and then tuning the algorithm parameters based on such estimations. Dynamic reconfiguration features of modern reconfigurable FPGA have been extensively exploited in order to integrate run-time adaptivity into the designed hardware accelerators. Tools and methodologies have been also developed in order to increase the overall system dependability during reconfiguration processes, thus providing safe run-time adaptation mechanisms. In addition, taking into account the target technology and the environments in which the developed hardware accelerators and systems may be employed, dependability issues have been analyzed, leading to the development of a platform for quickly assessing the reliability and characterizing the behavior of hardware accelerators implemented on reconfigurable FPGAs when they are affected by such faults

The Copernicus project

The Copernicus spacecraft, to be launched on May 4, 2009, is designed for scientific exploration of the planet Pluto. The main objectives of this exploration is to accurately determine the mass, density, and composition of the two bodies in the Pluto-Charon system. A further goal of the exploration is to obtain precise images of the system. The spacecraft will be designed for three axis stability control. It will use the latest technological advances to optimize the performance, reliability, and cost of the spacecraft. Due to the long duration of the mission, nominally 12.6 years, the spacecraft will be powered by a long lasting radioactive power source. Although this type of power may have some environmental drawbacks, currently it is the only available source that is suitable for this mission. The planned trajectory provides flybys of Jupiter and Saturn. These flybys provide an opportunity for scientific study of these planets in addition to Pluto. The information obtained on these flybys will supplement the data obtained by the Voyager and Galileo missions. The topics covered include: (1) scientific instrumentation; (2) mission management, planning, and costing; (3) power and propulsion system; (4) structural subsystem; (5) command, control, and communication; and (6) attitude and articulation control