Learning-Based Dequantization For Image Restoration Against Extremely Poor Illumination

All existing image enhancement methods, such as HDR tone mapping, cannot recover A/D quantization losses due to insufficient or excessive lighting, (underflow and overflow problems). The loss of image details due to A/D quantization is complete and it cannot be recovered by traditional image processing methods, but the modern data-driven machine learning approach offers a much needed cure to the problem. In this work we propose a novel approach to restore and enhance images acquired in low and uneven lighting. First, the ill illumination is algorithmically compensated by emulating the effects of artificial supplementary lighting. Then a DCNN trained using only synthetic data recovers the missing detail caused by quantization

Contrast enhancement and exposure correction using a structure-aware distribution fitting

Realce de contraste e correção de exposição são úteis em aplicações domésticas e técnicas, no segundo caso como uma etapa de pré-processamento para outras técnicas ou para ajudar a observação humana. Frequentemente, uma transformação localmente adaptativa é mais adequada para a tarefa do que uma transformação global. Por exemplo, objetos e regiões podem ter níveis de iluminação muito diferentes, fenômenos físicos podem comprometer o contraste em algumas regiões mas não em outras, ou pode ser desejável ter alta visibilidade de detalhes em todas as partes da imagem. Para esses casos, métodos de realce de imagem locais são preferíveis. Embora existam muitos métodos de realce de contraste e correção de exposição disponíveis na literatura, não há uma solução definitiva que forneça um resultado satisfatório em todas as situações, e novos métodos surgem a cada ano. Em especial, os métodos tradicionais baseados em equalização adaptativa de histograma sofrem dos efeitos checkerboard e staircase e de excesso de realce. Esta dissertação propõe um método para realce de contraste e correção de exposição em imagens chamado Structure-Aware Distribution Stretching (SADS). O método ajusta regionalmente à imagem um modelo paramétrico de distribuição de probabilidade, respeitando a estrutura da imagem e as bordas entre as regiões. Isso é feito usando versões regionais das expressões clássicas de estimativa dos parâmetros da distribuição, que são obtidas substituindo a mé- dia amostral presente nas expressões originais por um filtro de suavização que preserva as bordas. Após ajustar a distribuição, a função de distribuição acumulada (CDF) do modelo ajustado e a inversa da CDF da distribuição desejada são aplicadas. Uma heurística ciente de estrutura que detecta regiões suaves é proposta e usada para atenuar as transformações em regiões planas. SADS foi comparado a outros métodos da literatura usando métricas objetivas de avaliação de qualidade de imagem (IQA) sem referência e com referência completa nas tarefas de realce de contraste e correção de exposição simultâneos e na tarefa de defogging/dehazing. Os experimentos indicam um desempenho geral superior do SADS em relação aos métodos comparados para os conjuntos de imagens usados, de acordo com as métricas IQA adotadas.Contrast enhancement and exposure correction are useful in domestic and technical applications, the latter as a preprocessing step for other techniques or for aiding human observation. Often, a locally adaptive transformation is more suitable for the task than a global transformation. For example, objects and regions may have very different levels of illumination, physical phenomena may compromise the contrast at some regions but not at others, or it may be desired to have high visibility of details in all parts of the image. For such cases, local image enhancement methods are preferable. Although there are many contrast enhancement and exposure correction methods available in the literature, there is no definitive solution that provides a satisfactory result in all situations, and new methods emerge each year. In special, traditional adaptive histogram equalization-based methods suffer from checkerboard and staircase effects and from over enhancement. This dissertation proposes a method for contrast enhancement and exposure correction in images named Structure-Aware Distribution Stretching (SADS). The method fits a parametric model of probability distribution to the image regionally while respecting the image structure and edges between regions. This is done using regional versions of the classical expressions for estimating the parameters of the distribution, which are obtained by replacing the sample mean present in the original expressions by an edge-preserving smoothing filter. After fitting the distribution, the cumulative distribution function (CDF) of the adjusted model and the inverse of the CDF of the desired distribution are applied. A structure-aware heuristic to indicate smooth regions is proposed and used to attenuate the transformations in flat regions. SADS was compared with other methods from the literature using objective no-reference and full-reference image quality assessment (IQA) metrics in the tasks of simultaneous contrast enhancement and exposure correction and in the task of defogging/dehazing. The experiments indicate a superior overall performance of SADS with respect to the compared methods for the image sets used, according to the IQA metrics adopted

Novel haptic interface For viewing 3D images

In recent years there has been an explosion of devices and systems capable of displaying stereoscopic 3D images. While these systems provide an improved experience over traditional bidimensional displays they often fall short on user immersion. Usually these systems only improve depth perception by relying on the stereopsis phenomenon. We propose a system that improves the user experience and immersion by having a position dependent rendering of the scene and the ability to touch the scene. This system uses depth maps to represent the geometry of the scene. Depth maps can be easily obtained on the rendering process or can be derived from the binocular-stereo images by calculating their horizontal disparity. This geometry is then used as an input to be rendered in a 3D display, do the haptic rendering calculations and have a position depending render of the scene. The author presents two main contributions. First, since the haptic devices have a finite work space and limited resolution, we used what we call detail mapping algorithms. These algorithms compress geometry information contained in a depth map, by reducing the contrast among pixels, in such a way that it can be rendered into a limited resolution display medium without losing any detail. Second, the unique combination of a depth camera as a motion capturing system, a 3D display and haptic device to enhance user experience. While developing this system we put special attention on the cost and availability of the hardware. We decided to use only off-the-shelf, mass consumer oriented hardware so our experiments can be easily implemented and replicated. As an additional benefit the total cost of the hardware did not exceed the one thousand dollars mark making it affordable for many individuals and institutions

Efficient data structures for piecewise-smooth video processing

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 95-102).A number of useful image and video processing techniques, ranging from low level operations such as denoising and detail enhancement to higher level methods such as object manipulation and special effects, rely on piecewise-smooth functions computed from the input data. In this thesis, we present two computationally efficient data structures for representing piecewise-smooth visual information and demonstrate how they can dramatically simplify and accelerate a variety of video processing algorithms. We start by introducing the bilateral grid, an image representation that explicitly accounts for intensity edges. By interpreting brightness values as Euclidean coordinates, the bilateral grid enables simple expressions for edge-aware filters. Smooth functions defined on the bilateral grid are piecewise-smooth in image space. Within this framework, we derive efficient reinterpretations of a number of edge-aware filters commonly used in computational photography as operations on the bilateral grid, including the bilateral filter, edgeaware scattered data interpolation, and local histogram equalization. We also show how these techniques can be easily parallelized onto modern graphics hardware for real-time processing of high definition video. The second data structure we introduce is the video mesh, designed as a flexible central data structure for general-purpose video editing. It represents objects in a video sequence as 2.5D "paper cutouts" and allows interactive editing of moving objects and modeling of depth, which enables 3D effects and post-exposure camera control. In our representation, we assume that motion and depth are piecewise-smooth, and encode them sparsely as a set of points tracked over time. The video mesh is a triangulation over this point set and per-pixel information is obtained by interpolation. To handle occlusions and detailed object boundaries, we rely on the user to rotoscope the scene at a sparse set of frames using spline curves. We introduce an algorithm to robustly and automatically cut the mesh into local layers with proper occlusion topology, and propagate the splines to the remaining frames. Object boundaries are refined with per-pixel alpha mattes. At its core, the video mesh is a collection of texture-mapped triangles, which we can edit and render interactively using graphics hardware. We demonstrate the effectiveness of our representation with special effects such as 3D viewpoint changes, object insertion, depthof- field manipulation, and 2D to 3D video conversion.by Jiawen Chen.Ph.D

Deep Learning for Decision Making and Autonomous Complex Systems

Deep learning consists of various machine learning algorithms that aim to learn multiple levels of abstraction from data in a hierarchical manner. It is a tool to construct models using the data that mimics a real world process without an exceedingly tedious modelling of the actual process. We show that deep learning is a viable solution to decision making in mechanical engineering problems and complex physical systems. In this work, we demonstrated the application of this data-driven method in the design of microfluidic devices to serve as a map between the user-defined cross-sectional shape of the flow and the corresponding arrangement of micropillars in the flow channel that contributed to the flow deformation. We also present how deep learning can be used in the early detection of combustion instability for prognostics and health monitoring of a combustion engine, such that appropriate measures can be taken to prevent detrimental effects as a result of unstable combustion. One of the applications in complex systems concerns robotic path planning via the systematic learning of policies and associated rewards. In this context, a deep architecture is implemented to infer the expected value of information gained by performing an action based on the states of the environment. We also applied deep learning-based methods to enhance natural low-light images in the context of a surveillance framework and autonomous robots. Further, we looked at how machine learning methods can be used to perform root-cause analysis in cyber-physical systems subjected to a wide variety of operation anomalies. In all studies, the proposed frameworks have been shown to demonstrate promising feasibility and provided credible results for large-scale implementation in the industry

Bio-inspired Collision Detection with Motion Cues Enhancement in Dim Light Environments

Detecting looming objects robustly and timely is a huge challenge for artificial vision systems in complex natural scenes, including dim light scenes. Insects have evolved remarkable capacities in collision detection despite their tiny eyes and brains. The locusts’ LGMD1 neuron shows strong looming-sensitive property for both light and dark objects, which is a source of inspiration for developing collision detection systems. Furthermore, specialized visual processing strategies in nocturnal animals’ brains can provide inspiration for detecting faint motion like dim-light collision detection when challenged with low light conditions. This research aims to explore theLGMD1 based collision detection methods, adaptive low-light image enhancement methods, biologically-inspired solutions for enhancing faint motion cues as well as collision detection methods in low light conditions. The major contributions are summarized as follows. A new visual neural system model (LGMD1) is developed, which applies a neural competition mechanism within a framework of separated ON and OFF pathways to shut off the translating response. The competition-based approach responds vigorously to monotonous ON/OFF responses resulting from a looming object. However, it does not respond to paired ON-OFF responses that result from a translating object, thereby enhancing collision selectivity. Moreover, a complementary denoising mechanism ensures reliable collision detection. To verify the effectiveness of the model, we have conducted systematic comparative experiments on synthetic and real datasets. The results show that our method exhibits more accurate discrimination between looming and translational events—the looming motion can be correctly detected. It also demonstrates that the proposed model is more robust than comparative models. A framework is proposed for adaptively enhancing low-light images, which implements the processing of dark adaptation with proper adaptation parameters in R,G and B channels separately. Specifically, the dark adaptation processing consists of a series of canonical neural computations, including the power law adaptation, divisive normalization and adaptive rescaling operations. Experimental results show that the proposed bioinspired dark adaptation framework is more efficient and can better preserve the naturalness of the image compared with several representative low light image enhancement methods. A dim-light motion cues enhancement (DLMCE) model is designed for extracting extremely faint motion cues. This model integrates dark-adaptation, spatio-temporal constraint and neural summation mechanisms, which are achieved with canonical neural computations and neural summation in temporal and spatial domains, to enhance faint motion cues. With the DLMCE model, the image intensity and contrast are first increased by the dark adaptation processing, then the strong motion cues are extracted by the spatio-temporal constraint strategy, and these motion cues are further enhanced by neural summation mechanisms. Experimental results have demonstrated that the presented DLMCE model outperforms the existing methods for dim-light motion cues enhancement, and faint motion cues can be successfully detected in consecutive frames efficiently. As demonstrated in the experiments, the proposed DLMCE model provides a robust and effective solution for autonomous systems in detecting moving objects under low light conditions. A bio-inspired collision detection model is developed for detecting looming objects in dim light environments. The model combines the DLMCE model with the classical four-layered LGMD1 model to detect dimly illuminated approaching objects. To verify the effectiveness of the model, we have conducted comparative experiments on real looming datasets. The results have demonstrated that the proposed bio-inspired collision detection model can correctly recognize looming objects under low light conditions since the DLMCE model enhances the faint looming cues

Novel methods of object recognition and fault detection applied to non-destructive testing of rail’s surface during production

A series of rail image inspection algorithms have been developed for Tata Steels Scunthorpe rail production line. The following thesis describes the contributions made by the author in the design and application of these algorithms. A fully automated rail inspection system that has never been implemented before in any such company or setup has been developed. An industrial computer vision system (JLI) already exists for the image acquisition of rails during production at a rail manufacturing plant in Scunthorpe. An automated inspection system using the same JLI vision system has been developed for the detection of rail‟s surface defects during manufacturing process. This is to complement the human factor by developing a fully automated image processing based system to recognize the faults with an improved efficiency and to allow an exhaustive detection on the entire rail in production. A set of bespoke algorithms has been developed from a plethora of available image processing techniques to extract and identify components in an image of rail in order to detect abnormalities. This has been achieved through offline processing of the rail images using the blended use of different object recognition and image processing techniques, in particular, variation of standard image processing techniques. Several edge detection methods as well as adapted well known Artificial Neural Network and Principal Component Analysis techniques for fault detection on rail have been developed. A combination of customised existing image algorithms and newly developed algorithms have been put together to perform the efficient defect detection. The developed system is fast, reliable and efficient for detection of unique artefacts occurring on the rail surface during production followed by fault classification on the rail imaging system. Extensive testing shows that the defect detection techniques developed for automated rail inspection is capable of detecting more than 90% of the defects present in the available data set of rail images, which has more than 100,000 images under investigation. This demonstrates the efficiency and accuracy of the algorithms developed in this work