Removal of visual disruption caused by rain using cycle-consistent generative adversarial networks

This paper addresses the problem of removing rain disruption from images without blurring scene content, thereby retaining the visual quality of the image. This is particularly important in maintaining the performance of outdoor vision systems, which deteriorates with increasing rain disruption or degradation on the visual quality of the image. In this paper, the Cycle-Consistent Generative Adversarial Network (CycleGAN) is proposed as a more promising rain removal algorithm, as compared to the state-of-the-art Image De-raining Conditional Generative Adversarial Network (ID-CGAN). One of the main advantages of the CycleGAN is its ability to learn the underlying relationship between the rain and rain-free domain without the need of paired domain examples, which is essential for rain removal as it is not possible to obtain the rain-free image under dynamic outdoor conditions. Based on the physical properties and the various types of rain phenomena [10], five broad categories of real rain distortions are proposed, which can be applied to the majority of outdoor rain conditions. For a fair comparison, both the ID-CGAN and CycleGAN were trained on the same set of 700 synthesized rain-and-ground-truth image-pairs. Subsequently, both networks were tested on real rain images, which fall broadly under these five categories. A comparison of the performance between the CycleGAN and the ID-CGAN demonstrated that the CycleGAN is superior in removing real rain distortions

Image reconstruction under visual disruption caused by rain

This thesis contributes to single-image reconstruction under visual disruption caused by rain in the following areas: 1. Parameterization of a Convolutional Autoencoder (CAE) for small images [1] 2. Generation of a rain-free image using Cycle-Consistent Generative Adversarial Network (CycleGAN) [2] 3. Rain removal across spatial frequencies using the Multi-Scale CycleGANs (MS-CycleGANs) 4. Rain removal at spatial frequency’s sub-bands using theWavelet-CycleGANs (W-CycleGANs) Image reconstruction or restoration refers to reproducing a clean or disruption-free image from an original image corrupted with some form of noise or unwanted disturbance. The goal of image reconstruction is to remove such disruption from the original corrupted image while preserving the original detail of the image scene. In recent years, deep learning techniques have been proposed for removal of rain disruption, or rain removal. They were devised using the Convolutional Neural Network (CNN) [3], and a more recent type of deep learning network called the Generative Adversarial Network (GAN) [4]. Current state-of the-art deep learning rain removal method, called the Image De-raining Conditional Generative Adversarial Network (ID-CGAN) [5], has been shown to be unable to remove rain disruption completely, or preserving the original scene detail [2]. The focus of this research is to remove rain corruption from images without sacrificing the content of the scene, starting from the collection of real rain images to the testing methodologies developed for our Generative Adversarial Network (GAN) networks. This image rain removal or reconstruction research area has attracted much interest in the past decade as it forms an important aspect of outdoor vision systems where many computer vision algorithms could be affected by rain disruption, especially if only a single image is captured. The first contribution of this thesis in the area of image reconstruction or restoration is the parameterization of a Convolutional Autoencoder (CAE). A framework for deriving an optimum set of CAE parameters for the reconstruction of small input images based on the standard Modified National Institute of Standards and Technology (MNIST) and Street View House Numbers (SVHN) data sets are proposed, using the quantitative mean squared error (MSE) and the qualitative 2Ds’ visualization of the neurons’ activation statistics and entropy at the hidden layers of the CAE. This methodology’s results show that for small 32x32 pixels’ input images, having 2560 neurons at the hidden layer (bottleneck layer) and 32 convolutional feature maps can result in optimum reconstruction performance or good representations of the input image in the latent space for the CAE [1]. The second contribution of this thesis is the generation of a rain-free image using the proposed CycleGAN [2]. Its network model was trained on the same set of 700 rain and rainfree image-pairs used by the recent ID-CGAN work [5]. In the ID-CGAN paper, there was a thorough comparison with other existing techniques like sparse dictionary-based method, convolutional-coding based method, etc. The results using synthetic rain training images have shown that the ID-CGAN method has outperformed all other existing techniques. Hence, our first proposed algorithm, the CycleGAN, is only compared to the ID-CGAN, using the same set of real rain images provided by the authors. The CycleGAN is a practical image’s style transfer approach that falls into the unpaired category, which is capable of transferring an image with rain to an image that is rain-free, without the use of training image-pairs. This is important as natural or real rain images don’t have their corresponding image-pairs that are rain-free. For comparison purpose, a real rain image data set was created. The real rain’s physical properties and phenomena [6] were used to streamline our testing conditions into five broad types of real rain disruption. This testing methodology covers most of the different outdoor rain distortion scenarios captured in the real rain image data set. Hence, we can compare both ID-CGAN and CycleGAN networks using only real rain images. The comparison results using both real and synthetic rain has shown that the CycleGAN method has outperformed the ID-CGAN which represents the state-of-the-art techniques for rain removal [2]. The Natural Image Quality Evaluator (NIQE) is also introduced as a quantitative measure [7] to analyze rain removal results as it can predict the quality of an image without relying on any prior knowledge of the image’s distortions. The results are presented in Chapter 6. Subsequently, from the CycleGAN technique, the third contribution of the thesis is proposed based on the multi-scale representation of the CycleGAN, called the MS-CycleGANs technique. This proposed technique was built on the remaining gaps on rain removal using the CycleGAN. As highlighted in the rain removal paper using CycleGAN [2], the CycleGAN results could be further improved as its reconstructed output was still unable to remove the rain components at low frequency band and preserved as much original details of the scenes as possible. Hence, the MS-CycleGANs was introduced as a better algorithm than the CycleGAN, as it could train multiple CycleGANs to remove rain components at different spatial frequency bands. The implementation of the MS-CycleGANs is discussed after the CycleGAN, and its rain removal results are also compared to the CycleGAN. The results of the MS-CycleGANs framework has shown that the MS-CycleGANs can learn the characteristics between the rain and rain-free domain at different spatial frequency scales, which is essential for removing the individual frequency components of rain while preserving the scene details. In the final contribution towards image reconstruction for removal of visual disruptions caused by rain across spatial frequency’s sub-bands, the W-CycleGANs is proposed and implemented to exploit the properties of wavelet transform such as orthogonality and signal localization, to improve the CycleGAN results. For a fair comparison with the CycleGAN, both the proposed multi-scale representations of CycleGAN networks, namely the MS-CycleGANs and the W-CycleGANs, were trained and tested on the same set of rain images used by the ID-CGAN work [5]. A qualitative visual comparison of rain-removed images, especially at the enlarged rain-removed regions, is performed for the ID-CGAN, CycleGAN, MS-CycleGANs and W-CycleGANs. The comparison results among them has demonstrated the superiority of both the MS-CycleGANs and W-CycleGANs in removing rain distortions

Failure Analysis in Next-Generation Critical Cellular Communication Infrastructures

The advent of communication technologies marks a transformative phase in critical infrastructure construction, where the meticulous analysis of failures becomes paramount in achieving the fundamental objectives of continuity, security, and availability. This survey enriches the discourse on failures, failure analysis, and countermeasures in the context of the next-generation critical communication infrastructures. Through an exhaustive examination of existing literature, we discern and categorize prominent research orientations with focuses on, namely resource depletion, security vulnerabilities, and system availability concerns. We also analyze constructive countermeasures tailored to address identified failure scenarios and their prevention. Furthermore, the survey emphasizes the imperative for standardization in addressing failures related to Artificial Intelligence (AI) within the ambit of the sixth-generation (6G) networks, accounting for the forward-looking perspective for the envisioned intelligence of 6G network architecture. By identifying new challenges and delineating future research directions, this survey can help guide stakeholders toward unexplored territories, fostering innovation and resilience in critical communication infrastructure development and failure prevention

Enforcing Realism and Temporal Consistency for Large-Scale Video Inpainting

Today, people are consuming more videos than ever before. At the same time, video manipulation has rapidly been gaining traction due to the influence of viral videos, as well as the convenience of editing software. Although video manipulation has legitimate entertainment purposes, it can also be incredibly destructive. In order to understand the positive and negative consequences of media manipulation---as well as to maintain the integrity of mass media---it is important to investigate the capabilities of video manipulation techniques. In this dissertation, we focus on the manipulation task of video inpainting, where the goal is to automatically fill in missing parts of a masked video with semantically relevant content. Inpainting results should possess high visual quality with respect to reconstruction performance, realism, and temporal consistency, i.e., they should faithfully recreate missing contents in a way that resembles the real world and exhibits minimal flickering artifacts. Two major challenges have impeded progress toward improving visual quality: semantic ambiguity and diagnostic evaluation. Semantic ambiguity exists for any masked video due to several plausible explanations of the events in the observed scene; however, prior methods have struggled with ambiguity due to their limited temporal contexts. As for diagnostic evaluation, prior work has overemphasized aggregate analysis on large datasets and underemphasized fine-grained analysis on modern inpainting failure modes; as a result, the expected behaviors of models under specific scenarios have remained poorly understood. Our work improves on both models and evaluation techniques for video inpainting, thereby providing deeper insight into how an inpainting model's design impacts the visual quality of its outputs. To advance state-of-the-art in video inpainting, we propose two novel solutions that improve visual quality by expanding the available temporal context. Our first approach, bi-TAI, intelligently integrates information from multiple frames before and after the desired sequence. It produces more realistic results than prior work, which could only consume limited contextual information. Our second approach, HyperCon, suppresses flickering artifacts from frame-wise processing by identifying and propagating consistencies found in high frame-rate space; we successfully apply it to tasks as disparate as video inpainting and style transfer. Aside from methodological improvements, we also propose two novel evaluation tools to diagnose failure modes of modern video inpainting methods. Our first such contribution is the Moving Symbols dataset, which we use to characterize the sensitivity of a state-of-the-art video prediction model to controllable appearance and motion parameters. Our second contribution is the DEVIL benchmark, which provides a dataset and a comprehensive evaluation scheme to quantify how several semantic properties of the input video and mask affect video inpainting quality. Through models that exploit temporal context---as well as evaluation paradigms that reveal fine-grained failure modes of modern inpainting methods at scale---our contributions enforce better visual quality for video inpainting on a larger scale than prior work. We enable the production of more convincing manipulated videos for data processing and social media needs; we also establish replicable fine-grained analysis techniques to cultivate future progress in the field.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/169785/1/szetor_1.pd

Understanding the Humanitarian Consequences and Risks of Nuclear Weapons : New findings from recent scholarship

Publisher PD

Toward Co-Robotic Construction: Visual Site Monitoring & Hazard Detection to Ensure Worker Safety

Construction has remained the least automated and productive as well as the most hazardous industry. Moreover, it has been plagued by a significant lack of diversity in its workforce as well as aging laborers. To address these issues, co-robotic construction has emerged as a new paradigm of construction. The industry is gradually gearing up to embrace robotic solutions, and many construction robots with various degrees of autonomy are under development or in the early stage of deployment. Presenting a different horizon of construction—harmonious co-existence and co-work between workers and robots—co-robotic construction is expected to reform labor-intensive construction into the more productive, safer, and more inclusive industry. However, an in-depth understanding of the robots’ situational intelligence is still lacking, particularly conclusive logic and technologies to ensure workers’ safety nearby autonomous (or semi-) robots, which is fundamental in realizing the co-robotic construction. To fill the gap, this research established a comprehensive robotic hazard detection roadmap and developed core technologies to realize it, leveraging unmanned aerial vehicles, computer vision, and deep learning. In this dissertation, I describe how the developed technologies with a conclusive logic can pro-actively detect the robotics hazards taking various forms and scenarios in an unstructured and dynamic construction environment. The successful implementation of the robotic hazard detection roadmap in co-robotic construction allows for timely interventions such as pro-active robot control and worker feedback, which contributes to reducing robotic accidents. Eventually, this will make human-robot co-existence and collaboration safer, while also helping to build workers’ trust in robot co-workers. Finally, the ensured safety and trust between robots and workers would contribute to promoting construction enterprises to embrace robotic solutions, boosting construction reformation toward innovative co-robotic construction.PHDCivil EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/167981/1/daeho_1.pd

Rapid Mission Assurance Assessment via Sociotechnical Modeling and Simulation

How do organizations rapidly assess command-level effects of cyber attacks? Leaders need a way of assuring themselves that their organization, people, and information technology can continue their missions in a contested cyber environment. To do this, leaders should: 1) require assessments be more than analogical, anecdotal or simplistic snapshots in time; 2) demand the ability to rapidly model their organizations; 3) identify their organization’s structural vulnerabilities; and 4) have the ability to forecast mission assurance scenarios. Using text mining to build agent based dynamic network models of information processing organizations, I examine impacts of contested cyber environments on three common focus areas of information assurance—confidentiality, integrity, and availability. I find that assessing impacts of cyber attacks is a nuanced affair dependent on the nature of the attack, the nature of the organization and its missions, and the nature of the measurements. For well-manned information processing organizations, many attacks are in the nuisance range and that only multipronged or severe attacks cause meaningful failure. I also find that such organizations can design for resiliency and provide guidelines in how to do so

Digital Twins II

Treball desenvolupat en el marc del programa "European Project Semester".Digital Twins have been around since the early 2000s, but it has only been until now that they started to be affordable thanks to the Internet of Things. In the realm of smart cities, a Digital Twin is a virtual model of a city, a replica of the physical world, which are rapidly becoming indispensable tools to visualize the pulse of the city in real time with layered data sources of buildings, urban infrastructure, utilities, businesses, movement of people and vehicles. The advantages of implementing this concept is that it significantly increases the city's stability. Testing in a virtual model helps prevent emergencies, properly allocate resources that reduces costs and the chances of failure in the real world. This project is a continuation of the last year's theoretical study Digital Twins Ⅰ and its aim is to continue the research about Digital City Twins and explore the Big Data from the city sensors of Vilanova i la Geltrú. A group of five international students, led by the company Neapolis, are working on transforming the city into a smart one within the summer semester of the academic year 2020- 2021. In the process, we studied scientific articles, consulted with university professors from different countries (Spain, Belgium, Brazil), contacted IT and Data Security companies to obtain the necessary information. The report provides a study of practical examples using Digital Twins around the world, their impact on the city improvement, comparison of different platforms and software for developing Digital Twins and the reasoned choice of the best option for use in the next part of the project. Furthermore, it describes Information Infrastructure of Digital Cities, Big Data Management, Data Security and the implementation of Digital Twins in Vilanova i la Geltrú. The Big Data received from the city authorities was read and analyzed in the data part with necessary conclusions. This project made a great contribution to the further development of the Digital Twins for Vilanova i la Geltrú and will simplify the practical implementation for our followers of the next EPS project.Incomin