Multigrid Backprojection Super-Resolution and Deep Filter Visualization

We introduce a novel deep-learning architecture for image upscaling by large factors (e.g. 4x, 8x) based on examples of pristine high-resolution images. Our target is to reconstruct high-resolution images from their downscale versions. The proposed system performs a multi-level progressive upscaling, starting from small factors (2x) and updating for higher factors (4x and 8x). The system is recursive as it repeats the same procedure at each level. It is also residual since we use the network to update the outputs of a classic upscaler. The network residuals are improved by Iterative Back-Projections (IBP) computed in the features of a convolutional network. To work in multiple levels we extend the standard back-projection algorithm using a recursion analogous to Multi-Grid algorithms commonly used as solvers of large systems of linear equations. We finally show how the network can be interpreted as a standard upsampling-and-filter upscaler with a space-variant filter that adapts to the geometry. This approach allows us to visualize how the network learns to upscale. Finally, our system reaches state of the art quality for models with relatively few number of parameters.Comment: Spotlight paper in the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19

역연산에 기반한 합성곱신경망의 설명 및 시각화

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2021.8. 권혁진.Interpretability and explainability of machine learning systems have received ever-increasing attention, especially for convolutional neural networks (CNN). Although there are various interpretation techniques for learning algorithms, post-hoc local explanation methods (e.g., the attribution method that visualizes pixel-level contribution of input to its corresponding result) are under great interest because they can deal with the high dimensional parameters and nonlinear operations of CNNs. Therefore, this dissertation presents three new post-hoc local explanation methods to visualize and understand the working mechanisms of CNNs. At first, this dissertation presents a new method called guided nonlinearity (GNL) that improves the performance of attribution by backpropagating only positive gradients through nonlinear operations. GNL is inspired by the mechanism of action potential (AP) generation in the postsynaptic neuron that depends on the sum of excitatory (EPSP) and inhibitory postsynaptic potentials (IPSP). This dissertation assumes that paths consisting of excitatory synapses faithfully reflect the contributions of inputs to the output. Then this assumption is applied to CNNs by allowing only positive gradients backpropagate through nonlinear operations. Experimental results have shown that GNL outperforms existing methods for computing attributions in terms of the deletion metrics and yields fine-grained and human-interpretable attributions. However, the attributions from existing methods, including GNL, lack a common theoretical background and sometimes give contradicting results. To address this problem, this dissertation develops the operation-wise inverse method that computes the inverse of prediction in an operation-wise manner by considering that CNNs can be decomposed with four fundamental operations (convolution, max-pooling, ReLU, and fully-connected). The operation-wise inverse process assumes that the forward-pass of CNN is a sequential propagation of physical quantities that indicate the magnitude of specific image features. The inverses of fundamental operations are formulated as constrained optimization problems that inverse results should generate output features consistent with the forward-pass. Then, the inverse of prediction is computed by sequentially applying inverses of fundamental operations of CNN. Experimental results show that the proposed operation-wise approach can be a reference tool for computing attributions because it can provide equivalent visualization results to several conventional methods, and the attributions from the operation-wise method achieve state-of-the-art performances in terms of deletion score. Although the operation-wise method can provide a reference framework to compute attributions, applying the attribution concept to CNNs with multiple-valued predictions has not yet been addressed because the computation of attribution requires a single scalar value represents the prediction. To address this problem, this dissertation proposes the layer-wise inverse-based approach by decomposing CNNs into a set of layers that process only positive values that can be interpreted as neural activations. Then, the inverses of layers are formulated as constrained optimization problems that identify activations-of-interest in lower-layers. Then, the inverse of prediction is computed by sequentially applying inverses of layers of CNN as in the operation-wise method. Experimental results show that the proposed layer-wise inverse-based method can analyze CNNs for classification and regression in the same framework. Especially for the case of regression, the layer-wise approach showed that conventional CNNs for single image super-resolution overlook a portion of frequency bands that may result in performance degradation.해석 가능한 기계학습 알고리즘들은 최근 많은 관심을 받고 있으며, 이 중 합성곱신경망 (CNN)의 설명 및 시각화는 주요한 연구주제로서 취급되고 있다. 기계학습 알고리즘을 이해하기 위한 다양한 방법 중 특히 주어진 알고리즘의 예측 결과에 대한 입력의 기여도를 시각화하는 귀인 (attribution)과 같은 사후검정 (post-hoc) 국소설명 (local explanation) 방법은 고차원 매개 변수를 가진 비선형 함수에 적용할 수 있어서 CNN의 설명 및 시각화의 주요한 방법으로 사용되고 있다. 이에 따라 본 논문은 CNN의 작동 원리를 시각화하고 이해하는데 사용될 수 있는 세 가지 사후검정 국소설명 방법들을 제시한다. 첫 번째로, 본 논문은 비선형 연산의 양의 기울기 (positive valued gradient)만 역전파 (backpropagation)하여 귀인 성능을 향상시키는 유도된비선형법 (guided nonlinearity method)을 제시한다. 유도된비선형법의 설계는 흥분성 및 억제성 시냅스 후 전위의 합에 의존하는 시냅스 후 뉴런의 활동 전위 생성 메커니즘으로부터 비롯되었다. 본 논문은 흥분성 시냅스로 구성된 경로가 출력에 대한 입력의 기여도를 충실하게 반영하고 있다고 가정하였다. 그 후, 본 논문은 비선형 연산의 양의 기울기만 역전파 되도록 허용함으로써 이 가정을 CNN의 설명 및 시각화에 적용할 수 있도록 구현하였다. 본 논문은 실험을 통해, 제안된 유도된비선형법이 삭제척도 (deletion metric) 측면에서 기존의 방법들보다 향상된 성능을 보이며 해석 가능하고 세밀한 (fine-grained) 귀인을 산출함을 보였다. 그러나 유도된비선형법을 포함한 기존의 귀인 방법들은 서로 다른 이론을 기반으로 설계되었으며, 이로 인하여 서로 모순되는 귀인들을 계산하는 때도 있다. 이 문제를 해결하기 위해 본 논문에서는 CNN이 합성곱 (convolution), 최대풀링 (max-pooling), ReLU, 전연결 (full-connected)의 4가지 기본 연산들의 합성함수로 표현될 수 있다는 점에 기반하여, CNN을 통한 예측의 역상 (inverse image)을 기본 연산들의 역연산을 통해 계산하는 연산별역연산법 (operation-wise inverse-based method)을 제안한다. 연산별역연산법은 CNN의 정방향진행 (forward-pass)을 특정 이미지특징 (image feature)의 크기를 의미하는 물리량의 순차적 전파로 가정한다. 이 가정하에 연산별역연산법은 계산된 역상이 기존의 정방향진행 결과와 모순되지 않도록 설계된 제한된 최적화 문제 (constrained optimization problem)를 통해 기본 연산의 역연산을 계산한다. 본 논문은 실험을 통해 연산별역연산법이 기존의 여러 귀인 방법들보다 삭제척도 측면에서 향상되었으면서도 질적 측면에서 유사한 시각화 결과를 제공하는 것을 보임으로써 연산별역연산법이 귀인계산의 공통 프레임 워크 (reference framework)로 사용될 수 있음을 보였다. 한편, 영상 분류 문제와 같이 단일 예측을 대상으로 한 CNN과는 달리 복수의 예측값을 가지는 CNN에 대하여 귀인계산을 시도한 연구는 현재까지 보고되지 않았다. 이는 기존의 귀인 계산방법들은 CNN에 대하여 단일 스칼라 (scalar) 값을 출력하도록 요구하기 때문이다. 이 문제를 해결하기 위해 본 논문에서는 계층별역연산법 (layer-wise inverse-based method)을 제안한다. 계층별역연산법은 CNN을 인공 뉴런의 활성값 (neural activation)으로 해석할 수 있는 양의 실수들을 입출력으로 하는 계층 (layer)으로 분해하고, 제한된 최적화 문제로 정의되는 각 계층의 역연산을 정방향진행 결과에 순차적으로 적용함으로써 CNN을 통한 예측의 역상을 계산한다. 본 논문은 실험을 통해, 제안된 계층별역연산법이 영상 분류 및 회기를 대상으로 한 CNN들의 설명 및 시각화를 동일한 프레임 워크 (common framework)에서 처리할 수 있음을 확인하였다. 또한, 본 논문은 계층별역연산법을 통해 단일 영상 고해상화 (single image super-resolution)를 대상으로 한 CNN인 VDSR이 입력 영상의 주파수 대역의 일부를 간과하고 있고 이는 VDSR을 통한 고해상화시 특정 주파수 대역에서 영상 품질의 하락을 유발할 수 있음을 보였다.Contents 1 List of Tables 4 List of Figures 5 1 Introduction 7 1.1 Guided Nonlinearity 8 1.2 Inverse-based approach 9 1.2.1 Operation-wise method 10 1.2.2 Layer-wise method 11 1.3 Outline 14 2 RelatedWork 15 2.1 Activation-based approach 15 2.2 Perturbation-based approach 16 2.3 Backpropagation-based approach 17 2.4 Inverse-based approach 18 3 Guided Nonlinearity 19 3.1 Motivation and Overview 19 3.2 Proposed Guided Non-linearity 23 3.2.1 Integrated Gradients 23 3.2.2 Postulations 23 3.2.3 Proposed method 24 3.3 Experimental Results 27 3.3.1 Evaluation Metrics 29 3.3.2 Experiment details 29 3.3.3 Results and Discussions 30 3.4 Summary 30 4 Operation-wise Approach 32 4.1 Motivation and Overview 32 4.2 Proposed Method 35 4.2.1 Problem statement 36 4.2.2 Proposed constraints 36 4.2.3 Mathematical formulation 37 4.3 Implementation details 38 4.3.1 Inverse of ReLU and Max Pooling 38 4.3.2 Inverse of Fully Connected and Convolution Layers 39 4.4 Experimental Settings 40 4.4.1 Qualitative results 40 4.4.2 Quantitative Results 46 4.5 Summary 50 5 Layer-wise Approach 51 5.1 Motivation and Overview 51 5.2 Formulation of the Proposed Inverse Approach 55 5.2.1 Activation range 56 5.2.2 Minimal activation 56 5.2.3 Linear approximation 57 5.2.4 Layer-wise inverse 57 5.3 Details of inverse computation 59 5.3.1 Convolution block (linear part) + ReLU 59 5.3.2 Max-pooling layer 60 5.3.3 Fully connected block (linear part) + ReLU 61 5.3.4 Fully connected block (linear part) + Softmax 61 5.4 Application to the ImageNet classification task 62 5.4.1 Evaluation of output-reconstruction in terms of input-simplicity 62 5.4.2 Deletion and insertion scores 63 5.4.3 Selection of the regularization term weight 64 5.4.4 Comparison to Existing Methods 67 5.4.5 Output-reconstruction versus input-simplicity plot 68 5.4.6 Ablation study of the activation regularization 72 5.5 The inverse of single image super-resolution network 72 5.5.1 Experimental setting 72 5.5.2 Selection of the regularization term weight 74 5.5.3 Evaluation of the proposed inverse process 77 5.5.4 Frequency domain analysis of attribution 77 5.6 Summary 81 6 Conclusions 82 Bibliography 84 Abstract (In Korean) 95박

Intuitively Assessing ML Model Reliability through Example-Based Explanations and Editing Model Inputs

Interpretability methods aim to help users build trust in and understand the capabilities of machine learning models. However, existing approaches often rely on abstract, complex visualizations that poorly map to the task at hand or require non-trivial ML expertise to interpret. Here, we present two visual analytics modules that facilitate an intuitive assessment of model reliability. To help users better characterize and reason about a model's uncertainty, we visualize raw and aggregate information about a given input's nearest neighbors. Using an interactive editor, users can manipulate this input in semantically-meaningful ways, determine the effect on the output, and compare against their prior expectations. We evaluate our interface using an electrocardiogram beat classification case study. Compared to a baseline feature importance interface, we find that 14 physicians are better able to align the model's uncertainty with domain-relevant factors and build intuition about its capabilities and limitations

3D Shape Understanding and Generation

In recent years, Machine Learning techniques have revolutionized solutions to longstanding image-based problems, like image classification, generation, semantic segmentation, object detection and many others. However, if we want to be able to build agents that can successfully interact with the real world, those techniques need to be capable of reasoning about the world as it truly is: a tridimensional space. There are two main challenges while handling 3D information in machine learning models. First, it is not clear what is the best 3D representation. For images, convolutional neural networks (CNNs) operating on raster images yield the best results in virtually all image-based benchmarks. For 3D data, the best combination of model and representation is still an open question. Second, 3D data is not available on the same scale as images – taking pictures is a common procedure in our daily lives, whereas capturing 3D content is an activity usually restricted to specialized professionals. This thesis is focused on addressing both of these issues. Which model and representation should we use for generating and recognizing 3D data? What are efficient ways of learning 3D representations from a few examples? Is it possible to leverage image data to build models capable of reasoning about the world in 3D? Our research findings show that it is possible to build models that efficiently generate 3D shapes as irregularly structured representations. Those models require significantly less memory while generating higher quality shapes than the ones based on voxels and multi-view representations. We start by developing techniques to generate shapes represented as point clouds. This class of models leads to high quality reconstructions and better unsupervised feature learning. However, since point clouds are not amenable to editing and human manipulation, we also present models capable of generating shapes as sets of shape handles -- simpler primitives that summarize complex 3D shapes and were specifically designed for high-level tasks and user interaction. Despite their effectiveness, those approaches require some form of 3D supervision, which is scarce. We present multiple alternatives to this problem. First, we investigate how approximate convex decomposition techniques can be used as self-supervision to improve recognition models when only a limited number of labels are available. Second, we study how neural network architectures induce shape priors that can be used in multiple reconstruction tasks -- using both volumetric and manifold representations. In this regime, reconstruction is performed from a single example -- either a sparse point cloud or multiple silhouettes. Finally, we demonstrate how to train generative models of 3D shapes without using any 3D supervision by combining differentiable rendering techniques and Generative Adversarial Networks

Optimization of the Parameters of the YAP-(S)PETII Scanner for SPECT Acquisition

Abstract Single Photon Emission Computed Tomography (SPECT) could be considered as a milestone in terms of biomedical imaging technique, which visualizes Functional processes in-vivo, based on the emission of gamma rays produced within the body. The most distinctive feature of SPECT from other imaging modalities is that it is based on the tracer principle, discovered by George Charles de Hevesy in the first decade of the twentieth century. As known by everyone, the metabolism of an organism is composed of atoms within a molecule which can be replaced by one of its radioactive isotopes. By using this principle, we are able to follow and detect pathways of the photons which are emitted from the radioactive element inside the metabolism. SPECT produces images by using a gamma camera which consists of two major functional components, the collimator and the radiation detector. The collimator is a thick sheet of a heavy metal like lead, tungsten of gold with densely packed small holes and is put just in front of the photon detector. The radiation detector converts the gamma rays into scintillation light photons. In conventional SPECT, scanners utilize a parallel hole collimator. Defining a small solid angle, each collimator hole is located somewhere along this line and the photons might reach the detector by passing through these holes. Subsequently, we can create projection images of the radioisotope distribution. The quantity of photons which come to the radiation detector through the collimator holes specifies the image quality regarding signal to noise ratio. One of the crucial parts of all SPECT scanners is the collimator design. The main part of this dissertation is to investigate performance characteristics of YAP-(S)PETII scanner collimator and to obtain collimator characteristics curves for optimization purposes. Before starting the collimator performance investigation of YAP-(S)PETII scanner, we first performed simulation of it in SPECT mode with point source Tc-99m to measure collimator and system efficiency by using GATE–the Geant4 Application for Emission Tomography. GATE is an advanced, flexible, precise, opensource Monte Carlo toolkit developed by the international OpenGATE collaboration and dedicated to the numerical simulations in medical imaging. We obtained the results of collimator and system efficiency in terms of collimator length, radius and septa by using GATE_v4. Then, we compared our results with analytical formulation of efficiency and resolution. For those simulation experiments, we found that the difference between the simulated and the analytical results with regard to approximated geometrical collimator efficiency formulation of H. Anger, is within 20%. Then, we wrote a new ASCII sorter algorithm, which reads ASCII output of GATE_v4 and then creates a sinogram and reconstructs it to see the final simulation results. At the beginning, we used the analytical reconstruction method, filtered back projection (FBP), but this method produces severely blurred images. To solve this problem and increase our image quality, we tried different mathematical filters, like ramp, sheep-logan, low-pass cosine filters. After all of those studies mentioned above, we learned that GATE_v4 is not practical to measure collimator efficiency and resolution. On the other hand, the results of GATE_v4 did not show directly septal penetrated photon ratio. Under the light of these findings, we decided to develop a new user-friendly ray tracing program for optimization of low energy general purpose (LEGP) parallelhole collimators. In addition, we tried to evaluate the image quality and quantify the impact of high-energy contamination for I-123 isotope imaging. Due to its promising chemical characteristics, Iodine-123 is increasingly used in SPECT studies. 159 keV photons are used for imaging, however, high-energy photons result in an error in the projection data primarily by penetration of the collimator and scattering inside the crystal with energy close to the photons used for imaging. One of the way to minimize this effect is using a double energy window (DEW) method, because, it decreases noise in main (sensitive) energy window. By using this method, we tried to determine the difference between simulated and experimental projection results and scattered photon ratio (Sk) value of YAP-(S)PETII scanner for I-123 measurements. The main drawback of GATE simulations is that they are CPU-intensive. In this dissertation to handle this problem, we did the feasibility study of the Fully Monte Carlo based implementation of the system matrix derivation of YAP-(S)PETII scanner by using XtreemOS platform. To manage lifecycle of the simulation on the top XtreemOS, we developed a set of scripts. The main purpose of our study is to integrate a distributed platform like XtreemOS to reduce the overall simulation completion time and increase the feasibility of SPECT simulations in a research environment and establish an accurate and fast method for deriving the system matrix of the YAP-(S)PETII scanner by using Monte Carlo simulation approach. We developed also the ML-EM Algorithm to the reconstruct our GATE simulation results and to derive the system matrix directly from GATE output. In addition to the accuracy consideration, we intend to develop a flexible matrix derivation method and GATE output reconstruction tool