Logistic regression model training based on the approximate homomorphic encryption

Background: Security concerns have been raised since big data became a prominent tool in data analysis. For instance, many machine learning algorithms aim to generate prediction models using training data which contain sensitive information about individuals. Cryptography community is considering secure computation as a solution for privacy protection. In particular, practical requirements have triggered research on the efficiency of cryptographic primitives. Methods: This paper presents a method to train a logistic regression model without information leakage. We apply the homomorphic encryption scheme of Cheon et al. (ASIACRYPT 2017) for an efficient arithmetic over real numbers, and devise a new encoding method to reduce storage of encrypted database. In addition, we adapt Nesterov's accelerated gradient method to reduce the number of iterations as well as the computational cost while maintaining the quality of an output classifier. Results: Our method shows a state-of-the-art performance of homomorphic encryption system in a real-world application. The submission based on this work was selected as the best solution of Track 3 at iDASH privacy and security competition 2017. For example, it took about six minutes to obtain a logistic regression model given the dataset consisting of 1579 samples, each of which has 18 features with a binary outcome variable. Conclusions: We present a practical solution for outsourcing analysis tools such as logistic regression analysis while preserving the data confidentiality

Encrypted statistical machine learning: new privacy preserving methods

We present two new statistical machine learning methods designed to learn on fully homomorphic encrypted (FHE) data. The introduction of FHE schemes following Gentry (2009) opens up the prospect of privacy preserving statistical machine learning analysis and modelling of encrypted data without compromising security constraints. We propose tailored algorithms for applying extremely random forests, involving a new cryptographic stochastic fraction estimator, and na\"{i}ve Bayes, involving a semi-parametric model for the class decision boundary, and show how they can be used to learn and predict from encrypted data. We demonstrate that these techniques perform competitively on a variety of classification data sets and provide detailed information about the computational practicalities of these and other FHE methods.Comment: 39 page

동형암호 재부팅 기법에 관한 연구

학위논문 (박사)-- 서울대학교 대학원 : 자연과학대학 수리과학부, 2019. 2. 천정희.2009년 Gentry에 의해서 완전동형암호가 처음 설계된 이후로 최적화와 고속화를 위해서 다양한 기법들과 스킴들이 설계되어 왔다. 하지만 동형암호의 연산횟수를 무제한으로 늘리기 위해서 필수적인 재부팅 기법의 효율성 문제로 실제 응용에 적용하기에는 부적합하다는 평가를 많이 받아왔다. 본 논문에서는 재부팅 기법의 고속화를 위한 다양한 기법을 제시하고 이를 실제로 응용분야에 적용하였다. 본 논문에서는 대표적인 동형암호 스킴들에 대한 재부팅 기법에 대한 연구를 수행하였는데, 첫 번째로는 Microsoft Research와 IMB에서 만든 동형암호 라이브러리인 SEAL과 HElib에 적용가능한 재부팅 기법에 대한 연구를 수행하였다. 해당 재부팅 기법에서 핵심적이 과정은 암호화된 상태에서 복호화 함수를 계산하는 부분이다. 암호된 상태에서 최하위 비트를 추출하는 새로운 방법을 제시하여 재부팅 과정에서 소모되는 계산량과 표현되는 다항식의 차수를 줄이는데에 성공하였다. 두 번째로는, 비교적 최근에 개발된 근사계산 동형암호인 HEAAN 스킴의 재부팅 기법을 개선하는 연구를 수행하였다. 2018년에 삼각함수를 이용한 근사법을 통해서 처음 해당 스킴에 대한 재부팅 기법이 제시되었는데, 많은 데이터를 담고있는 암호문에 대해서는 전처리, 후처리 과정이 계산량의 대부분을 차지하는 문제가 있었다. 해당 과정들을 여러 단계로 재귀적인 함수들로 표현하여 계산량이 데이터 사이즈에 대해서 로그적으로 줄이는 것에 성공하였다. 추가로, 다른 스킴들에 비해서 많이 사용되지는 않지만, 정수기반 동형암호들에 대해서도 재부팅 기법을 개선하는 연구를 수행하였고 그 결과 계산량을 로그적으로 줄이는 것에 성공하였다. 마지막으로, 재부팅 기법의 활용성과 사용 가능성을 보이기 위해 실제 데이터 보안을 필요로 하는 기계학습 분야에 적용해보았다. 실제로 400,000건의 금융 데이터를 이용한 회귀분석을 암호화된 데이터를 이용해서 수행하였다. 그 결과 약 16시간 안에 80\% 이상의 정확도와 0.8 정도의 AUROC 값을 가지는 유의미한 분석 모델을 얻을 수 있었다.After Gentry's blueprint on homomorphic encryption (HE) scheme, various efficient schemes have been suggested. For unlimited number of operations between encrypted data, the bootstrapping process is necessary. There are only few works on bootstrapping procedure because of the complexity and inefficiency of bootstrapping. In this paper, we propose various method and techniques for improved bootstrapping algorithm, and we apply it to logistic regression on large scale encrypted data. The bootstrapping process depends on based homomorphic encryption scheme. For various schemes such as BGV, BFV, HEAAN, and integer-based scheme, we improve bootstrapping algorithm. First, we improved bootstrapping for BGV (HElib) and FV (SEAL) schemes which is implemented by Microsoft Research and IMB respectively. The key process for bootstrapping in those two scheme is extracting lower digits of plaintext in encrypted state. We suggest new polynomial that removes lowest digit of input, and we apply it to bootstrapping with previous method. As a result, both the complexity and the consumed depth are reduced. Second, bootstrapping for multiple data needs homomorphic linear transformation. The complexity of this part is O(n) for number of slot n, and this part becomes a bottleneck when we use large n. We use the structure of linear transformation which is used in bootstrapping, and we decompose the matrix which is corresponding to the transformation. By applying recursive strategy, we reduce the complexity to O(log n). Furthermore, we suggest new bootstrapping method for integer-based HE schemes which are based on approximate greatest common divisor problem. By using digit extraction instead of previous bit-wise approach, the complexity of bootstrapping algorithm reduced from O(poly(lambda)) to O(log^2(lambda)). Our implementation for this process shows 6 seconds which was about 3 minutes. To show that bootstrapping can be used for practical application, we implement logistic regression on encrypted data with large scale. Our target data has 400,000 samples, and each sample has 200 features. Because of the size of the data, direct application of homomorphic encryption scheme is almost impossible. Therefore, we decide the method for encryption to maximize the effect of multi-threading and SIMD operations in HE scheme. As a result, our homomorphic logistic regression takes about 16 hours for the target data. The output model has 0.8 AUROC with about 80% accuracy. Another experiment on MNIST dataset shows correctness of our implementation and method.Abstract 1 Introduction 1.1 Homomorphic Encryption 1.2 Machine Learning on Encrypted Data 1.3 List of Papers 2 Background 2.1 Notation 2.2 Homomorphic Encryption 2.3 Ring Learning with Errors 2.4 Approximate GCD 3 Lower Digit Removal and Improved Bootstrapping 3.1 Basis of BGV and BFV scheme 3.2 Improved Digit Extraction Algorithm 3.3 Bootstrapping for BGV and BFV Scheme 3.3.1 Our modications 3.4 Slim Bootstrapping Algorithm 3.5 Implementation Result 4 Faster Homomorphic DFT and Improved Bootstrapping 4.1 Basis of HEAAN scheme 4.2 Homomorphic DFT 4.2.1 Previous Approach 4.2.2 Our method 4.2.3 Hybrid method 4.2.4 Implementation Result 4.3 Improved Bootstrapping for HEAAN 4.3.1 Linear Transformation in Bootstrapping 4.3.2 Improved CoeToSlot and SlotToCoe 4.3.3 Implementation Result 5 Faster Bootstrapping for FHE over the integers 5.1 Basis of FHE over the integers 5.2 Decryption Function via Digit Extraction 5.2.1 Squashed Decryption Function 5.2.2 Digit extraction Technique 5.2.3 Homomorphic Digit Extraction in FHE over the integers 5.3 Bootstrapping for FHE over the integers 5.3.1 CLT scheme with M Z_t 5.3.2 Homomorphic Operations with M Z_t^a 5.3.3 Homomorphic Digit Extraction for CLT scheme 5.3.4 Our Method on the CLT scheme 5.3.5 Analysis of Proposed Bootstrapping Method 5.4 Implementation Result 6 Logistic Regression on Large Encrypted Data 6.1 Basis of Logistic Regression 6.2 Logistic Regression on Encrypted Data 6.2.1 HE-friendly Logistic Regression Algorithm 6.2.2 HE-Optimized Logistic Regression Algorithm 6.2.3 Further Optimization 6.3 Evaluation 6.3.1 Logistic Regression on Encrypted Financial Dataset 6.3.2 Logistic Regression on Encrypted MNIST Dataset 6.3.3 Discussion 7 Conclusions Abstract (in Korean)Docto

Homomorphic Encryption for Machine Learning in Medicine and Bioinformatics

Machine learning techniques are an excellent tool for the medical community to analyzing large amounts of medical and genomic data. On the other hand, ethical concerns and privacy regulations prevent the free sharing of this data. Encryption methods such as fully homomorphic encryption (FHE) provide a method evaluate over encrypted data. Using FHE, machine learning models such as deep learning, decision trees, and naive Bayes have been implemented for private prediction using medical data. FHE has also been shown to enable secure genomic algorithms, such as paternity testing, and secure application of genome-wide association studies. This survey provides an overview of fully homomorphic encryption and its applications in medicine and bioinformatics. The high-level concepts behind FHE and its history are introduced. Details on current open-source implementations are provided, as is the state of FHE for privacy-preserving techniques in machine learning and bioinformatics and future growth opportunities for FHE

Privacy-Preserving CNN Training with Transfer Learning

Privacy-preserving nerual network inference has been well studied while homomorphic CNN training still remains an open challenging task. In this paper, we present a practical solution to implement privacy-preserving CNN training based on mere Homomorphic Encryption (HE) technique. To our best knowledge, this is the first attempt successfully to crack this nut and no work ever before has achieved this goal. Several techniques combine to make it done: (1) with transfer learning, privacy-preserving CNN training can be reduced to homomorphic neural network training, or even multiclass logistic regression (MLR) training; (2) via a faster gradient variant called $\texttt{Quadratic Gradient}$, an enhanced gradient method for MLR with a state-of-the-art performance in converge speed is applied in this work to achieve high performance; (3) we employ the thought of transformation in mathematics to transform approximating Softmax function in encryption domain to the well-studied approximation of Sigmoid function. A new type of loss function is alongside been developed to complement this change; and (4) we use a simple but flexible matrix-encoding method named $\texttt{Volley Revolver}$ to manage the data flow in the ciphertexts, which is the key factor to complete the whole homomorphic CNN training. The complete, runnable C++ code to implement our work can be found at: https://github.com/petitioner/HE.CNNtraining. We select $\texttt{REGNET\_X\_400MF}$ as our pre-train model for using transfer learning. We use the first 128 MNIST training images as training data and the whole MNIST testing dataset as the testing data. The client only needs to upload 6 ciphertexts to the cloud and it takes $\sim 21$ mins to perform 2 iterations on a cloud with 64 vCPUs, resulting in a precision of $21.49\%$.Comment: In this work, we initiated to implement privacy-persevering CNN training based on mere HE techniques by presenting a faster HE-friendly algorith