Camino, Ramiro Daniel

English

Open Repository and Bibliography - Luxembourg

GAN Applications with Discrete DataReproducing and Bringing State-of-the-Art to IndustryRamiro CAMINOSEDAN Lab, SnT, University of Luxembourg, Luxembourgramiro.camino@uni.lu — (+352) 46 66 44 - 6855AbstractGaining insight into financial transactions is a legal require-ment for banks and financial service providers. Unfortunately, formost data available to banks and service provider, the truth aboutthe transaction purpose is unknown and illicit activities have a lowbase rate. Additionally, after reporting suspicious activity, bankswill no hearing back from investigations. Consequently, many su-pervised approaches, which inherently rely on the ground truth,cannot be used. In the following we study the use of GenerativeAdversarial Networks (GAN) for data augmentation to increasethe scarce amount of known fraudulent cases, and for data im-putation to improve the quality of samples.Introduction• Issues analyzing financial transactions and accounts:–Class imbalance: e.g. fraud small / non-fraud big–Unavailability and unreliability of labels–Missing values along multiple features– Impossibility to share private data• Possible applications of GANs:–Data augmentation for the minority class–Data imputation for missing values– Synthetic data generation to replace private data–Unsupervised feature extraction• State-of-the-Art for GANs:– Plenty of work and promising results for continuous do-mains like images, video and sound– Studies involving discrete samples are mainly focusedon sequences of words, characters or symbolsRelated WorkFigure 1: Generative Adversarial Network (GAN) [4]. The generatorweights can be trained thanks to the error backpropagated from thediscriminator loss and through the discriminator weights.Figure 2: When sampling from a discrete distribution, the backpropa-gation fails because the sample operation is non-differentiable.Figure 3: MedGAN [2] first pre-trains an autoencoder. During the sec-ond training phase, the decoder transforms the generator outputs intosamples where each feature is considered as a binary variable.Figure 4: ARAE [7] trains and autoencoder and a WGAN [1] with al-ternating steps. The discriminator is replaced by a critic (along withthe according loss), and takes as input codes instead of samples.Figure 5: The Gumbel-Softmax [6] and the Concrete-Distribution [10]were simultaneously proposed to sample from discrete distributions inthe domain of VAEs [8]. Later [9] adapted the technique to GANs forsequences of discrete elements as shown in this diagram.Figure 6: In [5] authors propose to add a gradient penalty to theWGAN loss (hence the name WGAN-GP) to improve training. Addi-tionaly, the authors claim that with WGAN-GP it is possible to generatediscrete sequences training directly with softmax outputs.Multi-Categorical GANFigure 7: Proposed architecture for either generator or decoder mod-els. After the model output, we place in parallel a dense layer percategorical variable, followed by an activation and a concatenationto obtain the final output. The choice between Gumbel-Softmax orSoftmax depends on the model. The reconstruction losses should betransformed into the sum of one cross entropy loss per variable.Experiments• Split dataset in Dtrain and Dtest• Train generative model with Dtrain• Generate Dsample with the trained model• Experiments proposed in [2]:– Probabilities by dimension: compare proportion ofones per dimension between Dtest and Dsample– Predictions by dimension: train one predictive modelper dimension with Dtrain and evaluate with Dtest; re-peat but training Dsample and compare results.• Predictions by categorical: analogous experiment buttraining one predictive model per categorical variable.• Example results shown in Figure 8Conclusions• Compared to unmodified MedGAN and ARAE, all ap-proaches improve the performance across all datasets• Even though we cannot identify a clear best model• The performance improvement comes at the cost of re-quiring to add extra information (the variable sizes). Insome cases, this information may not be available.Future Work• Analyze datasets with mixed variable types• Explore more hyperparameters and architecture options• Experiment with oversampling and undersampling• Improve validation with domain specific metrics0.00.20.40.60.81.0ARAEprobabilities by dimensions predictions by dimensions predictions by categorical0.00.20.40.60.81.0MedGAN0.00.20.40.60.81.0MC­ARAE0.00.20.40.60.81.0MC­MedGAN0.00.20.40.60.81.0MC­Gumbel0.0 0.2 0.4 0.6 0.8 1.0data score0.00.20.40.60.81.0MC­WGAN­GP0.0 0.2 0.4 0.6 0.8 1.0data score0.0 0.2 0.4 0.6 0.8 1.0data scoremodel scoremodel scoremodel scoremodel scoremodel scoremodel scoreFigure 8: Experiments for the US Census 1990 [3]. Each column cor-responds to a metric and each row a to sample from a different model.References[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou.Wasserstein GAN. arXiv preprint arXiv:1701.07875,2017.[2] Edward Choi, Siddharth Biswal, Bradley Malin, JonDuke, Walter F. Stewart, and Jimeng Sun. Generat-ing multi-label discrete patient records using genera-tive adversarial networks. In PMLR, pages 286–305.[3] D. Dua and E. Karra Taniskidou. UCI Machine Learn-ing Repository. 2017.[4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza,Bing Xu, David Warde-Farley, Sherjil Ozair, AaronCourville, and Yoshua Bengio. Generative adversarialnets. In Z. Ghahramani, M. Welling, C. Cortes, N. D.Lawrence, and K. Q. Weinberger, editors, Advancesin Neural Information Processing Systems 27, pages2672–2680. Curran Associates, Inc.[5] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vin-cent Dumoulin, and Aaron Courville. Improved trainingof wasserstein GANs.[6] Eric Jang, Shixiang Gu, and Ben Poole. Categoricalreparameterization with gumbel-softmax.[7] Junbo, Zhao, Yoon Kim, Kelly Zhang, Alexander M.Rush, and Yann LeCun. Adversarially regularized au-toencoders.[8] Diederik P. Kingma and Max Welling. Auto-encodingvariational bayes.[9] Matt J. Kusner and José Miguel Hernández-Lobato.GANS for sequences of discrete elements with thegumbel-softmax distribution.[10] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh.The concrete distribution: A continuous relaxation ofdiscrete random variables.

GAN Applications with Discrete Data

https://orbilu.uni.lu/bitstream/10993/43193/1/GAN%20Applications%20with%20Discrete%20Data.pdf

GAN Applications with Discrete Data

Abstract

Similar works

Full text

Available Versions

Open Repository and Bibliography - Luxembourg