Genetic rule extraction optimizing brier score

Most highly accurate predictive modeling techniques produceopaque models. When comprehensible models are required,rule extraction is sometimes used to generate a transparentmodel, based on the opaque. Naturally, the extractedmodel should be as similar as possible to the opaque. Thiscriterion, called fidelity, is therefore a key part of the optimizationfunction in most rule extracting algorithms. Tothe best of our knowledge, all existing rule extraction algorithmstargeting fidelity use 0/1 fidelity, i.e., maximize thenumber of identical classifications. In this paper, we suggestand evaluate a rule extraction algorithm utilizing a moreinformed fidelity criterion. More specifically, the novel algorithm,which is based on genetic programming, minimizesthe difference in probability estimates between the extractedand the opaque models, by using the generalized Brier scoreas fitness function. Experimental results from 26 UCI datasets show that the suggested algorithm obtained considerablyhigher accuracy and significantly better AUC than boththe exact same rule extraction algorithm maximizing 0/1 fidelity,and the standard tree inducer J48. Somewhat surprisingly,rule extraction using the more informed fidelity metricnormally resulted in less complex models, making sure thatthe improved predictive performance was not achieved onthe expense of comprehensibility