Liga, Davide

Palmirani, Monica

Open Repository and Bibliography - Luxembourg

EasyChair Preprint№ 9170Transfer Learning for Deontic RuleClassification: the Case Study of the GDPRDavide Liga and Monica PalmiraniEasyChair preprints are intended for rapiddissemination of research results and areintegrated with the rest of EasyChair.October 26, 2022April 2022Transfer Learning for Deontic RuleClassification:the Case Study of the GDPRDavide LIGA a,b,1, Monica PALMIRANI ba University of Luxembourgb Alma Human-AI, University of BolognaORCiD ID: Davide Liga https://orcid.org/0000-0003-1124-0299, Monica Palmiranihttps://orcid.org/0000-0002-8557-8084Abstract. This work focuses on the automatic classification of deontic sentences. Itpresents a novel Machine Learning approach which combines the power of TransferLearning with the information provided by two famous LegalXML formats. In par-ticular, different BERT-like neural architectures have been fine-tuned on the down-stream task of classifying rules from the European General Data Protection Regu-lation (GDPR) encoded in Akoma Ntoso and LegalRuleML. This work shows thatfine-tuned language models can leverage the information provided in LegalXMLdocuments to achieve automatic classification of deontic sentences and rules.Keywords. Rule classification, Norms, Legal Knowledge Representation, AI&LawThe ability to automatically detect deontic rules directly from natural languagesentences is a crucial long-term goal in the field of Artificial Intelligence and Law(AI&Law), and in legal argumentation [1,2]. One of the obstacles of this kind of tasks isthe lack of available data designed ad hoc for the classification of deontic rules. Since theannotation of this kind of datasets is time-consuming and requires experts of domain, theprocess of creating datasets to automatically recognize deontic rules can be costly. An-other obstacle, related to the first one, is that datasets might be too small to train MachineLearning classifiers, especially when dealing with deep neural architectures.To tackle these issues, a groundbreaking methodology has recently been employedin AI (and NLP), namely Transfer Learning, an approach where huge pre-trained neuralarchitectures are employed in downstream tasks. In this regard, BERT is one of the mostfamous examples, used in many downstream tasks even with very small datasets [3,4].On the one side, this work shows the potential of using LegalXML documents assource of data. On the other side, it exploits the ability of Transfer Learning to have goodperformances on downstream tasks even when dealing with small datasets. Furthermore,this work tackles the automatic classification of deontic rules directly from natural lan-guage, an AI&Law task which has been approached by the community only marginally.This task consists in classifying single legal sentences or single legal provisions as con-taining deontic modalities such as Obligations, Prohibitions and Permissions.1Corresponding Author: Davide Liga, davide.liga@uni.lu.October 20221. MethodologyWe consider two methodological aspects: the data extraction method (how we retrievedour data) and the classification method (what Machine Learning approach we used). Toextract labelled data for the classification of rules and deontic modalities, we combinedAkoma Ntoso and LegalRuleML as suggested in [5]. While LegalRuleML is an optimalrepresentation of the legal logical sphere, Akoma Ntoso is an optimal representation ofthe structure of legal document, including their natural language. This can facilitate thereconstruction of the atomic legal provisions from natural language sentences, especiallyin those cases where the deontic information is split in different structural portions withinthe legal source. In this work, these two formats have been used to create a dataset, whereatomic legal provisions are taken (and sometimes reconstructed) from Akoma Ntoso,while the logical/deontic classes are extracted from LegalRuleML.Regarding the classification methodology, we employed Transfer Learning, whichconsists in the downstream use of language models (i.e., neural architectures that havebeen pre-trained on a huge amount of data). Importantly, there are two major ways of per-forming Transfer Learning: a famous approach is to use the pre-trained language modelto extract embeddings to represent our data (as described in [4]). Another approach isthat of fine-tuning the pre-trained neural architecture on a downstream task. In this work,we used this second approach.2. Related WorksThe first studies which tackled the classification of deontic elements focused on the de-ontic elements as parts of a wider range of targets [6,7,8] or were strongly based on sym-bolic approaches of Artificial Intelligence [9,10]. Perhaps, the first studies which mainlyfocused on the deontic sphere are [11,12], which are also among the first which employedsub-symbolic methods such as Bi-LSTM and self-attention in the field of AI&Law.Since the publication of BERT [3], a growing number of studies employed TransferLearning methods. To the best of our knowledge, the first study which employed BERTfor the classification of deontic sentences is [13]. While [12] focused on just prohibitionsand obligations, [13] also focused on permissions, achieving an average precision andrecall of 90% and 89.66% respectively. Another recent work is [14], which used fourpre-trained architectures (BERT, DistilBERT, RoBERTa, and ALBERT) but focused juston the binary detection duties vs non-duties.Also our work presents a Transfer Learning approach based on BERT (and othersimilar models), which leverage the symbolic information of LegalXML formats (seealso [5]) exploiting the sub-symbolic power provided by different pre-trained languagemodels. The novelty and the power of Transfer Learning methodologies jointly with thecombined use of Akoma Ntoso and LegalRuleML are two major contributions of ourstudy, along with the design of the experimental settings in 4 different classification sce-narios: (1) Rule vs Non-rule, (2) Deontic vs Non-deontic, (3) Obligation vs Permission vsNone, (4) Obligation vs Permission vs Constitutive Rule vs None. Another point whichis worth mentioning is that LegalXML formats such as Akoma Ntoso and LegalRuleMLare documents which are written by legal experts, providing the machine learning algo-rithm with high quality data.October 20223. DataThe data used in this study consists of 707 atomic normative provisions2 extractedfrom the European General Data Protection Regulation (GDPR). To extrapolate thisdataset, we used the DAta Protection REgulation COmpliance (DAPRECO) KnowledgeBase[15], which is the LegalRuleML representation of the GDPR and the the biggestknowledge base in LegalRuleML [16], as well as the biggest knowledge base formalizedin Input/Output Logic [17]. The current version of the DAPRECO3 includes 966 formulæin reified Input/Output logic: 271 obligations, 76 permissions, and 619 constitutive rules.As explained in [15], the number of constitutive rules is much higher than permissionsand obligations because constitutive rules are needed to trigger special inferences for themodelled rules. This means that constitutive rules are an indicator of the existence of arule, without giving information about deontic modalities.Importantly, DAPRECO also contains the connections between each formula andthe corresponding structural element (paragraphs, point, etc) in the Akoma Ntoso rep-resentation of the GDPR4. In other words, using a LegalRuleML knowledge base likeDAPRECO and the corresponding Akoma Ntoso representation, it is possible to connectthe logical-deontic sphere of legal documents (in this case the 966 Input/Output formulæprovided by DAPRECO) to the natural language statements in the legal text (providedby the Akoma Ntoso representation of the GDPR).Importantly, this combination of Akoma Ntoso and LegalRuleML facilitate also thereconstruction of the exact target in terms of natural language. For example, many obli-gations of legal texts are split into lists, and Akoma Ntoso is useful to reconstruct thosepieces of natural language into a unique sentence.4. Experiment settings and resultsAt the end of the process of extraction, we achieved a total of 707 labelled provisions,which have been reconstructed whenever they were split into lists (thanks to the struc-tural information provided by Akoma Ntoso). The labels of these sentences are the sameas those provided by DAPRECO with the addition of a “none” category. We abbrevi-ated “obligationRule”, “permissionRule”, “constitutiveRule” in “obligation”, “permis-sion” and “constitutive” respectively.The class “obligation” is referred to those sentences which have at least one obliga-tion rule in their related formulæ. The class “permission” is referred to those sentenceswhich have at least one permission rule in their related formulæ. The class “constitutive”is referred to those sentences which just constitutive rules in their related formulæ. Theclass “none” is referred to all sentences which have no rule at all. These labels allowed 4different experimental settings, as shown in Table 1:Scenario 1 is a binary classification and aims at discriminating between rule andnon-rule instances. In this scenario, all labels other than “none” are considered rule, while2These provisions belong to the body of the GDPR (preamble and conclusions were excluded), and aregenerally paragraphs or list points, which may sometimes consist of multiple sentences.3The DAPRECO knowledge base can be freely downloaded from: https://github.com/dapreco/daprecokb/blob/master/gdpr/rioKB_GDPR.xml.4The Akoma Ntoso representation of the GDPR can be currently accessed from https://github.com/guerret/lu.uni.dapreco.parser/blob/master/resources/akn-act-gdpr-full.xml.October 2022Table 1. Number of instances per class per scenario.Classes Instancesrule 260Scenario 1non-rule 447Classes Instancesdeontic 204Scenario 2non-deontic 503Classes Instancesobligation 156permission 44Scenario 3none 503Classes Instancesobligation 156permission 44constitutive 56Scenario 4none 447Table 2. Results for the two stratified baselines applied to scenario 1, 2, 3 and 4. Within the brackets, thenumber of instances is reported. P = precision; R = recall, F1 = F1-score; Acc = Accuracy; Mcr = Macro F1.Baseline Scenario 1P R F1 Acc Mcrrule(39) .37 .33 .35non-rule(67).63 .67 .65.55 .50Baseline Scenario 2P R F1 Acc Mcrdeontic(30) .23 .23 .23non-deontic(76).70 .70 .70.57 .47Baseline Scenario 3P R F1 Acc Mcrobligation(24) .24 .17 .20permission(6) .20 .33 .25none(75) .73 .76 .75.60 .40Baseline Scenario 4P R F1 Acc Mcrobligation(23) .16 .13 .14permission(7) .20 .14 .17constitutive(8) .00 .00 .00none(67) .58 .63 .60.44 .23“non-rule” is just an alias for “none”. Scenario 2 focus on a binary classification betweendeontic instances (i.e., any sentence labelled as either “obligation” or “permission”) andnon-deontic instances (i.e., all instances which are labelled neither as “obligation” noras “permission”). Scenario 3 is a multiclassification which considers the classes “obliga-tion”, “permission” and “none” (with “constitutive” considered as part of the latter). Sce-nario 4 is a multiclassification which considers the classes “obligation”, “permission”,“constitutive” and “none”. For the multi-classifications (i.e. Scenario 3 and 4) four state-ments have been removed, since the classes “obligation” and “permission” overlapped.To assess the non-triviality of our experiments, we employed different kinds of base-line, showing their difficulty in performing each classification task. In Table 2, we re-ported the results of a “stratified” baseline, which was the better performing among thebaseline methods5. As can be seen from Table 2, we applied the “stratified” baseline onall the scenarios achieving quite low performances on all of them.As far as the experimental settings are concerned, the dataset was divided into 70%for the training phase, 15% for the test and 15% for the validation; and for all instances,a max length of 30 was applied.Regarding the Transfer Learning architecture, 3 pre-trained language model havebeen fine-tuned, namely BERT [3], DistilBERT [18], and the LegalBert trained on Eu-rLex [19]. These three neural architectures were fine-tuned by adding two linear layerswith a ReLu activation function and with a dropout of 0.2 after each activation, and a finaloutput layer was added for the classification, through a softmax activation function. The5The so-called “stratified” baselines can reach higher scores because they reflect the class distribution.October 2022Table 3. Results of the 4 scenarios. P = precision; R = recall, F1 = F1-score, S/T = Support/Total ratio.BERT DistilBERT LegalBERTScen. ClassesP R F1 P R F1 P R F1S/Trule .74 .95 .83 .77 .95 .85 .75 .77 .76 39/2601non-rule .96 .81 .88 .97 .84 .90 .87 .85 .86 68/447Accuracy .86 Accuracy .88 Accuracy .82Total:107/707Macro avg .86 Macro avg .87 Macro avg .81Weight. avg .86 Weight. avg .88 Weight. avg .82deontic .74 .90 .81 .82 .90 .86 .80 .77 .79 31/2002non-deontic .96 .87 .91 .96 .92 .94 .91 .92 .92 76/507Accuracy .88 Accuracy .92 Accuracy .88Total:107/707Macro avg .86 Macro avg .90 Macro avg .85Weight. avg .88 Weight. avg .92 Weight. avg .88obligationRule .74 .83 .78 .74 .83 .78 .63 .92 .75 24/156permissionRule .50 .83 .62 .36 .67 .47 .56 .83 .67 6/443none .97 .88 .92 .94 .84 .89 1.0 .82 .90 76/503Accuracy .87 Accuracy .83 Accuracy .84Total:106/703Macro avg .78 Macro avg .71 Macro avg .77Weight. avg .88 Weight. avg .84 Weight. avg .85obligationRule .70 .79 .75 .80 .83 .82 .84 .67 .74 24/156permissionRule .60 .50 .55 .40 .67 .50 .17 .67 .28 6/44constitutiveRule .36 1.0 .53 .47 .89 .62 .89 .89 .89 9/564none 1.0 .73 .84 .94 .76 .84 .96 .79 .87 67/447Accuracy .75 Accuracy .78 Accuracy .76Total:106/703Macro avg .67 Macro avg .69 Macro avg .69Weight. avg .78 Weight. avg .80 Weight. avg .81fine-tuning process of these 3 neural architectures was performed in 10 epochs (learningrate: 1e-3; batch size: 32).The final results on the validation set are reported in Table 3, where it can be seenthat DistilBERT outperforms the other classifiers in the binary classifications, with anaverage score reaching .88 in the first scenario and .92 in the second one.The results for the third and fourth scenarios are less straightforward and show thatBERT slightly outperforms other classifiers in the third scenario, while LegalBERT out-performed the other models in the fourth scenario. The main problem for the multiclassi-fications, is the class unbalance and the restricted amount of instances for some classes.In spite of this, scores are encouraging, especially considering the small amount of data.5. ConclusionsThe contribution of this work is showing how Transfer Learning methods can leveragethe information provided in LegalXML to train classifiers capable of automatically clas-sifying deontic sentences and rules. In the future, we would like to create a stronger con-nection with the ontological sphere by using PrOnto [20], strengthening this hybrid AIapproach, which combines symbolic knowledge with sub-symbolic methods.October 2022In this work, we were not targeting in the internal elements of the logical formulæ,we just addressed the ontological classes of each rule. However, in the future we wantto create classifiers that directly address the internal components of each rule, trying tofind a match between portions of natural language and portions of rules. In general, theability to connect each internal component (or at least some) of the deontic formulæcontained in DAPRECO directly to the portion of natural language where the componentis communicated or expressed is a crucial future direction, and an important step towardsthe long-term goal of filling the gap between natural language and the logical-inferentialsphere, which would generate a more reliable and explainable Artificial Intelligence.References[1] Ashley KD. Artificial intelligence and legal analytics: new tools for law practice in the digital age. 2017.[2] Wyner A, Peters W. On rule extraction from regulations. 2011:113-22.[3] Devlin J, Chang MW, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers forlanguage understanding. arXiv preprint arXiv:181004805. 2018.[4] Liga D, Palmirani M; 20. Transfer Learning with Sentence Embeddings for Argumentative EvidenceClassification. Proceedings of the 20th Workshop on Computational Models of Natural Argument. 2020.[5] Liga D, Palmirani M. Deontic Sentence Classification Using Tree Kernel Classifiers. 2023:54–73.[6] Kiyavitskaya N, Zeni N, Breaux TD, Antón AI, Cordy JR, Mich L, et al.; Springer. Automating theextraction of rights and obligations for regulatory compliance. 2008:154-68.[7] Waltl B, Muhr J, Glaser I, Bonczek G, Scepankova E, Matthes F. Classifying Legal Norms with ActiveMachine Learning. 2017:11-20.[8] Gao X, Singh MP. Extracting normative relationships from business contracts. 2014:101-8.[9] Dragoni M, Villata S, Rizzi W, Governatori G. Combining Natural Language Processing Approachesfor Rule Extraction from Legal Documents. 2018:287–300.[10] de Maat E, Winkels R. Automated Classification of Norms in Sources of Law. 2010:170–191.[11] Neill JO, Buitelaar P, Robin C, Brien LO. Classifying sentential modality in legal language: a use casein financial regulations, acts and directives. 2017:159-68.[12] Chalkidis I, Androutsopoulos I, Michos A. Obligation and prohibition extraction using hierarchicalrnns. arXiv preprint arXiv:180503871. 2018.[13] Joshi V, Anish PR, Ghaisas S. Domain adaptation for an automated classification of deontic modalitiesin software engineering contracts. 2021:1275-80.[14] Shaghaghian S, Feng LY, Jafarpour B, Pogrebnyakov N; IEEE. Customizing Contextualized LanguageModels for Legal Document Reviews. 2020:2139-48.[15] Robaldo L, Bartolini C, Palmirani M, Rossi A, Martoni M, Lenzini G. Formalizing GDPR provisionsin reified I/O logic: the DAPRECO knowledge base. Journal of Logic, Language and Information.2020;29(4):401-49.[16] Athan T, Boley H, Governatori G, Palmirani M, Paschke A, Wyner A. Oasis legalruleml. In: Proceedingsof the Fourteenth International Conference on AI and Law; 2013. p. 3-12.[17] Makinson D, Van Der Torre L. Input/output logics. Journal of philosophical logic. 2000;29(4):383-408.[18] Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaperand lighter. arXiv preprint arXiv:191001108. 2019.[19] Chalkidis I, Fergadiotis M, Malakasiotis P, Aletras N, Androutsopoulos I. LEGAL-BERT: The muppetsstraight out of law school. arXiv preprint arXiv:201002559. 2020.[20] Palmirani M, Martoni M, Rossi A, Bartolini C, Robaldo L. Pronto: Privacy ontology for legal compli-ance. 2018:142-51.

Transfer Learning for Deontic Rule Classification: the Case Study of the GDPR

https://orbilu.uni.lu/bitstream/10993/53791/1/EasyChair-Preprint-9170.pdf

Transfer Learning for Deontic Rule Classification: the Case Study of the GDPR

Abstract

Similar works

Full text

Available Versions

Open Repository and Bibliography - Luxembourg