Sunnah Arabic Corpus is an annotated linguistic resource that consists of 144K words/170K tokens of the Hadith narratives (an utterance attributed to prophet Mohammed) extracted from Riyāḍu Aṣṣāliḥīn book. As a first layer of annotation, the corpus has been fully diacritized. In addition, each orthographic word/token is segmented into its syntactic words. And each syntactic word is tagged with its part-of-speech in addition to multiple morphological features. Several hadith translations in different languages are provided and aligned at the narrative/paragraph level. Hadith Arabic Corpus follows the successful Quranic Arabic Corpus in its standards (corpus.quran.com). Sunnah Arabic Corpus is freely available under the Creative Commons Attribution-ShareAlike 4.0 International License

Alosaimy, A

Atwell, E

White Rose Research Online

This is a repository copy of Sunnah Arabic Corpus: Design and Methodology..White Rose Research Online URL for this paper:http://eprints.whiterose.ac.uk/125569/Version: Accepted VersionProceedings Paper:Alosaimy, A and Atwell, E orcid.org/0000-0001-9395-3764 (2017) Sunnah Arabic Corpus: Design and Methodology. In: Proceedings of the 5th International Conference on Islamic Applications in Computer Science and Technologies (IMAN 2017). IMAN 2017, 26-28 Dec 2017, Semarang, Indonesia. . This is an author produced version of the paper 'Sunnah Arabic Corpus: Design and Methodology', presented at IMAN 2017. eprints@whiterose.ac.ukhttps://eprints.whiterose.ac.uk/Reuse Unless indicated otherwise, fulltext items are protected by copyright with all rights reserved. The copyright exception in section 29 of the Copyright, Designs and Patents Act 1988 allows the making of a single copy solely for the purpose of non-commercial research or private study within the limits of fair dealing. The publisher or other rights-holder may allow further reproduction and re-use of this version - refer to the White Rose Research Online record for this item. Where records identify the publisher as the copyright holder, users can verify any specific terms of use on the publisher’s website. Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing eprints@whiterose.ac.uk including the URL of the record and the reason for the withdrawal request. 5th  International Conference on Islamic Applications in Computer Science And Technology, 26 – 28 December 2017, Semarang, Indonesia    Sunnah Arabic Corpus: Design and Methodology  Abdulrahman Alosaimy1, Eric Atwell2  School of Computing, University of Leeds, United Kingdom 1scama@leeds.ac.uk,2e.s.atwell@leeds.ac.uk  ABSTRACT Sunnah Arabic Corpus is an annotated linguistic resource that consists of 144K words/170K tokens of the Hadith narratives (an utterance attributed to prophet Mohammed) extracted from Riyāḍu Aṣṣāliḥīn book. As a first layer of annotation, the corpus has been fully diacritized. In addition, each orthographic word/token is segmented into its syntactic words. And each syntactic word is tagged with its part-of-speech in addition to multiple morphological features. Several hadith translations in different languages are provided and aligned at the narrative/paragraph level. Hadith Arabic Corpus follows the successful Quranic Arabic Corpus in its standards (corpus.quran.com). Sunnah Arabic Corpus is freely available under the Creative Commons Attribution-ShareAlike 4.0 International License  Keywords: Arabic, corpus, annotation, Hadith, Sunnah, morphology 1. Introduction and Motivation Language resources (LRs) are recognized as key components in the development of Natural Language Processing. Annotated corpora, as one example of LR, used to perform statistical analysis, hypothesis testing, verifying grammar within a language domain and for building statistical computational models. Several scholars show the lack of freely available Arabic resources, especially gold standard annotated corpora (Albared, Omar, & Ab Aziz, 2009; Yaseen et al., 2006). In the case of Classical Arabic, Quranic Arabic Corpus (Dukes & Habash, 2010) and Al-Mus’haf Corpus (Zeroual & Lakhouaja, 2016) are two available annotated corpus but they are limited to Quranic texts. Mohamed (2012) built a small corpus of religious texts (and one of the texts is a small Hadith book) and confirmed the need of a larger classical corpus. Sunnah Arabic Corpus (SAC) is the first corpus of Arabic Hadith (prophet sayings) that is freely available morpho-syntactically annotated corpus using a fine-grained tagset that conforms with traditional Arabic grammar. It follows the success of the Quranic Arabic Corpus (QAC). In the rest of the paper, we will list the main features and potential uses of SAC, its collection process and content, and finally its availability and accessibility.  2. Corpus Collection Process Sunnah Arabic Corpus currently has only one book: Riyāḍu Aṣṣāliḥīn (aka The Meadows of the Righteous) a compilation of 1896 hadith narratives written by Al-Nawawi and published on 1334. The book will henceforth be referred to as Riyad. Riyad was chosen for several reasons:  1. It has widely accepted as a valid source of prophet sayings. 2. Its codex was validated and investigated by several scholars by scientific palaeographical process. 5th  International Conference on Islamic Applications in Computer Science And Technology, 26 – 28 December 2017, Semarang, Indonesia  2  3. A small part (42 narratives, 4479 words, ~5% of the book), famously known as Al-Arbaeen Al-Nawawyyah, has been studied linguistically in traditional Ia’rab Arabic grammar (by two books )AlOmari, 2005; Yosef, 2003)). 4. It has been translated into at least 18 languages (e.g. English). 5. It’s narratives has been explained by 6 written books, at least by 11 scholar (spoken explanation). While it is available through many websites (e.g. IslamHouse.com, Sunnah.com), we chose to download an e-book version of the book from Shamela1, a downloadable library that contains at least 5300 Arabic books in Islamic studies , as this library become the standard library of Arabic classical books. It has been used to obtain Arabic classical text in building several corpora (Alrabia, Al-Salman, Atwell, & Alhelewh, 2014; Belinkov, Magidow, Romanov, Shmidman, & Koppel, 2016; Zerrouki & Balla, 2017).  Two versions of Riyad were available in Shamela library and we chose the version with ID# 2348 (Alfahal, 2007). This version is the one investigated by Maher Alfahal who made his investigation and commentaries open freely. Both versions has same numbering, hadith text (with some slight differences), but they both differ greatly in the commentaries.  Diacratization of both version is not full (not every letter has its short vowel); Maher’s version is more thorough and accurate using a sample of five narratives randomly chosen. Quranic verses are fully diacratized in both versions (a common standard in book editing).                                                   1 http://shamela.ws/index.php/book/2348 <?xml version="1.0" encoding="utf-8" standalone="no"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xml:lang="ar" lang="ar" dir="rtl" xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <link href="../style.css" rel="stylesheet" type="text/css" /> <title>ریاض الصالحین ت الفحل</title></head> <body class="rtl">   <div dir="rtl" id="book-container"><hr/> <a id='C159'></a><a id='C160'></a> <span class="title">(6) -  كتاب عی ادة المریض و تشییع الم ی ت والص الة علیھ       ّ     ّ َ        َ          َ      َ   و حضور د فنھ  و المكث ع ن د  قبره  ب عد  د فنھ َ  َ ِ   َِْ       َ ِ  َ      َ</span><br /><span class="red">144 - </span><span class="title">باب عیادة المریض</span><br /><span class="red">894 - </span> َ َ  َ َ َ  عن الب ر اء  بن عاز ب  رضي V عنھما، ق ال : أم ر ن ا ر   ُسول       ََ  ِ    ٍِ           V-  علیھ وسلم V َِْبع ی اد ة  ال م ریض ، و ات ب اع  الج ن از ة ، و ت ش م  -صلى َ  َِ َ  َ  ِ  َ ِّ  َ ِ یت  الع اط س ،  ِ َ َِ َْ   ِ  ِ  َ  ِ  َ ِ  َّو إب ر ار  ال م ق س م ، و ن ص ر  الم ظ ل وم ، و إج اب ة  الد اع ي، و إف ش اء  الس   ْ  َ   ِ َّ  ِ َ َ   َ  ِ ُْ  َ  ِ َْ َ َ َْ ِ ع ل ی ھ .   َِ    ٌال م . متفق  َ  ْ َ  ِ ُْ ِِْ )1 )<span class="footnote-hr">&nbsp;</span><span class="footnote">(1) 239انظر الحدیث ( ).</span> </div><hr/> <div class="center"> الصفحة: ¦  1الجزء: ¦  894الحدیث: 273</div></body></html>  Figure 7.2.1: XML version of one page of Riyad book extracted from its EPUB version. 5th  International Conference on Islamic Applications in Computer Science And Technology, 26 – 28 December 2017, Semarang, Indonesia 3  Shamela books are available in three formats: PDF, EPUB, and BOK (used for their downloadable desktop software). PDF format is used for the scanned images of the book, and the text is not extractable easily. BOK version is not suitable as it requires their software to open. Therefore, we chose to proceed with EPUB format, an e-book file cross-platform widely-used format to view and read the book. Since EPUB format is XML-based, the extraction of the xml version of the book is easy.  We chose to use the EPUB version. However, we found that the xml version of Riyad does not tell the difference between some components of the text (like footnotes’ co-reference, and page numbers). It does not separate the chain of narrators, prophet sayings, and citation. It does not either provide a table of contents (see Figure 7.2.1 for details). Therefore, we developed a custom software2  to extract the narratives and verses in a structured format which identifies footnotes, chapters, and sections, remove inline annotations (for example page break of original book: “[p.34 started]”) and separate footnotes from original text and link it with co-reference. It also merge narratives that spans into multiple pages. It also import Quranic units annotations from QAC corpus. Section 7.3 describes the output of the software, i.e. corpus content. 3. Corpus Content Riyad is a collection of 2330 units (precisely 435 Quranic verses and 1896 hadith narratives). It is classified into 20 chapters, and each chapter contains several sections, with a total of 372 sections the covers Islamic morals, acts of worship, and manners. Each section covers a specific topic, and verses and narratives that support the topic.  The corpus consist of ~119K Arabic words: which about 110K words compose Hadith narratives. The rest are either compose authors commentaries, his introduction, or Quranic verses. More statistics about the corpus are in Table 1. After removing short vowels and punctuations, the number of word types of hadith narratives is ~17K. The word frequency list is presented in Table 2.  Table 1: Some statistics about the Sunnah Arabic Corpus. Counts  Counts  Tokens 170453 Word Types 17786 Words  144106 Fully diacritized Words 102746 Sentences 7670  86.08% Paragraphs 2075 Distinct 5-grams 90347 Documents 372 Hadith Narratives 1896  Table 2: The frequency list of Sunnah Arabic Corpus. Word Count Word Count Word Count Word Count  160 قلت  251 ھذا  528 إلى  7883 هللا  157 أنس  243 تعالى  528 یا  3476 علیھ  157 بھا  243 یوم  523 كان  3182 قال  145 و  237 أي  516 لھ  2528 صلى  142 فإذا  218 عمر  450 وال  2470 وسلم  141 منھ  213 فیھ  447 حتى  2068 من  139 بین  212 الذي  412 إن  1990 رسول                                                 2 http://github.com/aosaimy/wasim/external_tools/ 5th  International Conference on Islamic Applications in Computer Science And Technology, 26 – 28 December 2017, Semarang, Indonesia  4   138 وما  210 صحیح  405 أو  1856 رضي  137 النار  206 لي  373 إذا  1419 عنھ  134 عائشة  206 لم  347 أبو  1417 في  132 أنھ  206 الترمذي  339 وقال  1406 وعن  130 فلما  205 قالت  337 ابن  1173 أن  130 شيء  202 رجل  336 ھریرة  1151 رواه  128 وهللا  200 فإن  331 عنھما  876 ما  127 إني  195 وھو  319 حدیث  867 ال  124 مع  194 ھو  303 یقول  833 فقال  124 فقلت  192 عنھا  297 حسن  829 بن  123 الرجل  192 الجنة  282 عبد  825 عن  122 الصالة  188 قد  275 روایة  808 على  122 أھل  185 وعنھ  274 وفي  804 أبي  121 أحد  177 كل  269 البخاري  747 متفق  120 أحدكم  174 اللھم  266 بھ  650 النبي  118 علي  174 وإن  266 ذلك  615 مسلم  118 عند  169 ومن  258 الناس  564 ثم  117 شیئا 165 سمعت  255 داود  530 إال   For each unit, we keep a record of its numbering, chapter and page in the original printed book, and automatically split its text into sentences: with tags to describe the purpose of that sentence using a simple rule based segmenter.  Hadith units were POS tagged and annotated semi-automatically using Wasim annotation tool, and Quranic units are matched with their annotation of QAC annotation. The format for storing annotation is CoNLL-U format v2.0, which is used by the Universal Dependencies project (Nivre & Agic, 2017). We used a slightly modified tagset of the Quranic Arabic Corpus.  4. Potential Uses • It can help Arabic learner by understanding the interaction of sentence components, since it follows Arabic grammar of ʾiʿrāb (إعراب).  • It may help linguistic researchers interested in Hadith to study the stylistic and vocabulary and other linguistic studies.  • It can help researchers in translation studies to compare different translations of same Hadith.  • For researchers of Arabic Language Processing, it may help improving machine translation and Classical Arabic understanding using morphological and syntactical annotations.  5. Main features The project aims to provide an easy-to-use online access to: • Fully diacritized text • A part-of-speech concordance search results organized by lemma or surface form. 5th  International Conference on Islamic Applications in Computer Science And Technology, 26 – 28 December 2017, Semarang, Indonesia 5  • A morpheme-based part-of-speech tagged corpus with its morphological features3. • I‘rāb of a sample of hadiths in a novel visualized way4. • Morphological and lemma-based search for the corpus. • Narrator-based search for the corpus. • A parallel text of English-Arabic aligned on the hadith level. • A parallel text of Arabic-Arabic commentaries aligned on the hadith level. 6. A Layer of Annotation: Diacritization Since Riyad is highly cited, we have fully diacritized the text, by automatically “borrowing” diacritization from other books. We developed a software that matches the undicritized version of one word in Riyadh with its equivalent in other books using their word n-gram concordance. The original source of Riyad is about 47.1% fully diacritized 5 , and after borrowing diacritization, the percentage jumps to 86.08%. In terms of letters, 48.66% was diacritized, and after borrowing diacritization, the percentage jumps to 76.41% with low diacritic error rate (0.004). 7. Accessibility and Availability Sunnah Arabic Corpus is freely available under the Creative Commons Attribution-ShareAlike 4.0 International License. This permissive license allows commercial uses and allows adaptations be shared as long as others share alike. The corpus will also be available online6 soon which allows an easy-to-use corpus functionalities. 8. Conclusion and Future Work We have introduced the corpus and described its collection process, content, and its distribution and availability. We described briefly its potential uses in different fields including linguistics studies, natural languages processing, and translation studies.  We are working on the morphological and syntactical annotation of the corpus, and plan to publish its data, guidelines and evaluation soon. For future work, it will be very helpful to manually align the corpus to different languages/commentaries in sentence level. In addition, word-to-word translation proved to be helpful in Quran understanding, and we might consider automatic word alignment with other languages.   9. References Albared, M., Omar, N., & Ab Aziz, M. J. (2009). Arabic part of speech disambiguation: A survey. International Review on Computers and Software, 4(5), 517–532. Alfahal, M. Y. (2007). Riyad-us-Saliheen (with commentary on Ahadith). Dar Ibn Katheer, Damascus, Syria. AlOmari, O. (2005). إعراب األربعین النوویة - The Iaarab Of The Nawawi Forty Book.                                                 3 Manually verified version will be available next year. 4 Rest of Hadiths will be added later. 5 The definition of fully diacritized form is hard to compute accurately. For example, using our diacritization standard, the letter the precedes a long vowel are not diacritized. However, deciding whether Yaa letter is consonant or a vowel is ambiguous.  6 http://corpus.al-osaimy.com 5th  International Conference on Islamic Applications in Computer Science And Technology, 26 – 28 December 2017, Semarang, Indonesia  6  Alrabia, M., Al-Salman, A., Atwell, E., & Alhelewh, N. (2014). KSUCCA : A Key To Exploring Arabic Historical Linguistics. International Journal of Computational Linguistics. Belinkov, Y., Magidow, A., Romanov, M., Shmidman, A., & Koppel, M. (2016). Shamela: A Large-Scale Historical Arabic Corpus. arXiv Preprint arXiv:1612.08989. Nivre, J., & Agic, L. ˇZeljko. (2017). Universal dependencies 2.0 CoNLL 2017 shared task development and test data. LINDAT/CLARIN digital library at the Institute of Formal and Applied. Yaseen, M., Attia, M., Maegaard, B., Choukri, K., Paulsson, N., Haamid, S., … Ragheb, A. (2006). Building Annotated Written and Spoken Arabic LR’s in NEMLAR Project. Lrec, 533–538. Yosef, H. A. (2003). إعراب األربعین حدیث ا النوویة         ً                - The Iarab of Al-Nawawi’s Forty Hadith. Cairo: AlMukhtar. Zeroual, I., & Lakhouaja, A. (2016). A new Quranic Corpus rich in morphosyntactical information. International Journal of Speech Technology, 1–8. Zerrouki, T., & Balla, A. (2017). Tashkeela: Novel corpus of Arabic vocalized texts, data for auto-diacritization systems. Data in Brief, 11, 147–151. http://doi.org/10.1016/j.dib.2017.01.011    Abstract in Arabic ً موسوم صرفی ا ونحوی ا لغوي مصدر  يھالسنة العربیة ذخیرة       ً ألف  170كلمة / ألف  144 ما یقارب یتكون من        قد تم ف. كطبقة أولى من الشرح، ریاض الصالحینالمستخرجة من كتاب  رمز من األحادیث النبویة الشریفةكل ثم تم توسیم إلى كلماتھا النحویة. بشكل كامل. باإلضافة إلى ذلك، تم تقسیم كل كلمة إمالئیة تشكیل النص لذخیرة باإلضافة لذلك، تحتوي ا. الخصائص الصرفیةباإلضافة إلى العدید من المناسب جزء الكالم بنحویة ة كلمعلى مستوى متوازیة مع النص العربي للحدیث الشریف من الترجمات بلغات مختلفة على مجموعة  التوسیم  في معاییرھذه الذخیرة تتبع ذخیرة القرآن الكریم من جامعة لیدز . الحدیث/الفقرة(corpus.quran.com).نترنت.ھذه الذخیرة متوفرة بشكل حر ومجاني عبر موقع الذخیرة على شبكة اإل    

Sunnah Arabic Corpus: Design and Methodology.

https://eprints.whiterose.ac.uk/125569/1/final.pdf

Sunnah Arabic Corpus: Design and Methodology.

Abstract

Similar works

Full text

Available Versions

White Rose Research Online