Tenny, Carol L.

DSpace@MIT

Q180.J3.M22ol7 1111Ii'IbM.I.T.7)11ILALICERAMA COrTEXT FREE RULE SYSTEM FOR PARSING JAPANESECarol L TennyDepartment of Linguistics and Philosophy andAthena Language Learning Project,Massachusetts Institute of Technologymm0--_'m "!'~~~~~~~ i fi - c·'I !-1~~~~~~·iDISTRIBUTED COURTESY OFMIT-JAPAN SCIENCE AND TECHNOLOGY PROGRAMRoom E53-447Massachusetts Institute of TechnologyCambridge, MA 02139 (617) 253-2449A CONTEXT FREE RULE SYSTEM FOR PARSING JAPANESECarol L TennyDepartment of Linguistics and Philosophy. andAthena Language Learning Project,Massachusetts Institute of Technology__I__CI____________- 1. IntroductionA project is underway at MIT to develop computer based language teachingmaterials that will combine highly advanced computer technology with modernlanguage teaching methods. The Athena Language Project draws on MIT sconsiderable resources in computation, linguistics and artificialintelligence to produce language lab materials that will incorporateinteractive video and audio components as well as computer intelligencecapable of understanding commands in natural language. The final product isenvisioned as a set of language exercises that will allow the student tointeract with the computer in a creative way, in the language he is learningby giving commands or responding to fictional situations posed by thecomputer. The computer in turn understands and responds to the student inthe language of instruction. The languages included in this project arethose currently taught at MIT German, Spanish, French, Russian, English as aSecond language, and Japanese.Japanese is fast becoming a language of international importance intechnical and scientific exchange as well as in the business world. It hasonly recently begun to be taught at MIT. The popularity of the Japaneselanguage offering at MIT has outstripped the ability of the foreign languagesdepartment to accomodate all students who want to study Japanese. In view ofthe popularity and importance of the language, it was essential that Japanesebe included in the project To this end, the MIT Japan Science andTechnology Program has funded the development of a Japanese component of thesoftware. A rule system and lexicon for parsing Japanese were designed bythe author. The Athena Language Project is translating the rule design into- 1______ __s_LISP code for implementation within the program that translates naturallanguages into abstract representations of meaning.The parsing rules are the component of the software that is tailored to theparticular language The system used by the Athena project has, built intoit, some knowledge of the world and a capacity for manipulating certainfictional situations. This part of the system remains the same for eachlanguage to which it is applied, but the system must have in addition, aseparate set of rules for understanding input sentences in each language.These rules work together with a lexicon for the language, to translate asentence or a string of words in the natural language into meaning structureswhich the system can then manipulate. This invariably means taking a stringof words organized linearly, in one dimension, and reorganizing them intogroupings with hierarchichal as well as linear relationships between theparts. The precise way in which this is done by human intelligence is thetopic of some research and speculation, but several working models for thisprocess are in use in work on artificial intelligence. The system employedby the Athena Language Project is the Malone parser, a rule-based parser witha lookahead of one, operating on a system of context-free rules. The set ofrules designed for Japanese will not be listed in this document, but in theremainder of the document, the general coverage of the rules, as well as theapproach taken towards particular problems, will be outlined.In order to design this rule system, it was necessary to first establish asatisfactory idealization of Japanese grammar. This meant deciding on twothings: 1) a version of the grammar that could compromise between therigorous order required by the parser, and the natural fluidity expected by___^ _ _ __ILlsl I iCsll II ____the human user, and 2) how much and which part of the language the rulesshould be able to handle. The parsing rules cannot be expected to modeleverything a human being can do, because the parser responds only to thestrict grammar or syntax of a sentence, while the human handling of languageis affected by pragmatic, semantic and discoursal as well as syntacticconditions. Decisions had to be made about how much of the syntax the ruleswould cover, and what kinds of linguistic information they should be preparedto handle. Linguistic, computational and pedagogical considerations enteredinto this problem. Linguistically, the goal was to incorporate as manyinsights as possible from current linguistic theory into the structure ofJapanese grammar. The more the system is based on sound and simpleprinciples, the easier it will be to expand, alter or otherwise develop itlater. Computationally, it was necessary to keep the rules as simple aspossible, and avoid building into them extra machinery that might belinguistically accurate but would greatly inhibit the efficient working ofthe rules. And pedagogically, the rules should be designed to handlelinguistic constructions that a fourth semester student of Japanese mightreasonably be expected to produce.2. Approaches to Problems Posed by Japanese Grammar2.1 The Topic Marker '"Wa"A particular problem is posed by the topic marker "wa". There are noprecedents for parsing this type of construction in the other, Indo-Europeanlanguages included in the project. The use of the topic marker "wa" by humanspeakers is governed not only by syntactic principles, but by semantic anddiscourse factors as well, some of which are more subtle than a system such-3----- ---- - - --------- ----. ..... + .. .G ..... .as this one can capture. The following set of idealizations were employedfor the rule system:li) There is only one topic marker per sentence.ii) The topic marker is only used in matrix clauses.ii) The topic marker may only occur at the beginning of theclause.This means that the system of rules will accept the following sentences:Hanako wa gohan o tabemashitagohan wa Hanako ga tabemashitaTobkyo wa Taroo ga ikimashitaTaroo wa Tokyo ni ikimashitaHanako wa Taroo ga Tokyo ni iku to iimashitaTokyo wa fuyu ga samuiTaroo wa otosan ga shindaThe system will reject the following sentences. At this point it does nothandle constructions with the contrastive use of "wa", which are somewhatfreer than the straightforward topical use of "wa".2gohan o Hanako wa tabemashitaTokyo ni Taroo wa ikimashitaTaroo wa Tobkyo ni iku to Hanako wa iimashita2.2 Free Word OrderThe relatively free rd order of Japanese, compared to English, French,1. Professor Michio Tsutsui of the MIT-Wellesley Exchange Program kindlyadvised on this problem. Any inadequacies however, are the author's. Theseidealizations are still being used on a trial basis. They may be changed aspedagogical or computational considerations require.2. For more discussion on the use of the topic marker "wa" see Kuno, S.(1973) The Structure of the Japanese Language MIT Press, Cambridge, pp.35-79.- 4-_ _·I·__ I CIL1___Ceb_·lbCFI tl- I · IIC1---·---ll._. -I_-·- -- ·L-lSpanish or German, presents a major problem for a rule-based parser such asthe Malone parser. Each rule must express a possible fixed word order -- ofwhich there are a great many in Japanese. This means that coverage of even asmall number of basic constructions requires a large number of rules,reducing the overall efficiency of the parser. This problem was in partcircumvented by labeling several different kinds of constructions with onename, so that fewer rules would be required to address all theconstructions. However, this solution has reduced efficiency elsewhere inthe grammar because some extra machinery is required to distinguish thevarious constructions at a later stage.The problem of free word order necessitates another compromise betweenlinguistic and computational demands. Japanese is treated by these rules asa non-configurational language, in spite of the growing evidence to thecontrary.3 A non-configurational language is one in which no word order ismore basic than any other, yet some facts about how pronouns are construedwith noun phrases within a sentence point to the idea that the word order,Subject-Object-Verb is most basic in Japanese. To build this into the rulesystem would increase the number of rules dramatically but make the systemonly slightly closer to human ability. These facts about pronoun construalwould not be likely to be within the reach of second year students, orperhaps even any non-native speakers of Japanese. It was judged that the3. See Saito, M. (1985) "Some Asymmetries in Japanese and their TheoreticalImplications", Ph.D. dissertation, MIT Department of Linguistics andPhilosophy; and Whitman, J. (1982) "Configurationality Parameters",unpublished ms. Harvard University Department of Linguistics. A revisedversion is to appear in T. Imai and M. Saito, eds., Issues in JapaneseLinguistics, Foris Press, Dordrecht.- 5--~~~- -- ~~~~~'  ~~~~~~~' ~~~~~^------ - r-1_ -~ ·~-- · - *~- -·--r·~ l~----- -- -----gains in linguistic coverage that would result from a configurationaltreatment of Japanese would not be worth the added complexity it wouldgenerate in the rule system.2.3 Noun Phrase DropThe meaning structures into which the natural language is translated areessentially annotated case frames. That is, they are the simple listing of apredicate (e.g., a verb) and its arguments (e.g., noun phrases), and therelation between them. For example, the following sentence would berepresented with the case frame below it.Hanako ga Taroo ni tegami o okurimashita."Hanako sent a letter to Taroo."predicate: okurusubject argument: Hanakoobject argument: tegamidative argument: TarooIn Japanese the relation between predicates and arguments is expressed viacase markers ("ga" for subject, "o" for object, etc.) rather than wordorder, as in English. Translating a Japanese sentence into its case frame isrelatively straightforward as long as the roles of the case markers areunambiguous, and all the noun phrases are mentioned overtly in the sentence.However, Japanese allows noun phrases to be dropped from sentences, on agreater scale than in the other languages included in the project. TheAthena system operates by building the case frame required by the particularverb and inserting arguments into the case frame as they are encountered inthe sentence. Then the case frame is checked, and if some argument slots areempty, the sentence is rejected. This poses a particular problem for- 6----1--"-~cl~l------ ---lI1CI - 1. I^--··-----I-·_1111111----------Japanese, because in Japanese the arguments may not be overtly in thesentence, yet be understood by the speaker and hearer as being there.Incompletely filled case frames must be permitted in Japanese, and dummyarguments inserted into the empty argument slots. (The dummy arguments canthen be assigned meaning in the discourse component of the grammar.) Somegeneral code for the handling of case frames had to be rewritten toaccomodate noun phrase drop in Japanese.2.4 Morphology and the LexiconA fourth problem presented by Japanese is the rich morphology of thelanguage. The morphological analyzer used by the Athena project issophisticated, but not as sophisticated as Japanese morphology. Adjustmentshad to be made in the Athena system to allow morphology to change the caseframe of a verb. For instance, the verb "nageru" ('throw') would have a caseframe requiring two arguments -- the subject that throws, and the object thatis thrown. But when the morpheme "sase" is applied to make the verb"nagesaseru" ('make someone throw'), the new case frame requires threearguments -- the thing thrown, the person throwing, and the person who causesthe thrower to engage in the action of throwing. Morphological processessuch as this are common in the languages of the world, and taking measures tostreamline handling of the phenomenon will be beneficial to the system in thelong run.Japanese, as well as the other languages in the project, will be moreeffectively handled when a word-formation component is added to the Athenasystem -- a change planned for the summer of 1986. This will allow the rulesto recognize morphological operations that change the category of a word, and- 7----I- . -·-----------I .--.-------- I --will reduce the size of the lexicon. An example of this is suffixation thatchanges a noun into an adjective. e.g.:"otoko" + "rashii" --> "otokorashii"man -like like a manThe lexicon is the repository of the information about a language that isirregular and cannot be generated by rules. This is the information thatmust simply be listed. The Japanese lexicon includes the following:words or word stems:-- their meanings-- their case frames, if they are predicates-- what endings may attach to themword endings:-- how they change the meaning of what they attach to-- what words or word endings they may attach to-- what other word endings may attach to themSome of the tenses that have been accomodated by word endings listed in thelexicon are illustrated below, with the verb "ik-", ('to go'):ikui kimasuittaikimashitai kanaii kimasenikanakattaikitaiikitakunaiikitakattaikitakunakattaitte kudasaiikinasaiikanai de kudasaiikebaikanakerebaittaraikanakattaraitteikanai depresent-future informalpresent-future formalpast perfect informalpast perfect formalpresent-future informal negativepresent-future formal negativepast perfect informal negativepresent-future volitionalpresent-future volitional negativepast perfect volitionalpast perfect volitional negativepresent-future formal imperativepresent-future informal imperativepresent-future formal imperative negativeprovisionalprovisional negativeconditionalconditional negativegerundgerund negative- -I_________llqlLP_111__--- .l_lC----*-- --- L---· ---L----*-b(Pm*bCCICC^X·I-------^-··C- ----·II -----I-_itte iru present-future imperfect informalitte imasu present-future imperfect formalitte ita past-imperfect informalitte imashita past-imperfect formalitte inai present-future imperfect informal negativeitte inakatta past-imperfect informal negativeThe lexicon will continue to be expanded as more words and word endings areentered into it. It is expected that idioms will also eventually be includedin the lexicons. The goal of the project is to enter the equivalent of asmall dictionary into the lexicon of each language.3. The Coverage of the Rule SystemIn order to illustrate the coverage of the system a few sentences areincluded below. These sentences and constructions are a sample of those thatcan be understood by the rule system. The symbol represents dummyarguments. The English glosses are listed for convenience, and do notreflect the way the meanings of the Japanese sentences are represented in thesystem.tabemasu.' eats .Hanako ga tabemasu.'Hanako eats .'pan o tabemasu.' eats bread.'Hanako ga pan o tabemasu.'Hanako eats bread.'Hanako ga Taroo ni hon o agemasu.'Hanako gives a book to Taroo.'Taroo ni Hanako ga hon o agemasu.'Hanako gives a book to Taroo.'Hanako ga Taroo ni tegami o yuubin de okurimashita.'Hanako sent a letter to Taroo by mail.'- 9 -._ _ .. I--·------ .ILIOI**IIIC---------·II -1-*--13 -·"--·IIIPIIIIC---"-------"--··I·--·kesa soto e detara ame ga furimashita.'This morning, when went outside it rained.'ashita hayaku ikimasu.'Tomorrow will go early.'gakusei ga sannin kimasu.'Three students will cme.'Osaka kara kimasu.' will come from Osaka.'Taroo ga Osaka ni itta to iimashita.' said that Taroo went to Osaka.'Hanako ga Taroo ni watashita hon o yomimashita.' read the book that Hanako handed to Taroo.'Hanako ga uchi e kaetta koto...'The fact that Hanako went home...'Taroo ga Tbkyo ni itta toki, yuki ga futte imashita.'When Taroo went to Tobkyo, it was snowing.'Taroo ga Tokyo ni ikeba, Asakusa ni ikimasu.'If Taroo goes to Tokyo, will go to Asakusa.'Taroo ga kita ka wakarimasu ka?'Do you know whether Taroo came?'Taroo ga doko ni ikimashita ka?'Where did Taroo go?'nani o tabemasu ka?'What are/is eating?'imoto wa ikutsu desu ka?'How old is sister?'naze kaerimasu ka?'Why are/is going home?'Hanako ga dare ni nani o agemasu ka?'What will Hanako give to whom?'Taroo wa gakusei desu.'Taroo is a student.'Kyoto no natsu ga attakai desu.'The Kyoto summers are hot.'- 10 -_ 1IC 1 -- IC------hon desu.'It's (a) book(s).'ii desu.'It's alright.'yasui hon...'(an) inexpensive book(s)...'hon to zasshi...'(a) book(s) and (a) magazine(s)...'kuruma ya jitensha...'automobile(s), bicycle(s) and so on...'gakusei igai...'other than students...'futsuka inai...'within two days...'sanjyuu meetoru hodo...'about thirty meters...'binboo na gakusei to kanemochi-no roojin igai ni wa...'as for those besides poor students and rich old folks... ''The completed system will have the ability to correct some errors made bystudents. In most of the languages error recovery will be applied tosubject--verb agreement in number, gender and person, but in Japanese, errorrecovery will be performed for number--counter agreement. The rules requireChinese-root numbers to be followed by counters, and also check that thecounters agree in class with their associated nouns. This will allow thesystem to recognize that the following are mistakes, and to correct thestudent:*hon sannin'books -- three people'*otoko sansatsu'men -- three volumes'- 11 -__ _ ______CIF_·ICI_I·_ ----IIII IIIIIIIIII-X---C. --·---C I-4. ConclusionSome problems particular to writing parsing rules for Japanese have beendiscussed above. With the addition of the non-Indo-European language ofJapanese to the Athena Language Project, the language processing systems usedby the project have been stretched at times. The solution of problems posedby Japanese has resulted in a more flexible system with wider applicabilityto a greater range of languages. And finally, students of Japanese at MITwill be provided with the state-of-the-art language learning aids projectedfor students of the other languages traditionally taught at the Institute.- 12 ----- ··--- ----e ---- -- -------- ---11-clL------ACKNCW LEDGEMENTSI am grateful to Professor Michio Tsutsui for his helpful cmments anddiscussion on issues of Japanese grammar; to Janet Hirata for discussionabout linguistic and computational issues regarding the rule system; and toSue Felshin and Dr. Janet Murray of the Athena Project for their continuedgeneral support. In addition I would like to thank Dr. Richard Samuels andthe MIT-Japan Science and Technology Program for funding this work.- 13 --'-"-I-^ ----- ---- ----------------

A Context Free Rule System For Parsing Japanese

http://dspace.mit.edu/bitstream/1721.1/7579/1/JP-WP-86-01.pdf

A Context Free Rule System For Parsing Japanese

Abstract

Similar works

Full text

Available Versions

DSpace@MIT