This paper presents a general framework how controlled natural languages can
be evaluated and compared on the basis of user experiments. The subjects are
asked to classify given statements (in the language to be tested) as either
true or false with respect to a certain situation that is shown in a graphical
notation called "ontographs". A first experiment has been conducted that
applies this framework to the language Attempto Controlled English (ACE)

Kuhn, Tobias

English

arXiv

This paper presents a general framework how controlled natural languages can be evaluated and compared on the basis of user experiments. The subjects are asked to classify given statements (in the language to be tested) as either true or false with respect to a certain situation that is shown in a graphical notation called "ontographs". A first experiment has been conducted that applies this framework to the language Attempto Controlled English (ACE)

Kuhn, T

ZORA

University of ZurichZurich Open Repository and ArchiveWinterthurerstr. 190CH-8057 Zurichhttp://www.zora.uzh.chYear: 2009How to Evaluate Controlled Natural LanguagesKuhn, TKuhn, T (2009). How to Evaluate Controlled Natural Languages. In: Workshop on Controlled Natural Language(CNL 2009), Marettimo, Italy, 08 June 2009 - 10 June 2009.Postprint available at:http://www.zora.uzh.chPosted at the Zurich Open Repository and Archive, University of Zurich.http://www.zora.uzh.chOriginally published at:Workshop on Controlled Natural Language (CNL 2009), Marettimo, Italy, 08 June 2009 - 10 June 2009.Kuhn, T (2009). How to Evaluate Controlled Natural Languages. In: Workshop on Controlled Natural Language(CNL 2009), Marettimo, Italy, 08 June 2009 - 10 June 2009.Postprint available at:http://www.zora.uzh.chPosted at the Zurich Open Repository and Archive, University of Zurich.http://www.zora.uzh.chOriginally published at:Workshop on Controlled Natural Language (CNL 2009), Marettimo, Italy, 08 June 2009 - 10 June 2009.How to Evaluate Controlled Natural LanguagesAbstractThis paper presents a general framework how controlled natural languages can be evaluated andcompared on the basis of user experiments. The subjects are asked to classify given statements (in thelanguage to be tested) as either true or false with respect to a certain situation that is shown in agraphical notation called "ontographs". A first experiment has been conducted that applies thisframework to the language Attempto Controlled English (ACE).How to Evaluate Controlled Natural LanguagesTobias KuhnDepartment of Informatics, University of Zurich, Switzerlandtkuhn@ifi.uzh.chhttp://www.ifi.uzh.ch/cl/tkuhnAbstract. This paper presents a general framework how controlled nat-ural languages can be evaluated and compared on the basis of user ex-periments. The subjects are asked to classify given statements (in thelanguage to be tested) as either true or false with respect to a certainsituation that is shown in a graphical notation called “ontographs”. Afirst experiment has been conducted that applies this framework to thelanguage Attempto Controlled English (ACE).1 IntroductionControlled natural languages (CNL) are claimed to be easier to learn and un-derstand than other formal languages. User studies are the only way to verifythis claim. Many such studies have been conducted on tools that use some kindof CNL, e.g. the one described in [1]. However, it is hard to determine in thesecases how much the CNL contributes to the understandability and how muchis due to other aspects of the tool. Furthermore, it is hard to compare CNLsto other formal languages with such studies because different languages oftenrequire different tools. For those reasons, it would be very desirable to be ableto test a CNL in a tool-independent way.[3] presents one of the rare attempts to test a CNL (concretely the Rabbit lan-guage that is based on English) independently of a particular tool. The authorsconducted an experiment where the subjects were given one Rabbit statement ata time and they had to choose from four paraphrases in natural English only oneof which was correct. There are two problems with this approach. First, sincenatural language is highly ambiguous, we have to make sure somehow that thesubjects understand the natural language paraphrases in the right way, whichjust takes the same problem to the next level. Second, since the formal statementand the paraphrases look very similar in many cases (both rely on English), itis hard to determine whether understanding is actually necessary to fulfill thetask. The subjects might do the right thing without understanding the sentences(e.g. just by following some syntactic patterns), or by misunderstanding both —statement and paraphrase — in the same way.2 ApproachThe approach that we propose here solves the discussed problems and relies ona graphical notation which we call “ontographs”. Each ontograph consists of alegend that introduces types and relations and of a mini world that introducesindividuals, their types, and their relations. Individuals are represented by dif-ferent symbols depending on the types they belong to. Relations are representedby arrows that point from one individual to another. Figure 1 shows four suchontographs. Ontographs require that we have complete information about themini world and, as a consequence, do not need an explicit notation for negation.Everything that is not shown in the ontograph is not true.We assumed that ontographs are very easy to understand and our resultsconfirm this assumption. The use of intuitive graphical icons is one of the rea-sons of the good understandability. More important, however, is the fact thatthe ontograph notation has no generalization capabilities and does not supportpartial knowledge. This excludes many potential misunderstandings. An onto-graph explicitly shows every existing individual and depicts every single relationinstance between them. There is not much to misunderstand with these basic el-ements. While we can make general statements like “every man loves a woman”about the mini world defined by an ontograph, there is no way to express suchstatements in this general way in the ontograph notation itself.Ontographs are designed to be used in experiments to test the understand-ability of formal languages. Ontographs could also be used to test the writabilityby asking the subjects to describe the given situation. However, only the firstapproach has been investigated so far.In order to test the understandability of a language in an experiment, anontograph and several statements (written in the language to be tested) areshown to the subjects who have to decide which of the statements are true andwhich are false with respect to the mini world depicted by the ontograph.An important property of ontographs is that they use a graphical notationthat is syntactically very different from textual languages like CNLs. This makesit virtually impossible to accomplish a task like the one described above justby looking at the syntax. If subjects manage to systematically classify givenstatements correctly as true or false with respect to a certain ontograph then wecan conclude that the subjects understood the statements and the ontograph.Using the presented testing framework, we conducted a small experiment thattests the language Attempto Controlled English (ACE) [2] which is a mature con-trolled natural language. The goal was to find out how well the framework worksand at the same time how understandable ACE is. We recruited 15 subjects whowere not experts in knowledge representation.Four ontographs (see Figure 1) — each accompanied by 10 ACE statements— were shown to the subjects who had to classify each of the statements as true,false, or “don’t know”. There was a time limit of five minutes for each ontograph.There was no explanation how the ACE statements have to be interpreted withthe one exception that it was explained that “something” can stand for personsand objects.The ACE sentences are chosen in a way that they cover a broad variety ofsemantic structures. All sentences would be expressible in OWL and would covermost of the axiom types provided by the OWL standard. The first ontographMini World LegendpersonmanwomantravelerofficergolferJohnTomBillSueMaryLaraseespicturebuys$$$$$personmanwomanMaryLaraTomJohnpresentpersonlovesseestravelerofficeraquariumpresent$ buys$$$TomSueBillLaralovessees? asks? ??admireshelpspersonOntograph 11/1 – Mary is a traveler.1/2 + Bill is not a golfer.1/3 + Mary is an officer or is a woman.1/4 – Sue is an officer and is a traveler.1/5 – Every traveler is a man.1/6 + No golfer is a woman.1/7 + Every woman is an officer and every officer is a woman.1/8 – Every traveler who is not a woman is a golfer.1/9 + Every man is a golfer or is a traveler.1/10 + Every woman who is a golfer is a traveler.Ontograph 22/1 + John sees Tom.2/2 + Mary does not see Tom.2/3 – Tom buys a picture.2/4 – John sees no woman.2/5 + Tom sees every woman.2/6 + Tom sees nothing but women.2/7 + Lara buys nothing but presents.2/8 – No woman sees herself.2/9 + Every woman buys nothing but pictures.2/10 + No man who buys a picture is seen by a woman.Ontograph 33/1 + Everything that loves something is a person.3/2 – Everything that is loved by something is a person.3/3 + Everything that sees something is an officer or is a traveler.3/4 – Everything that is bought by something is an aquarium or is an officer.3/5 – Everything buys at most 1 thing.3/6 + Everything is bought by at most 1 thing.3/7 – Lara sees at least 2 persons.3/8 + Bill is seen by at most 1 traveler.3/9 – Every officer is loved by at least 2 persons.3/10 + Every officer sees exactly 1 traveler.Ontograph 44/1 + If X asks Y then Y asks X.4/2 + If X sees Y then Y does not see X.4/3 – Nothing asks itself.4/4 – If X loves something that loves Y then X loves Y.4/5 + If X admires Y then X sees Y.4/6 – If X helps Y then Y admires X.4/7 – If X admires Y then X does not see Y.4/8 + If X admires Y then Y does not see X.4/9 – If X admires Y then X sees Y. If X sees Y then X admires Y.4/10 + If X sees something that admires Y then X sees Y.Fig. 1. The four ontographs that have been used for the experiment together with therespective ACE statements. True statements are marked by +, false ones by –.1/11/21/31/41/51/61/71/81/91/10 2/12/22/32/42/52/62/72/82/92/10 3/13/23/33/43/53/63/73/83/93/10 4/14/24/34/44/54/64/74/84/94/100%20%40%60%80%100%time exceededdon't knowwrongcorrectstatementpercentage of subjectsFig. 2. This chart shows for each statement how many subjects classified it correctly,classified it wrongly, said that they don’t know, or exceeded the time limit.contains only individuals and types but no relations. The second ontograph in-troduces relations. The statements of the third ontograph use more complicatedstructures like domain, range, and cardinality restrictions. The statements of thefourth ontograph, finally, use no individuals and no types but talk only aboutrelations. In this way, we cover a subset of ACE that corresponds to a subset offirst-order logic that is similar to the one used by OWL.3 ResultsFigure 2 shows the result of the experiment. Overall, the decision rate (i.e.the percentage of subjects who classified a particular sentence as either true orfalse) was 93%, and on average each decision took 21 seconds. The decisionswere correct in 85% of the cases (compared to 50% which can be achieved bymere guessing). The chart shows that the incorrect decisions are not equallydistributed. In the case of five statements the incorrect behavior is predominant.The statements 1/3, 1/10, 2/7, and 2/9 represent special cases and for this reasonit is not unexpected that they are often classified wrongly.The statement 1/3 is a very simple statement using “or” that is true but wasoften classified as false. One could think that many subjects misinterpreted the“or” as being exclusive (instead of inclusive as ACE defines it). However, if thatis the case then the subjects should also interpret the “or” of statement 1/9 asexclusive, but they did not. A more plausible explanation is that the subjectsrecognized that the statement 1/3 is imprecise in the sense that using “and”instead of “or” would be more accurate. This prevented some of the subjectsfrom realizing that the statement is nevertheless true in a logical sense. Thingsare different with statement 1/9 where the replacement of “or” by “and” wouldnot be more accurate but would make the true statement false. As a result, 1/9was classified correctly by almost all subjects. It seems that people in such casesoften fail to distinguish accuracy from logical truth.The statement 1/10 is another special case. It is a conditional statement witha false precondition. It is not surprising that people with no background in logicsfail to classify this statement in a correct way.2/7 and 2/9, finally, are common mistakes that were also encountered in OWL:people tend to think that “nothing but presents” implies “at least 1 present” [4].We assume that these types of mistakes are not specific to ACE, but can beencountered in any other language. Disregarding those four special cases, we geta decision rate of 94% with 92% correct decisions.Since all three sentences containing “nothing but” were often misunderstood,we will investigate how this can be improved. Also the statement 3/6 requiresmore investigation.Apart from the five exceptional statements, the correct decision was chosen atleast twice as often than the wrong one. This preference is statistically significanton a 5% level for all statements except 4/7 and 4/10 (the null hypothesis beinga random 50% decision).Another interesting point is that we did not tell the subjects explicitly thatthe ontograph notation shows the complete information about the mini worldand that everything that is not shown in the ontograph can be considered false.The result for statement 2/2 shows that the subjects understood this very wellwithout any explanation. They understood that the fact that the ontograph doesnot show a sees-relation from Mary to Tom means that “Mary does not see Tom”is a true statement.4 ConclusionsWe presented a framework to test controlled natural languages relying on thegraphical notation of ontographs. We applied this framework to test ACE andcome to the conclusion that most ACE sentences are understood very well andvery quickly. These results and individual discussions with the subjects let usconclude that ontographs are a suitable and powerful concept for understandabil-ity evaluations of CNLs. A larger and more thorough experiment that comparesACE to another formal language is currently being set up.References1. Abraham Bernstein and Esther Kaufmann. GINO - A Guided Input Natural Lan-guage Ontology Editor. In International Semantic Web Conference, pages 144–157,2006.2. Norbert E. Fuchs, Kaarel Kaljurand, and Tobias Kuhn. Attempto Controlled En-glish for Knowledge Representation. In Reasoning Web, 4th International SummerSchool 2008, Tutorial Lectures, 2008.3. Glen Hart, Martina Johnson, and Catherine Dolbear. Rabbit: Developing a Con-trolled Natural Language for Authoring Ontologies. In ESWC 2008, 2008.4. Alan L. Rector, Nick Drummond, Matthew Horridge, Jeremy Rogers, HolgerKnublauch, Robert Stevens, Hai Wang, and Chris Wroe. OWL Pizzas: PracticalExperience of Teaching OWL-DL: Common Errors & Common Patterns. In EKAW2004, 2004.

How to Evaluate Controlled Natural Languages

Abstract

Similar works

Full text

Available Versions

ZORA