How to Put Usability into Focus: Using Focus Groups to Evaluate the Usability of Interactive Theorem Provers

In recent years the effectiveness of interactive theorem provers has increased to an extent that the bottleneck in the interactive process shifted to efficiency: while in principle large and complex theorems are provable (effectiveness), it takes a lot of effort for the user interacting with the system (lack of efficiency). We conducted focus groups to evaluate the usability of Isabelle/HOL and the KeY system with two goals: (a) detect usability issues in the interaction between interactive theorem provers and their user, and (b) analyze how evaluation and survey methods commonly used in the area of human-computer interaction, such as focus groups and co-operative evaluation, are applicable to the specific field of interactive theorem proving (ITP). In this paper, we report on our experience using the evaluation method focus groups and how we adapted this method to ITP. We describe our results and conclusions mainly on the ``meta-level,'' i.e., we focus on the impact that specific characteristics of ITPs have on the setup and the results of focus groups. On the concrete level, we briefly summarise insights into the usability of the ITPs used in our case study

Pseudo-contractions as Gentle Repairs

Updating a knowledge base to remove an unwanted consequence is a challenging task. Some of the original sentences must be either deleted or weakened in such a way that the sentence to be removed is no longer entailed by the resulting set. On the other hand, it is desirable that the existing knowledge be preserved as much as possible, minimising the loss of information. Several approaches to this problem can be found in the literature. In particular, when the knowledge is represented by an ontology, two different families of frameworks have been developed in the literature in the past decades with numerous ideas in common but with little interaction between the communities: applications of AGM-like Belief Change and justification-based Ontology Repair. In this paper, we investigate the relationship between pseudo-contraction operations and gentle repairs. Both aim to avoid the complete deletion of sentences when replacing them with weaker versions is enough to prevent the entailment of the unwanted formula. We show the correspondence between concepts on both sides and investigate under which conditions they are equivalent. Furthermore, we propose a unified notation for the two approaches, which might contribute to the integration of the two areas

Users' efficiency, effectiveness, cognitive style and emotional reactions to computer interface based on different data model

Informaciona usklađenost kompjuterskog sistema i korisnika, kao predmet kognitivne ergonomije, počiva na dobro osmišljenom korisničkom interfejsu koji strukturom i informacionim obimom treba da omogući optimalnu interakciju u složenom sistemu čovek‐računar. U osnovi vizuelne forme interfejsa stoji, pored ostalog, logika po kojoj su podaci raspoređeni, međusobno povezani i dostupni za upotrebu, a koja se definiše modelom podataka. Smatra se da je postizanje veće kompatibilnosti između predstave koju korisnik ima o sistemu i načina na koji sistem zaista funkcioniše ključ za efikasno i efektivno korišćenje interaktivnih sistema. U tom smislu, podaci koji se u bazi mogu predstaviti u obliku zvezdaste sheme (dimenziono modelovani podaci), gde su veze između objekata i hijerarhijski odnosi načelno dati u okviru jednog objekta, i podaci koji imaju strukturu u kojoj je svaki hijerarhijski nivo predstavljen posebnim objektom (transakciono modelovani podaci), za individualne korisnike mogu biti u različitoj meri upotrebljivi. Brzina i tačnost obavljanja poslovnih zahteva putem dimenziono i transakciono modelovanih podataka može zavisiti, kako od vrste i složenosti zadatka, tako i od psiholoških osobenosti korisnika. Kognitivni stil, specifičan način kognitivnog funkcionisanja, izdvojio se kao relevantan faktor. S druge strane, isticanje sve većeg značaja afektivne komponente procene sistema i veze sa objektivnim merama upotrebljivosti, uticalo je na to da se u eksperimentalni nacrt uvrste i dimenzije emocionalnih reakcija na izgled sistema. U istraživanju su se, u dva modela podataka, ispitivale brzina i tačnost rada ispitanika sa u različitom stepenu izraženim metaforičnim, racionalnim i empiričnim kognitivnim stilom, a s obzirom na izazvane emocionalne reakcije pobuđenosti, prijatnosti i dominantnosti. U eksperimentu sa ponovljenim nacrtom učestvovalo je 303 ispitanika, slučajno raspoređenih s obzirom na redosled izlaganja modela podataka, i ujednačenih po karakteristikama relevantnim za Mr Ivana Kovačević – DOKTORSKA DISERTACIJA 2 istraživanje. Rezultati govore u prilog bolje upotrebljivosti dimenzionog modela u kom ispitanici brže obavljaju zadatke. Brzina i tačnost rada nisu u očekivanoj korelaciji i nejednako variraju s obzirom na različite prediktore, te ih je smisleno odvojeno posmatrati. Manipulativna varijabla redosleda izlaganja modela se pokazala najznačajnijim prediktorom brzine rada. Ispitanici postižu mnogo bolje rezultate ukoliko prvo koriste dimenzioni model, kao pojavno manje složen, nego u situaciji kada je transakcioni model prvo korišćen. Ovaj nalaz govori u prilog tome da je brzina rada zavisna od specifičnog redosleda iskustva. U skladu sa tim su i rezultati koji pokazuju da kognitivni stil nema naročitog efekta na brzinu rada, za razliku od emocionalnih reakcija prijatnosti i kontrole koje su negativno povezane sa potrebnim vremenom da se zadaci obave. S druge strane, tačnost je relativno nezavisna od redosleda izlaganja, i konzistentna karakteristika u oba modela. Ispitanici koji teže tačnosti, to čine bez obzira na prethodna ili trenutna iskustva, stoga se čini da je tačnost rada inherentna karakteristika korisnika. Tako su nađene i veze metaforičnosti, racionalnosti i empiričnosti sa tačnošću rada. Tačnost opada pri prelasku sa transakcionog na dimenzioni model podataka, u funkciji porasta metaforičnosti (moguć efekat zamora), dok se prelaskom sa dimenzionog na transakcioni, povećava u transakcionom u funkciji povećanja skora na racionalnosti (moguć efekat uvežbanosti). Empiričnost je povezana sa slabijom tačnošću rada u transakcionom modelu, kada je on prvi model u kom se radi (moguć efekat neuvežbanosti). Pojednostavljeno rečeno, ukoliko težimo efikasnosti (većoj brzini rada), insistiraćemo na jednostavnijim ili pak postupno usložnjavajućim interfejsima. Ukoliko težimo efektivnosti (većoj tačnosti rada), rešenja ćemo tražiti u personalizaciji korisničkih interfejsa.Informational compatibility of computer system and user, as the subject of cognitive ergonomics, depends on the characteristics of user interface. Well conceived user interface, with its structure and amount of information, provides optimal interaction in complex computer‐user system. Visual form of interface is based on logics in which data are displayed (arranged), interconnected and available for use, defined by data models. It is assumed that achieving higher compatibility between user perception of the system and the manner in which it really operates is the key for effectiveness and efficacy in using interactive systems. There are at least two different data base arrangements with different usability level for individual users. Dimensionally modeled data are arranged in a shape of star schema, where all relations between objects and all hierarchical associations are given within one object. Transactionallly modeled data have structure within which every hierarchical level is represented with particular object. Speed and accuracy of performing tasks by using dimensionally or transactionally modeled data are dependent on type and complexity of tasks, as well as on psychological characteristics of users. Cognitive style, as the specific manner of cognitive functioning, and dimensions of emotional reactions on system appearance, are seen as relevant factors. Research consisted of experimentally measuring the speed and accuracy of performing tasks in two data environments (dimensional and transactional data models) by participants with different levels of metaphorical, rational and empirical cognitive styles. Emotional reactions of arousal, pleasure and dominance provoked during the experience of using data models were assessed as well. In this experiment 303 participants, balanced by relevant characteristics, were randomly assigned in groups according to the order of exposing to data models. Results proved better overall performance of dimensionally modeled data Mr Ivana Kovačević – DOKTORSKA DISERTACIJA 4 which is seen as a data model with higher usability. Speed and accuracy of performance are shown not to be in expected correlation, varying differently according to the different predictors. Therefore, it was concluded it is plausible to consider them as separate entities. Manipulative variable, the order of data model display, proved to be the most important predictor of speed performance. Participants achieved better results when using dimensionally modeled data first, with its appearance less complex, than in situation when the transactional model was primarily used. This result showed that performance speed was dependent on specific order of user experience. Accordingly, it was found that cognitive style had no particular effect on speed, while emotional reactions of pleasure and dominance were negatively related to the time required for task completion. On the other hand, accuracy was relatively independent of data model order and it is a consistent characteristic within subject in both models. Participants who tended to be accurate, did that regardless of their previous or current experience with data model, thus it appeared that performance accuracy was user`s inherent quality. Relations between accuracy and metaphorical, rational and empirical cognitive styles were also found. Accuracy of dimensionally modeled data performance was poorer when using the transactional model first, in function of higher level of metaphorical style (possible effect of fatigue). Conversely, when using the dimensional model first, accuracy is higher in the transactional model in function of increasing the score on rational dimension (possible effect of training). Empirical cognitive style is correlated with lower accuracy in the transactional model, when it was a the first model of choice (possible effect of lack of practice). Generally, abridged implications of findings were the following, if we strive to achieve efficacy (speed), we should insist on simplifying, or at least gradually building more complexity in user interfaces. Otherwise, if we are looking for effectiveness (accuracy), we should try to find a solution in the personalization of user interface

An analysis of errors in interactive proof attempts

The practical utility of interactive, user-guided, theorem proving depends on the design of good interaction environments, the study of which should be grounded in methods of research into human–computer interaction (HCI). This paper discusses the relevance of classifications of programming errors developed by the HCI community to the problem of interactive theorem prov-ing. A new taxonomy of errors is proposed for interaction with theorem provers and its adequacy as