2,316 research outputs found

    Optimising content clarity for human-machine systems

    Get PDF
    This paper details issues associated with the production of clearly expressed and comprehensible technical documentation for domestic appliances and human-machine systems, and describes an approach to optimising the clarity of such content. The aim is to develop support for authors in checking the likely comprehensibility of chosen forms of expression by reference to an external measure of 'likely familiarity'. Our DOcumentation Support Tool (DoST) will assist in identifying words and expression forms that are likely to be unfamiliar to end users

    Investigating the effects of controlled language on the reading and comprehension of machine translated texts: A mixed-methods approach

    Get PDF
    This study investigates whether the use of controlled language (CL) improves the readability and comprehension of technical support documentation produced by a statistical machine translation system. Readability is operationalised here as the extent to which a text can be easily read in terms of formal linguistic elements; while comprehensibility is defined as how easily a text’s content can be understood by the reader. A biphasic mixed-methods triangulation approach is taken, in which a number of quantitative and qualitative evaluation methods are combined. These include: eye tracking, automatic evaluation metrics (AEMs), retrospective interviews, human evaluations, memory recall testing, and readability indices. A further aim of the research is to investigate what, if any, correlations exist between the various metrics used, and to explore the cognitive framework of the evaluation process. The research finds that the use of CL input results in significantly higher scores for items recalled by participants, and for several of the eye tracking metrics: fixation count, fixation length, and regressions. However, the findings show slight insignificant increases for readability indices and human evaluations, and slight insignificant decreases for AEMs. Several significant correlations between the above metrics are identified as well as predictors of readability and comprehensibility

    Controlled language and readability

    Get PDF
    Controlled Language (CL) rules specify constraints on lexicon, grammar and style with the objective of improving text translatability, comprehensibility, readability and usability. A significant body of research exists demonstrating the positive effects CL rules can have on machine translation quality (e.g. Mitamura and Nyberg 1995; Kamprath et al 1998; Bernth 1999; Nyberg et al 2003), acceptability (Roturier 2006), and post-editing effort (O’Brien 2006). Since CL rules aim to reduce complexity and ambiguity, claims have been made that they consequently improve the readability of text (e.g., Spaggiari, Beaujard and Cannesson 2003; Reuther 2003). Little work, however, has been done on the effects of CL on readability. This paper represents an attempt to investigate the relationship in an empirical manner using both qualitative and quantitative methods

    Do software models based on the UML aid in source-code comprehensibility? Aggregating evidence from 12 controlled experiments

    Get PDF
    In this paper, we present the results of long-term research conducted in order to study the contribution made by software models based on the Unified Modeling Language (UML) to the comprehensibility of Java source-code deprived of comments. We have conducted 12 controlled experiments in different experimental contexts and on different sites with participants with different levels of expertise (i.e., Bachelor’s, Master’s, and PhD students and software practitioners from Italy and Spain). A total of 333 observations were obtained from these experiments. The UML models in our experiments were those produced in the analysis and design phases. The models produced in the analysis phase were created with the objective of abstracting the environment in which the software will work (i.e., the problem domain), while those produced in the design phase were created with the goal of abstracting implementation aspects of the software (i.e., the solution/application domain). Source-code comprehensibility was assessed with regard to correctness of understanding, time taken to accomplish the comprehension tasks, and efficiency as regards accomplishing those tasks. In order to study the global effect of UML models on source-code comprehensibility, we aggregated results from the individual experiments using a meta-analysis. We made every effort to account for the heterogeneity of our experiments when aggregating the results obtained from them. The overall results suggest that the use of UML models affects the comprehensibility of source-code, when it is deprived of comments. Indeed, models produced in the analysis phase might reduce source-code comprehensibility, while increasing the time taken to complete comprehension tasks. That is, browsing source code and this kind of models together negatively impacts on the time taken to complete comprehension tasks without having a positive effect on the comprehensibility of source code. One plausible justification for this is that the UML models produced in the analysis phase focus on the problem domain. That is, models produced in the analysis phase say nothing about source code and there should be no expectation that they would, in any way, be beneficial to comprehensibility. On the other hand, UML models produced in the design phase improve source-code comprehensibility. One possible justification for this result is that models produced in the design phase are more focused on implementation details. Therefore, although the participants had more material to read and browse, this additional effort was paid back in the form of an improved comprehension of source code

    Eye tracking as an MT evaluation technique

    Get PDF
    Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output

    Seeing through the eyes of the respondent: an eye-tracking study on survey question comprehension

    Full text link
    "To ensure that the data obtained through surveys are reliable and lead to valid conclusions, respondents must comprehend the questions as intended by the survey designer and find it easy to answer them accurately. Applying a psycholinguistic perspective to survey question design, Lenzner, Kaczmirek, and Lenzner (2010) identified seven text features that undermine reading comprehension and thus increase the cognitive burden imposed by survey questions. In this study, we extend the earlier findings by Lenzner et al. (2010) in two ways. First, we use eye tracking as a more direct method to examine whether comprehension is indeed impeded by these text features. While response time is a valuable indicator of the overall cognitive effort required to answer a survey question, it does not enable us to distinguish between the time required to read and understand a question (comprehension stage) and the time it takes to provide an answer (including retrieval, judgment, and response selection). In contrast, recording respondents' eye movements while answering a Web survey allows us to identify the specific parts of the question they struggle with during the comprehension stage." (author's abstract

    Readability as a basis for information security policy assessment

    Get PDF
    Most organisations now impose information security policies (ISPs) or 'conditions of use' agreements upon their employees. The need to ensure that employees are informed and aware of their obligations toward information security is apparent. Less apparent is the correlation between the provision of such policies and their compliance. In this paper, we report our research into the factors that determine the efficacy of information security policies (ISPs). Policies should comprise rules or principles that users can easily understand and follow. Presently, there is no ready mechanism for estimating the likely efficacy of such policies across an organisation. One factor that has a plausible impact upon the comprehensibility of policies is their readability. The present study investigates the effectiveness of applying readability metrics as an indicator of policy comprehensibility. Results from a preliminary study reveal variations in the comprehension test results attributable to the difficulty of the examined policies. The pilot study shows some correlation between the software readability formula results and human comprehension test results and supports our view that readability has an impact upon understanding ISPs. These findings have important implications for users’ compliance with information security policies and suggest that the application of suitably selected readability metrics may allow policy designers to evaluate their draft policies for ease of comprehension prior to policy release. Indeed, there may be grounds for a readability compliance test that future ISPs must satisfy

    GRASP/Ada 95: Reverse Engineering Tools for Ada

    Get PDF
    The GRASP/Ada project (Graphical Representations of Algorithms, Structures, and Processes for Ada) has successfully created and prototyped an algorithmic level graphical representation for Ada software, the Control Structure Diagram (CSD), and a new visualization for a fine-grained complexity metric called the Complexity Profile Graph (CPG). By synchronizing the CSD and the CPG, the CSD view of control structure, nesting, and source code is directly linked to the corresponding visualization of statement level complexity in the CPG. GRASP has been integrated with GNAT, the GNU Ada 95 Translator to provide a comprehensive graphical user interface and development environment for Ada 95. The user may view, edit, print, and compile source code as a CSD with no discernible addition to storage or computational overhead. The primary impetus for creation of the CSD was to improve the comprehension efficiency of Ada software and, as a result, improve reliability and reduce costs. The emphasis has been on the automatic generation of the CSD from Ada 95 source code to support reverse engineering and maintenance. The CSD has the potential to replace traditional prettyprinted Ada source code. The current update has focused on the design and implementation of a new Motif compliant user interface, and a new CSD generator consisting of a tagger and renderer. The Complexity Profile Graph (CPG) is based on a set of functions that describes the context, content, and the scaling for complexity on a statement by statement basis. When combined graphicafly, the result is a composite profile of complexity for the program unit. Ongoing research includes the development and refinement of the associated functions, and the development of the CPG generator prototype. The current Version 5.0 prototype provides the capability for the user to generate CSDs and CPGs from Ada 95 source code in a reverse engineering as well as forward engineering mode with a level of flexibility suitable for practical application. This report provides an overview of the GRASP/Ada project with an emphasis on the current update

    Evaluating readability as a factor in information security policies

    Get PDF
    This thesis was previously held under moratorium from 26/11/19 to 26/11/21Policies should be treated as rules or principles that individuals can readily comprehend and follow as a pre-requisite to any organisational requirement to obey and enact regulations. This dissertation attempts to highlight one of the important factors to consider before issuing any policy that staff members are required to follow. Presently, there is no ready mechanism for estimating the likely efficacy of such policies across an organisation. One factor that has a plausible impact upon the comprehensibility of policies is their readability. Researchers have designed a number of software readability metrics that evaluate how difficult a passage is to comprehend; yet, little is known about the impact of readability on the interpretation of information security policies and whether analysis of readability may prove to be a useful insight. This thesis describes the first study to investigate the feasibility of applying readability metrics as an indicator of policy comprehensibility through a mixed methods approach, with the formulation and implementation of a seven phase sequential exploratory fully mixed methods design. Each one was established in light of the outcomes from the previous phase. The methodological approach of this research study is one of the distinguishing characteristics reported in the thesis, which was as follows: * eight policies were selected (from a combination of academia and industry sector institutes); * specialists were requested their insights on key policy elements; * focus group interviews were conducted; * comprehension tests were developed (Cloze tests); * a pilot study of comprehension tests was organised (preceded by a small-scale test); * a main study of comprehension tests was performed with 600 participants and reduce that for validation to 396; * a comparison was made of comprehension results against readability metrics. The results reveal that the traditional readability metrics are ineffective in predicting human estimation. Nevertheless, readability, as measured using a bespoke readability metric, may yield useful insight upon the likely difficulty that end-users may face in comprehending a written text. Thereby, our study aims to provide an effective approach to enhancing the comprehensibility of information security policies and afford a facility for future research in this area. The research contributes to our understanding of readability in general and offering an optimal technique to measure the readability in particular. We recommend immediate corrective actions to enhance the ease of comprehension for information security policies. In part, this may reduce instances where users avoid fully reading the information security policies, and may also increase the likelihood of user compliance. We suggest that the application of appropriately selected readability assessment may assist policy makers to test their draft policies for ease of comprehension before policy release. Indeed, there may be grounds for a readability compliance test that future information security policies must satisfy.Policies should be treated as rules or principles that individuals can readily comprehend and follow as a pre-requisite to any organisational requirement to obey and enact regulations. This dissertation attempts to highlight one of the important factors to consider before issuing any policy that staff members are required to follow. Presently, there is no ready mechanism for estimating the likely efficacy of such policies across an organisation. One factor that has a plausible impact upon the comprehensibility of policies is their readability. Researchers have designed a number of software readability metrics that evaluate how difficult a passage is to comprehend; yet, little is known about the impact of readability on the interpretation of information security policies and whether analysis of readability may prove to be a useful insight. This thesis describes the first study to investigate the feasibility of applying readability metrics as an indicator of policy comprehensibility through a mixed methods approach, with the formulation and implementation of a seven phase sequential exploratory fully mixed methods design. Each one was established in light of the outcomes from the previous phase. The methodological approach of this research study is one of the distinguishing characteristics reported in the thesis, which was as follows: * eight policies were selected (from a combination of academia and industry sector institutes); * specialists were requested their insights on key policy elements; * focus group interviews were conducted; * comprehension tests were developed (Cloze tests); * a pilot study of comprehension tests was organised (preceded by a small-scale test); * a main study of comprehension tests was performed with 600 participants and reduce that for validation to 396; * a comparison was made of comprehension results against readability metrics. The results reveal that the traditional readability metrics are ineffective in predicting human estimation. Nevertheless, readability, as measured using a bespoke readability metric, may yield useful insight upon the likely difficulty that end-users may face in comprehending a written text. Thereby, our study aims to provide an effective approach to enhancing the comprehensibility of information security policies and afford a facility for future research in this area. The research contributes to our understanding of readability in general and offering an optimal technique to measure the readability in particular. We recommend immediate corrective actions to enhance the ease of comprehension for information security policies. In part, this may reduce instances where users avoid fully reading the information security policies, and may also increase the likelihood of user compliance. We suggest that the application of appropriately selected readability assessment may assist policy makers to test their draft policies for ease of comprehension before policy release. Indeed, there may be grounds for a readability compliance test that future information security policies must satisfy

    Intralingual translation and cascading crises: evaluating the impact of semi-automation on the readability and comprehensibility of health content

    Get PDF
    During crises, intralingual translation (or simplification) of medical content can facilitate comprehension among lay readers and foster their compliance with instructions aimed to avoid or mitigate the cascading effects of crises. The onus of simplifying health-related texts often falls on medical experts, and the task of intralingual translation tends to be nonautomated. Medical authors are asked to check and remember different sets of plain language guidelines, while also relying on their interpretation of how and when to implement these guidelines. Accordingly, even simplified health-related texts present characteristics that make them difficult to read and comprehend, particularly for an audience with low (health) literacy. Against this background, this chapter describes an experimental study aimed at testing the impact that using a controlled language (CL) checker to semi-automate intralingual translation has on the readability and comprehensibility of medical content. The study focused on the plain language summaries and abstracts produced by the non-profit organisation Cochrane. Using Coh-Metrix and recall, this investigation found that the introduction of a CL checker influenced some readability features, but not lay readers’ comprehension, regardless of their native language. Finally, strategies to enhance the comprehensibility of health content and reduce the vulnerability of readers in crises are discussed
    • 

    corecore