Spanish translations of Cochrane plain language summaries: assessing the impact of a controlled language checker on machine translation quality

Background Cochrane aims to make high-quality health information accessible for users worldwide (Chandler et al. 2017). To this end, contributors simplify Systematic Reviews into Plain Language Summaries (PLS), which are translated (Cochrane Community 2016). Simplification and translation may represent onerous and time-consuming tasks when conducted without assistance from technology. Contributors produce PLS by manually implementing a written set of guidelines (Cochrane Methods 2013) and translations are mainly produced by volunteers (Cochrane Community 2016). Cochrane is considering using machine translation (MT) systems and controlled language (CL) checkers (Birch 2017; Von Elm et al. 2013). A CL is a set of rules adopted to make a text more comprehensible and translatable. A CL checker is the software that checks for adherence to those rules (O’Brien 2010). The adoption of a CL at the authoring stage can increase machine translatability and reduce the effort required to correct the MT outputs (Allen 2003). Objectives This study investigated if introducing a CL checker called Acrolinx in the authoring process of PLS increases the quality of the Spanish translations of PLS produced by Google Translate, a freely available MT system. Method and Materials The experimental materials were: a corpus of 12 PLS authored by manually implementing Cochrane guidelines and their Spanish MT translations; and a corpus of 12 PLS authored with assistance from Acrolinx and their Spanish MT translations. 41 Spanish native speakers conducted the MT evaluation task. Each evaluator was given two PLS (one per corpus) and their Spanish MT outputs. Evaluators assigned scores to the Spanish outputs on fluency and adequacy, commonly employed in MT quality evaluation (Castilho et al. 2017). Results and Conclusions There was little difference in fluency and adequacy scores between the two corpora of PLS, possibly because neither approach to simplification had been developed with MT quality in mind. Moreover, fluency and adequacy scores were relatively high, suggesting that the MT system produced reasonably good translations. While human validation/correction of the MT output is needed for accuracy, the high adequacy and fluency scores for Spanish indicate that integrating MT in the translation workflow might be a viable option. Further studies should be conducted on other language pairs

Simplifying, reading, and machine translating health content: an empirical investigation of usability

Text simplification, through plain language (PL) or controlled language (CL), is adopted to increase readability, comprehension and machine translatability of (health) content. Cochrane is a non-profit organisation where volunteer authors summarise and simplify health-related English texts on the impact of treatments and interventions into plain language summaries (PLS), which are then disseminated online to the lay audience and translated. Cochrane’s simplification approach is non-automated, and involves the manual checking and implementation of different sets of PL guidelines, which can be an unsatisfactory, challenging and time-consuming task. This thesis examined if using the Acrolinx CL checker to automatically and consistently check PLS for readability and translatability issues would increase the usability of Cochrane’s simplification approach and, more precisely: (i) authors’ satisfaction; and (ii) authors’ effectiveness in terms of readability, comprehensibility, and machine translatability into Spanish. Data on satisfaction were collected from twelve Cochrane authors by means of the System Usability Scale and follow-up preference questions. Readability was analysed through the computational tool Coh-Metrix. Evidence on comprehensibility was gathered through ratings and recall protocols produced by lay readers, both native and non-native speakers of English. Machine translatability was assessed in terms of adequacy and fluency with forty-one Cochrane contributors, all native speakers of Spanish. Authors seemed to welcome the introduction of Acrolinx, and the adoption of this CL checker reduced word length, sentence length, and syntactic complexity. No significant impact on comprehensibility and machine translatability was identified. We observed that reading skills and characteristics other than simplified language (e.g. formatting) might influence comprehension. Machine translation quality was relatively high, with mainly style issues. This thesis presented an environment that could boost volunteer authors’ satisfaction and foster their adoption of simple language. We also discussed strategies to increase the accessibility of online health content among lay readers with different skills and language backgrounds