Rossetti, Alessandra

English

DCU Online Research Access Service

COCHRANE PLAIN LANGUAGE SUMMARIES: A study of authors’ satisfaction and users’ comprehension INTERACT International Network on Crisis Translation Alessandra Rossetti Dublin City University alessandra.rossetti2@mail.dcu.ie   BACKGROUND AND MOTIVATION (1)  Cochrane: a not-for-profit organisation providing high-quality health information on the impact of treatments and surgical interventions by means of systematic reviews.    Plain language summaries (PLS) summarise and simplify systematic reviews for the lay public.    Volunteer authors manually checking and implementing different sets of simplification guidelines:  difficult and time-consuming task (Temnikova 2012; Aikawa et al. 2007).    Contradictions, inconsistencies and vagueness:  reduced readability of PLS (Karačić et al. 2017; Flodgren 2016).  2 BACKGROUND AND MOTIVATION (2)   Introducing technological support/assistance for authors might:  (i) increase their satisfaction;  (ii) ensure higher consistency and clarity of simplification rules, thus making texts more comprehensible (Leroy et al. 2013).    Acrolinx: a tool which ensures comprehensibility  by automatically checking texts against a set of simplification rules, e.g. on style, tone of voice, and terminology;  by proving suggestions and examples (Rodríguez Vázquez 2016).  3 RESEARCH QUESTIONS  Does integrating Acrolinx into Cochrane’s standard workflow of PLS production   increase Cochrane volunteer authors’ satisfaction?   facilitate reading comprehension of Cochrane PLS among lay public?  4 CHARACTERISTICS OF THE INTEGRATION  Acrolinx as a plugin in Microsoft Word        12 Cochrane authors asked to use Acrolinx  to check for readability/comprehensibility issues in their PLS (previously produced by following Cochrane guidelines);  to revise the PLS accordingly. 5 METHOD(S) AND PARTICIPANTS  Authors’ satisfaction: System Usability Scale (SUS) (Brooke 1996) and follow-up preference questions  10 statements, Likert scale;  the higher the SUS score, the higher the satisfaction.   12 Cochrane authors (health professionals)   Reading comprehension: ratings and (free and cued) recall (Crossley and McNamara 2016)  free recall: everything a reader can remember about a text;  cued recall: everything a reader can remember about  a specific theme/section of the text;  controlling for reading skills and prior knowledge.     59 native speakers of English and 23 non-native speakers of English (no health background)   Within-subjects design: pre-Acrolinx PLS, post-Acrolinx PLS and abstract (non-simplified summary/baseline) per reader 6 FINDINGS ON AUTHORS’ SATISFACTION (1)  On average, Cochrane authors were more satisfied with Acrolinx (M=75.41, SD=14.49) than with Cochrane sets of guidelines (M=62.29, SD=26.53).    The difference was not statistically significant, t(11)=1.25, p=0.23.    7 FINDINGS ON AUTHORS’ SATISFACTION (2)  Authors showed a preference for using Acrolinx in combination with Cochrane PLS guidelines in the future.         All authors recognised the need for support when simplifying content. 8 0 1 2 3 4 5 6 7 8 9 10I would use both Cochrane PLS guidance andAcrolinxI would use Acrolinx onlyI would use Cochrane PLS guidance onlyI would not use any authoring support materialOtherFuture use of authoring support FINDINGS ON READING COMPREHENSION (RATINGS) 9 051015202530StronglydisagreeSomewhatdisagreeNeitheragree nordisagreeSomewhatagreeStronglyagreeText easy to read? Abstracts Pre-Acrolinx PLS Post-Acrolinx PLSNative speakers 012345678910StronglydisagreeSomewhatdisagreeNeitheragree nordisagreeSomewhatagreeStronglyagreeText easy to read? Abstracts Pre-Acrolinx PLS Post-Acrolinx PLSNon-native speakers  In both groups, compared with pre-Acrolinx PLS, slightly higher number of participants who found post-Acrolinx PLS easy to read;   Most native and non-native speakers (strongly or somewhat) disagreed with the fact that abstracts were easy to read.  FINDINGS ON READING COMPREHENSION (FREE AND CUED RECALL) (1) 10 Free recall, native speakers of English (*) and (**): statistically significant differences (at 0.05 significance level), as shown by within-subjects ANOVAs and follow-up pairwise comparisons. Corpora Free recall score Mean (SD) Pre-Acrolinx PLS 13.39 (7.59) (*) Post-Acrolinx PLS 12.87 (7.28) (**) Abstracts (baseline) 9.08 (5.21) (*) (**) Cued recall, native speakers of English Corpora Cued recall score Mean (SD) Pre-Acrolinx PLS 14.9 (9.09) (*) Post-Acrolinx PLS 15.72 (12.58) (**) Abstracts (baseline) 29.77 (21.4) (*) (**) Free recall, non-native speakers of English Cued recall, non-native speakers of English Corpora Free recall score Mean (SD) Pre-Acrolinx PLS 8.36 (5.09) (*) Post-Acrolinx PLS 8.26 (5.38) (**) Abstracts (baseline) 5.26 (4.05) (*) (**) Corpora Cued recall score Mean (SD) Pre-Acrolinx PLS 6.3 (8.25) (*) Post-Acrolinx PLS 7.68 (9.4) (**) Abstracts (baseline) 26.45 (29.97) (*) (**) FINDINGS ON READING COMPREHENSION (FREE AND CUED RECALL) (2)  For both native and non-native speakers, the introduction of Acrolinx did not prove beneficial in terms of reading comprehension of PLS.    Comprehension of abstracts (as assessed via free recall) was significantly lower than comprehension of both corpora of PLS.   Comprehension of abstracts (as assessed via cued recall) was significantly higher than comprehension of PLS, possibly as a result of the following characteristics of abstracts:   increased use of bold headings  to signal specific themes/sections;  increased cohesion between headings and content of sections;  reduced length of sections. 11 FINDINGS ON READING COMPREHENSION (FREE AND CUED RECALL) (3)  For both the native and non-native sample, the within-subjects ANCOVAs on free and cued recall including the covariates (reading skills and prior knowledge*) were not significant.   Free recall, native speakers of English: F(1.638, 91.719)=1.263, p=0.283, ηp2=0.022.  Cued recall, native speakers of English: F(2, 114)=0.267, p=0.766, ηp2=0.005.  Free recall, non-native speakers of English: F(2, 40)=0.581, p=0.564, ηp2=0.028.  Cued recall, non-native speakers of English: F(2,40)=0.597, p=0.556, ηp2=0.029.   However, none of the two covariates had significant effects on free and cued recall scores (p>0.05).    *Prior knowledge excluded as a covariate from the ANCOVA on cued recall of native speakers because it did not meet the assumption of homogeneity of regression slopes. 12 IMPLICATIONS AND FUTURE WORK   Identifying a simplification scenario that boosts authors’ satisfaction might lead them to simplify more health content for the lay public.    Possible characteristics of the integration of Acrolinx:  Cochrane sets of guidelines to be used at the summarisation stage;  Acrolinx to be used at the simplification stage.    Need to test other authoring support tools and their impact on comprehension, to reduce the vulnerability of lay users of health content.    Need to further investigate the relative impact of reading skills and prior knowledge on comprehension.   Overall, simplification is beneficial in terms of comprehension, but other text characteristics can further enhance comprehensibility, e.g. formatting, layout, or content segmentation (Rusko et al. 2012; Tait et al. 2005; Frost et al. 1999).  13 THANK YOU FOR YOUR ATTENTION! Alessandra Rossetti Dublin City University alessandra.rossetti2@mail.dcu.ie   INTERACT International Network on Crisis Translation REFERENCES  Aikawa, T., Schwartz, L., King, R., Corston-Oliver, M. and Lozano, M. 2007. Impact of controlled language on translation quality and post-editing in a statistical machine translation environment. IN: Maegaard, B. (ed.) Proceedings of the Machine Translation Summit XI. Copenhagen, Denmark, 10-14 September. European Association for Machine Translation, pp. 1-7.   Brooke, J. 1996. SUS – A quick and dirty usability scale. Usability Evaluation in Industry, 89(194), pp. 4-7.   Crossley, S. A. and McNamara, D. S. 2016. Text-based recall and extra-textual generations resulting from simplified and authentic texts. Reading in a Foreign Language, 28(1), pp. 1-19.   Flodgren, G. 2016. Are Cochrane plain language summaries plain? IN: Abstracts of the 24th Cochrane Colloquium. Challenges to Evidence-Based Health Care and Cochrane. Seoul, Korea, 23-27 October. Available at: https://goo.gl/hk5HLG [Accessed 9 August 2018].   Frost, M. H., Thompson, R. and Thiemann, K. B. 1999. Importance of format and design in print patient information. Cancer Practice, 7(1), pp. 22-27.   Karačić, J., Buljan, I., Hren, D., Dondi, P. and Marušić, A. 2017. Readability of different formats of information about Cochrane systematic reviews: A cross sectional study. IN: Abstracts of the 9th Croatian Cochrane Symposium. Split, Croatia, 9-10 June. Available at: https://bit.ly/2vRRZTF [Accessed 9 August 2018].   Leroy, G., Kauchak, D. and Mouradi, O. 2013. A user-study measuring the effects of lexical simplification and coherence enhancement on perceived and actual text difficulty. International Journal of Medical Informatics, 82(8), pp. 717-730.   Rodríguez Vázquez, S. 2016. Assuring Accessibility during Web Localisation: An Empirical Investigation on the Achievement of Appropriate Text Alternatives for Images. PhD thesis. University of Geneva.   Rusko, E., Van der Waarde, K. and Heiniö, R. L. 2012. Challenges to read and understand information on pharmaceutical packages. IN: Singh, J. (ed.) Proceedings of the 18th IAPRI World Packaging Conference. Lancaster, Pennsylvania: DEStech Publication, pp. 79-85.   Tait, A. R., Voepel-Lewis, T., Malviya, S. and Philipson, S. J., 2005. Improving the readability and processability of a pediatric informed consent document: Effects on parents’ understanding. Archives of Pediatrics and Adolescent Medicine, 159(4), pp. 347-352.   Temnikova, I. 2012. Text Complexity and Text Simplification in the Crisis Management Domain. PhD thesis. University of Wolverhampton.  

Cochrane plain language summaries: A study of authors' satisfaction and users' comprehension

http://doras.dcu.ie/24406/1/Cochrane_plain_language_summaries_A_stud.pdf

Cochrane plain language summaries: A study of authors' satisfaction and users' comprehension

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service