A Framework for Exploring and Evaluating Mechanics in Human Computation Games

Human computation games (HCGs) are a crowdsourcing approach to solving computationally-intractable tasks using games. In this paper, we describe the need for generalizable HCG design knowledge that accommodates the needs of both players and tasks. We propose a formal representation of the mechanics in HCGs, providing a structural breakdown to visualize, compare, and explore the space of HCG mechanics. We present a methodology based on small-scale design experiments using fixed tasks while varying game elements to observe effects on both the player experience and the human computation task completion. Finally we discuss applications of our framework using comparisons of prior HCGs and recent design experiments. Ultimately, we wish to enable easier exploration and development of HCGs, helping these games provide meaningful player experiences while solving difficult problems.Comment: 11 pages, 5 figure

A Systematic Approach for Using DICOM Structured Reports in Clinical Processes: Focus on Breast Cancer

The final publication is available at Springer via http://dx.doi.org/10.1007/s10278-014-9728-6.This paper describes a methodology for redesigning the clinical processes to manage diagnosis, follow-up, and response to treatment episodes of breast cancer. This methodology includes three fundamental elements: (1) identification of similar and contrasting cases that may be of clinical relevance based upon a target study, (2) codification of reports with standard medical terminologies, and (3) linking and indexing the structured reports obtained with different techniques in a common system. The combination of these elements should lead to improvements in the clinical management of breast cancer patients. The motivation for this work is the adaptation of the clinical processes for breast cancer created by the Valencian Community health authorities to the new techniques available for data processing. To achieve this adaptation, it was necessary to design nine Digital Imaging and Communications in Medicine (DICOM) structured report templates: six diagnosis templates and three summary templates that combine reports from clinical episodes. A prototype system is also described that links the lesion to the reports. Preliminary tests of the prototype have shown that the interoperability among the report templates allows correlating parameters from different reports. Further work is in progress to improve the methodology in order that it can be applied to clinical practice.We thank the subject matter experts for sharing their insights through this study. We are especially appreciative of the efforts of the Radiology Unit and Medical Oncology Unit teams at the University Hospital Dr. Peset. This work was partially supported by the Vicerectorat d'Investigacio de la Universitat Politecnica de Valencia (UPVLC) to develop the project "Mejora del proceso diagnostico del cancer de mama" with reference UPV-FE-2013-8.Medina, R.; Torres Serrano, E.; Segrelles Quilis, JD.; Blanquer Espert, I.; Martí Bonmatí, L.; Almenar-Cubells, D. (2015). A Systematic Approach for Using DICOM Structured Reports in Clinical Processes: Focus on Breast Cancer. Journal of Digital Imaging. 28(2):132-145. doi:10.1007/s10278-014-9728-6S132145282Ratib O: Imaging informatics: From image management to image navigation. Yearb Med Inform 2009; 167–172Oakley J. Digital Imaging: A Primer for Radiographers, Radiologists and Health Care Professionals. Cambridge University Press, 2003.Prokosch HU, Dudeck J: Hospital information systems: Design and development characteristics, impact and future architecture. Elsevier health sciences, 1995Foster I, Kesselman C, Tuecke S. The anatomy of the grid: Enabling scalable virtual organizations. Int J High Perform Comput Appl 2001; 15(3):200–222.Oram A: Peer-to-Peer: Harnessing the power of disruptive technologies. O’Reilly Media, 2001National Institute of Standards and Technology. The NIST Definition of Cloud Computing. 2011. http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf (accessed 29 Jan 2013)Oster S, Langella S, Hastings S, Ervin D, Madduri R, Phillips J, Kurc T, Siebenlist F, Covitz P, Shanbhag K, Foster I, Saltz J. caGrid 1.0: An enterprise grid infrastructure for biomedical research. J Am Med Inform Assoc 2008; 15:138–149.Natter MD, Quan J, Ortiz DM, et al. An i2b2-based, generalizable, open source, self-scaling chronic disease registry. J Am Med Inform Assoc 2013; 20:172–179.Ohno-Machado L, Bafna V, Boxwala AA, et al. iDASH: Integrating data for analysis, anonymization, and sharing. J Am Med Inform Assoc 2012; 19:196–201.Channin DS, Mongkolwat P, Kleper V, Rubin DL. Computing human image annotation. Conf Proc IEEE Eng Med Biol Soc 2009; 1:7065–8.Sittig DF, Wright A, Osheroff JA, et al. Grand challenges in clinical decision support. J Biomed Inform 2008; 41(2):387–392.Wagholikar KB, Sundararajan V, Deshpande AW. Modeling paradigms for medical diagnostic decision support: a survey and future directions. J Med Syst 2012; 36(5):3029–3049.Rubin DL. Creating and curating a terminology for radiology: Ontology modeling and analysis. J Digit Imaging 2008; 21(4):355–362.Kahn CE, Jr., Langlotz CP, Burnside ES, Carrino JA, Channin DS, Hovsepian DM, et al. Toward best practices in radiology reporting. Radiology 2009; 252(3):852–856.Taira PK, Soderlang SG, JAbovits RM. Automatic structuring of radiology free-text reports. Radiographics 2001; 21(1); 237–245.Fujii H, Yamagishi H, Ando Y, Tsukamoto N, Kawaguchi O, Kasamatsu T, et al. Structuring of free-text diagnostic report. Stud. Health Technol. Inform. 2007; 129: 669–673.Murff HJ, FitzHenry F, Matheny ME, Gentry N, Kotter KL, Crimin K, Dittus RS, Rosen AK, Elkin PL, Brown SH, Speroff T. Automated identification of postoperative complications within an electronic medical record using natural language processing. JAMA 2011; 306(8):848–855.Clunie DA: DICOM structured reporting. PixelMed Publishing, 2000D’Avolio LW, Nguyen TM, Farwell WR, Chen Y, Fitzmeyer F, Harris OM, Fiore LD. Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC). J Am Med Inform Assoc 2012; 17:375–382.Napel SA, Beaulieu CF, Redriguez C, Cui J, Xu J, Grupta A, et al. Automated retrieval of CT images of liver lesions on the basis of image similarity: Method and preliminary results. Radiology 2010; 256(1): 243–252.Langlotz CP. RadLex: A new method for indexing online educational materials. Radiographics 2006; 26(6):1595–1597.Crestania F, Vegas J, de la Fuente P. A graphical user interface for the retrieval of hierarchically structured documents. Inf Process Manag 2004; 40(2):269–289.Weiss DL, Langlotz CP. Structured reporting: Patient care enhancement or productivity nightmare? Radiology 2008. 249(3):739–747.Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19(3):413–422.Patrick R, Julien G, Christian L, Antoine G. Automatic medical encoding with SNOMED categories. BMC Med Inform Decis Mak 2008; 8(Suppl 1): S1–S6.Lopez-Garcia P, Boeker M, Illarramendi A, Schulz S. Usability-driven pruning of large ontologies: The case of SNOMED CT, J Am Med Inform Assoc 2012; 19:e102-e109.World Health Organization. International Statistical Classification of Diseases and Related Health Problems 10th Revision. http://apps.who.int/classifications/apps/icd/icd10online/ (accessed 29 Jan 2013)American College of Radiology (ACR) Breast Imaging Reporting and Data System Atlas (BI-RADS® Atlas)World Health Organization. International Classification of Diseases for Oncology, 3rd Edition (ICD-O-3). http://www.who.int/classifications/icd/adaptations/oncology/en/index.html (accessed 29 Jan 2013)Greene FL. TNM: Our language of cancer. CA Cancer J Clin 2004; 54(3):129–130.American Joint Committee of Cancer (AJCC). AJCC Cancer Staging Manual. Seventh Edition. Springer, 2010Hussein R, Engelmann U, Schroeter A, Meinzer HP. DICOM structured reporting: Part 1. Overview and characteristics, Radiographics 2004; 24(3):891–896.Sluis D, Lee KP, Mankovich N. DICOM SR - integrating structured data into clinical information systems. Medicamundi 2002; 46(2):31–36.Percha B, Nassif H, Lipson J, Burnside E, Rubin D. Automatic classification of mammography reports by BI-RADS breast tissue composition class. J Am Med Inform Assoc 2012; 19(5):913–916.Ciatto S, Houssami N, Apruzzese A, Bassetti E, Brancato B, Carozzi F, Catarzi S, Lamberini MP, Marcelli G, Pellizzoni R, Pesce B, Risso G, Russo F, Scorsolini A. Reader variability in reporting breast imaging according to BI-RADS assessment categories (the Florence experience). Breast 2006; 15(1):44–51.National Electrical Manufacturers Association (NEMA). Digital Imaging and Communications in Medicine (DICOM). Part 16: Content Mapping Resource. http://medical.nema.org/dicom/2004/04_16PU.PDF (accessed 29 Jan 2013)Dolin RH, Alschuler L, Boyer S, Beebe C, Behlen FM, Biron PV, Shvo AS. HL7 clinical document architecture, release 2. J Am Med Inform Assoc 2006; 13:30–39.Blanquer I, Hernández V, Meseguer JE, Segrelles D. Content-based organisation of virtual repositories of DICOM objects. Future Gener Comput Syst 2009; 25(6):627–637.Blanquer I, Hernández V, Segrelles D, Torres E. Enhancing privacy and authorization control scalability in the grid through ontologies. IEEE Trans Inf Technol Biomed 2009; 12(1):16–24.Salavert J, Maestre C, Segrelles D, Blanquer I, Hernández V, Medina R, Martí L: Grid prototype to support cancer of breast diagnostics in clinic practice. Proc of the 4th. Iberian Grid Infrastructure Conf. Netbiblo, 2010Segrelles D, Franco JM, Medina R, Blanquer I, Salavert J, Hernandez V, Martí L, Díaz G, Ramos R, Guevara MA, González N, Loureiro J, Ramos I. Exchanging data for breast cancer diagnosis on heterogeneous grid platforms. Computing and Informatics 2012; 31(1):3–15.Ali MS, Consens M, Lalmas M. Extended structural relevance framework: A framework for evaluating structured document retrieval. Inf Retrieval 2012; 15:558–590.Welter P, Riesmeier J, Fischer B, Grouls C, Kuhl C, Deserno, TM. Bridging the integration gap between imaging and information systems: A uniform data concept for content-based image retrieval in computer-aided diagnosis. J Am Med Inform Assoc 2011; 18:506–510.Jenkins CW. Application prototyping: A case study. Perform Eval Rev 1981; 10(1):21–27.Generalitat Valenciana. Conselleria de Sanitat. Oncoguía de Cáncer de Mama Comunidad Valenciana. http://publicaciones.san.gva.es/publicaciones/documentos/V.2478-2006.pdf (accessed 29 Jan 2013)Maestre C, Segrelles-Quilis JD, Torres E, Blanquer I, Medina R, Hernández V, Martí L. Assessing the usability of a science gateway for medical knowledge bases with TRENCADIS. J Grid Computing 2012; 10:665–688.Lewis J. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int J Hum-Comput Interact 1995; 7(1):57–78.Lewis JR. Psychometric evaluation of the PSSUQ using data from five years of usability studies. Int J Hum-Comput Interact 2002; 14(3–4):463–488.Shapiro SS, Wilk MB. An analysis of variance test for normality (complete samples). Biometrika 1965; 52(3–4):591–611.Chhatwal J, Alagoz O, Lindstrom MJ, Kahn Jr CE, Shaffer KA, Burnside ES. A logistic regression model based on the national mammography database format to aid breast cancer diagnosis. AJR Am J Roentgenol 2009; 192:1117–1127

Comparative Usability Study of Two Space Logistics Analysis Tools

Future space exploration missions and campaigns will require sophisticated tools to help plan and analyze logistics. To encourage their use, space logistics tools must be usable: a design concept encompassing terms such as efficiency, effectiveness, and satisfaction. This paper presents a usability study of two such tools: SpaceNet, a discrete event simulation tool and a comparable spreadsheet-based tool. The study follows a randomized orthogonal design having within-subjects evaluation of the two tools with 12 volunteer subjects (eight subjects with space backgrounds, four without). Each subject completed two sessions of testing, each with a 30-45 minute tutorial and a two-part space exploration scenario. The first part tests the creation a model to verify a simple uncrewed mission to lunar orbit. The second part tests the evaluation of an existing model to improve the effectiveness of a crewed mission to the lunar surface. The subjects completed a questionnaire after each session and a semi-structured interview following the second session. The study results indicate that the SpaceNet tool is more efficient for portions of the model creation task including modeling multi-burn transports and the spreadsheet tool is more effective for the model evaluation task. Qualitative evaluation indicates subjects liked the graphical nature and error-detection of the SpaceNet tool, but felt it took too long to edit information and appeared as a “black box.” Subjects liked the ability to view the entire model state within the spreadsheet tool, however were concerned with limited dynamic state feedback and underlying modeling assumptions. Future tools should combine the best features, including allowing modification of the entire model from a single interface, providing visibility of underlying logic, and integrated graphical and error-checking feedback.United States. Dept. of DefenseUnited States. Air Force Office of Scientific ResearchAmerican Society for Engineering Education. National Defense Science and Engineering Graduate Fellowship32 CFR 168aSamsung Fellowshi

Eye-tracking in Translation and Interpreting Studies: The growing popularity and methodological problems

The emerging eye-tracking technique has opened a window of opportunities not only in medical research but also in Translation and Interpreting Studies. In recent years this research method has been used to trace the processes of reading, translation and interpreting. Eye-tracking has recently become a popular technique to examine cognitive effort involved in written translation, audiovisual translation and conference interpreting. Thanks to the use of an eye-tracker one is able to investigate the whole process and not limit oneself to analysing the quality of the output. To be more precise, by means of eye-tracking experimenters may investigate moment-by-moment changes in the cognitive effort necessary to perform a given translation/interpreting task. Useful as the eye-tracking technique may be, researchers must often face methodological and apparatus-related challenges. The present paper is intended to discuss the eye-tracking methodology and then to address the potential problems of applying this method to investigate the processes of translation and interpreting. Among the notions to be discussed are: the types of eye-trackers and their usability, accuracy vs. ecological validity, accommodation (O'Brien 2010), sampling, the use of inferential statistics for small experimental groups as well as ethics. I will also refer to my own research on the notion of language-pair specificity in sight translation (Korpal 2012) as well as a collaborative work on numerical data processing in simultaneous interpreting (Korpal and Stachowiak, manuscript in preparation)