A Systematic Approach for Using DICOM Structured Reports in Clinical Processes: Focus on Breast Cancer

The final publication is available at Springer via http://dx.doi.org/10.1007/s10278-014-9728-6.This paper describes a methodology for redesigning the clinical processes to manage diagnosis, follow-up, and response to treatment episodes of breast cancer. This methodology includes three fundamental elements: (1) identification of similar and contrasting cases that may be of clinical relevance based upon a target study, (2) codification of reports with standard medical terminologies, and (3) linking and indexing the structured reports obtained with different techniques in a common system. The combination of these elements should lead to improvements in the clinical management of breast cancer patients. The motivation for this work is the adaptation of the clinical processes for breast cancer created by the Valencian Community health authorities to the new techniques available for data processing. To achieve this adaptation, it was necessary to design nine Digital Imaging and Communications in Medicine (DICOM) structured report templates: six diagnosis templates and three summary templates that combine reports from clinical episodes. A prototype system is also described that links the lesion to the reports. Preliminary tests of the prototype have shown that the interoperability among the report templates allows correlating parameters from different reports. Further work is in progress to improve the methodology in order that it can be applied to clinical practice.We thank the subject matter experts for sharing their insights through this study. We are especially appreciative of the efforts of the Radiology Unit and Medical Oncology Unit teams at the University Hospital Dr. Peset. This work was partially supported by the Vicerectorat d'Investigacio de la Universitat Politecnica de Valencia (UPVLC) to develop the project "Mejora del proceso diagnostico del cancer de mama" with reference UPV-FE-2013-8.Medina, R.; Torres Serrano, E.; Segrelles Quilis, JD.; Blanquer Espert, I.; Martí Bonmatí, L.; Almenar-Cubells, D. (2015). A Systematic Approach for Using DICOM Structured Reports in Clinical Processes: Focus on Breast Cancer. Journal of Digital Imaging. 28(2):132-145. doi:10.1007/s10278-014-9728-6S132145282Ratib O: Imaging informatics: From image management to image navigation. Yearb Med Inform 2009; 167–172Oakley J. Digital Imaging: A Primer for Radiographers, Radiologists and Health Care Professionals. Cambridge University Press, 2003.Prokosch HU, Dudeck J: Hospital information systems: Design and development characteristics, impact and future architecture. Elsevier health sciences, 1995Foster I, Kesselman C, Tuecke S. The anatomy of the grid: Enabling scalable virtual organizations. Int J High Perform Comput Appl 2001; 15(3):200–222.Oram A: Peer-to-Peer: Harnessing the power of disruptive technologies. O’Reilly Media, 2001National Institute of Standards and Technology. The NIST Definition of Cloud Computing. 2011. http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf (accessed 29 Jan 2013)Oster S, Langella S, Hastings S, Ervin D, Madduri R, Phillips J, Kurc T, Siebenlist F, Covitz P, Shanbhag K, Foster I, Saltz J. caGrid 1.0: An enterprise grid infrastructure for biomedical research. J Am Med Inform Assoc 2008; 15:138–149.Natter MD, Quan J, Ortiz DM, et al. An i2b2-based, generalizable, open source, self-scaling chronic disease registry. J Am Med Inform Assoc 2013; 20:172–179.Ohno-Machado L, Bafna V, Boxwala AA, et al. iDASH: Integrating data for analysis, anonymization, and sharing. J Am Med Inform Assoc 2012; 19:196–201.Channin DS, Mongkolwat P, Kleper V, Rubin DL. Computing human image annotation. Conf Proc IEEE Eng Med Biol Soc 2009; 1:7065–8.Sittig DF, Wright A, Osheroff JA, et al. Grand challenges in clinical decision support. J Biomed Inform 2008; 41(2):387–392.Wagholikar KB, Sundararajan V, Deshpande AW. Modeling paradigms for medical diagnostic decision support: a survey and future directions. J Med Syst 2012; 36(5):3029–3049.Rubin DL. Creating and curating a terminology for radiology: Ontology modeling and analysis. J Digit Imaging 2008; 21(4):355–362.Kahn CE, Jr., Langlotz CP, Burnside ES, Carrino JA, Channin DS, Hovsepian DM, et al. Toward best practices in radiology reporting. Radiology 2009; 252(3):852–856.Taira PK, Soderlang SG, JAbovits RM. Automatic structuring of radiology free-text reports. Radiographics 2001; 21(1); 237–245.Fujii H, Yamagishi H, Ando Y, Tsukamoto N, Kawaguchi O, Kasamatsu T, et al. Structuring of free-text diagnostic report. Stud. Health Technol. Inform. 2007; 129: 669–673.Murff HJ, FitzHenry F, Matheny ME, Gentry N, Kotter KL, Crimin K, Dittus RS, Rosen AK, Elkin PL, Brown SH, Speroff T. Automated identification of postoperative complications within an electronic medical record using natural language processing. JAMA 2011; 306(8):848–855.Clunie DA: DICOM structured reporting. PixelMed Publishing, 2000D’Avolio LW, Nguyen TM, Farwell WR, Chen Y, Fitzmeyer F, Harris OM, Fiore LD. Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC). J Am Med Inform Assoc 2012; 17:375–382.Napel SA, Beaulieu CF, Redriguez C, Cui J, Xu J, Grupta A, et al. Automated retrieval of CT images of liver lesions on the basis of image similarity: Method and preliminary results. Radiology 2010; 256(1): 243–252.Langlotz CP. RadLex: A new method for indexing online educational materials. Radiographics 2006; 26(6):1595–1597.Crestania F, Vegas J, de la Fuente P. A graphical user interface for the retrieval of hierarchically structured documents. Inf Process Manag 2004; 40(2):269–289.Weiss DL, Langlotz CP. Structured reporting: Patient care enhancement or productivity nightmare? Radiology 2008. 249(3):739–747.Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19(3):413–422.Patrick R, Julien G, Christian L, Antoine G. Automatic medical encoding with SNOMED categories. BMC Med Inform Decis Mak 2008; 8(Suppl 1): S1–S6.Lopez-Garcia P, Boeker M, Illarramendi A, Schulz S. Usability-driven pruning of large ontologies: The case of SNOMED CT, J Am Med Inform Assoc 2012; 19:e102-e109.World Health Organization. International Statistical Classification of Diseases and Related Health Problems 10th Revision. http://apps.who.int/classifications/apps/icd/icd10online/ (accessed 29 Jan 2013)American College of Radiology (ACR) Breast Imaging Reporting and Data System Atlas (BI-RADS® Atlas)World Health Organization. International Classification of Diseases for Oncology, 3rd Edition (ICD-O-3). http://www.who.int/classifications/icd/adaptations/oncology/en/index.html (accessed 29 Jan 2013)Greene FL. TNM: Our language of cancer. CA Cancer J Clin 2004; 54(3):129–130.American Joint Committee of Cancer (AJCC). AJCC Cancer Staging Manual. Seventh Edition. Springer, 2010Hussein R, Engelmann U, Schroeter A, Meinzer HP. DICOM structured reporting: Part 1. Overview and characteristics, Radiographics 2004; 24(3):891–896.Sluis D, Lee KP, Mankovich N. DICOM SR - integrating structured data into clinical information systems. Medicamundi 2002; 46(2):31–36.Percha B, Nassif H, Lipson J, Burnside E, Rubin D. Automatic classification of mammography reports by BI-RADS breast tissue composition class. J Am Med Inform Assoc 2012; 19(5):913–916.Ciatto S, Houssami N, Apruzzese A, Bassetti E, Brancato B, Carozzi F, Catarzi S, Lamberini MP, Marcelli G, Pellizzoni R, Pesce B, Risso G, Russo F, Scorsolini A. Reader variability in reporting breast imaging according to BI-RADS assessment categories (the Florence experience). Breast 2006; 15(1):44–51.National Electrical Manufacturers Association (NEMA). Digital Imaging and Communications in Medicine (DICOM). Part 16: Content Mapping Resource. http://medical.nema.org/dicom/2004/04_16PU.PDF (accessed 29 Jan 2013)Dolin RH, Alschuler L, Boyer S, Beebe C, Behlen FM, Biron PV, Shvo AS. HL7 clinical document architecture, release 2. J Am Med Inform Assoc 2006; 13:30–39.Blanquer I, Hernández V, Meseguer JE, Segrelles D. Content-based organisation of virtual repositories of DICOM objects. Future Gener Comput Syst 2009; 25(6):627–637.Blanquer I, Hernández V, Segrelles D, Torres E. Enhancing privacy and authorization control scalability in the grid through ontologies. IEEE Trans Inf Technol Biomed 2009; 12(1):16–24.Salavert J, Maestre C, Segrelles D, Blanquer I, Hernández V, Medina R, Martí L: Grid prototype to support cancer of breast diagnostics in clinic practice. Proc of the 4th. Iberian Grid Infrastructure Conf. Netbiblo, 2010Segrelles D, Franco JM, Medina R, Blanquer I, Salavert J, Hernandez V, Martí L, Díaz G, Ramos R, Guevara MA, González N, Loureiro J, Ramos I. Exchanging data for breast cancer diagnosis on heterogeneous grid platforms. Computing and Informatics 2012; 31(1):3–15.Ali MS, Consens M, Lalmas M. Extended structural relevance framework: A framework for evaluating structured document retrieval. Inf Retrieval 2012; 15:558–590.Welter P, Riesmeier J, Fischer B, Grouls C, Kuhl C, Deserno, TM. Bridging the integration gap between imaging and information systems: A uniform data concept for content-based image retrieval in computer-aided diagnosis. J Am Med Inform Assoc 2011; 18:506–510.Jenkins CW. Application prototyping: A case study. Perform Eval Rev 1981; 10(1):21–27.Generalitat Valenciana. Conselleria de Sanitat. Oncoguía de Cáncer de Mama Comunidad Valenciana. http://publicaciones.san.gva.es/publicaciones/documentos/V.2478-2006.pdf (accessed 29 Jan 2013)Maestre C, Segrelles-Quilis JD, Torres E, Blanquer I, Medina R, Hernández V, Martí L. Assessing the usability of a science gateway for medical knowledge bases with TRENCADIS. J Grid Computing 2012; 10:665–688.Lewis J. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. Int J Hum-Comput Interact 1995; 7(1):57–78.Lewis JR. Psychometric evaluation of the PSSUQ using data from five years of usability studies. Int J Hum-Comput Interact 2002; 14(3–4):463–488.Shapiro SS, Wilk MB. An analysis of variance test for normality (complete samples). Biometrika 1965; 52(3–4):591–611.Chhatwal J, Alagoz O, Lindstrom MJ, Kahn Jr CE, Shaffer KA, Burnside ES. A logistic regression model based on the national mammography database format to aid breast cancer diagnosis. AJR Am J Roentgenol 2009; 192:1117–1127

Increasing the Efficiency on Producing Radiology Reports for Breast Cancer Diagnosis by Means of Structured Reports

Background: Radiology reports are commonly written on free-text using voice recognition devices. Structured reports (SR) have a high potential but they are usually considered more difficult to fill-in so their adoption in clinical practice leads to a lower efficiency. However, some studies have demonstrated that in some cases, producing SRs may require shorter time than plain-text ones. This work focuses on the definition and demonstration of a methodology to evaluate the productivity of software tools for producing radiology reports. A set of SRs for breast cancer diagnosis based on BI-RADS have been developed using this method. An analysis of their efficiency with respect to free-text reports has been performed. Material and Methods: The methodology proposed compares the Elapsed Time (ET) on a set of radiological reports. Free-text reports are produced with the speech recognition devices used in the clinical practice. Structured reports are generated using a web application generated with TRENCADIS framework. A team of six radiologists with three different levels of experience in the breast cancer diagnosis was recruited. These radiologists performed the evaluation, each one introducing 50 reports for mammography, 50 for ultrasound scan and 50 for MRI using both approaches. Also, the Relative Efficiency (REF) was computed for each report, dividing the ET of both methods. We applied the T-Student (T-S) test to compare the ETs and the ANOVA test to compare the REFs. Both tests were computed using the SPSS software. Results: The study produced three DICOM-SR templates for Breast Cancer Diagnosis on mammography, ultrasound and MRI, using RADLEX terms based on BIRADs 5th edition. The T-S test on radiologists with high or intermediate profile, showed that the difference between the ET was only statistically significant for mammography and ultrasound. The ANOVA test performed grouping the REF by modalities, indicated that there were no significant differences between mammograms and ultrasound scans, but both have significant statistical differences with MRI. The ANOVA test of the REF for each modality, indicated that there were only significant differences in Mammography (ANOVA p = 0.024) and Ultrasound (ANOVA p = 0.008). The ANOVA test for each radiologist profile, indicated that there were significant differences on the high profile (ANOVA p = 0.028) and medium (ANOVA p = 0.045). Conclusions: In this work, we have defined and demonstrated a methodology to evaluate the productivity of software tools for producing radiology reports in Breast Cancer. We have evaluated that adopting Structured Reporting in mammography and ultrasound studies in breast cancer diagnosis improves the performance in producing reports.INDIGO - DataCloud receives funding from the European Union's Horizon 2020 research and innovation programme under grant agreement RIA 653549.Segrelles Quilis, JD.; Medina, R.; Blanquer Espert, I.; Marti Bonmati, L. (2017). Increasing the Efficiency on Producing Radiology Reports for Breast Cancer Diagnosis by Means of Structured Reports. Methods of Information in Medicine. 56:1-13. https://doi.org/10.3414/ME16-01-0091S1135

Assessing the usability of a science gateway for medical knowledge bases with TRENCADIS

Biomedical applications are often built on top of knowledge bases that contain medical images and clinical reports. Currently, these bases are being used to improve diagnosis, research and teaching, but in many cases, the infrastructure required has a prohibitive cost for many medical centres. However, resources can be attached from existing e-Science infrastructures. Therefore, many efforts have been made to establish best practices that allow the use of such infrastructures. However, e-Science relies on open, distributed, collaborative environments, built on top of very specialized technologies, such as Grid and Cloud computing, which require reasonable technical skills for their usage. Therefore, science gateways have become essential tools that assist users in interacting with e-Science applications. This paper describes TRENCADIS, a technology that supports the creation and operation of virtual knowledge bases. To this end, it provides developers with components and APIs for building secure data services that can be annotated and queried through ontology templates, based on DICOM and DICOM-SR. This technology was used in this paper to build a gateway for assisting diagnosis and research in breast cancer. We also present here the results of a study conducted to evaluate the gateway, from the point of view of the usability perceived by a group of physicians and radiologists. © 2012 Springer Science+Business Media Dordrecht.The authors wish to thank the financial support received from The Spanish Ministry of Education and Science to develop the project "Code-Cloud", with reference TIN2010-17804, and the financial support received from The Vicerectorat d'Investigacio de la Universitat Politecnica de Valencia (UPV) to develop the project "Diseno de Componentes Cloud Facilitadores del Despliegue y la Alta Disponibilidad de Servicios TRENCADIS, para compartir Imagenes Medicas DICOM e informes Asociados DICOM-SR", with reference 20111013.Maestre Urbano, CV.; Segrelles Quilis, JD.; Torres Serrano, E.; Blanquer Espert, I.; Medina, R.; Hernández García, V.; Martí, L. (2012). Assessing the usability of a science gateway for medical knowledge bases with TRENCADIS. Journal of Grid Computing. 10(4):665-688. https://doi.org/10.1007/s10723-012-9243-2S665688104Afgan, E., Goecks, J., Baker, D., Coraor, N., Team The Galaxy, Nekrutenko, A., Taylor, J.: Galaxy—a gateway to tools in e-Science. In: Yang, X., Wang, L., Jie, W. (eds.) Guide to e-Science: Next Generation Scientific Research and Discovery, pp. 145–180. Springer (2011)Aisen, A.M., Broderick, L.S., Winer-Muram, H., Brodley, C.E., Kak, A.C., Pavlopoulou, C., Dy, J., Shyu, C.R., Marchiori, A.: Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. Radiology 228(1), 265–270 (2003)Allcock, W.: GridFTP: protocol extensions to FTP for the Grid. Tech. rep., Argonne National Laboratory (2003). http://www.ggf.org/documents/GFD.20.pdfAndronico, G., Ardizzone, V., Barbera, R., Becker, B., Bruno, R., Calanducci, A., Carvalho, D., Ciuffo, L., Fargetta, M., Giorgio, E., Rocca, G., Masoni, A., Paganoni, M., Ruggieri, F., Scardaci, D.: e-Infrastructures for e-Science: a global view. J. Grid Computing 9(2), 155–184 (2011). doi: 10.1007/s10723-011-9187-yBastien, J.M.C.: Usability testing: a review of some methodological and technical aspects of the method. Int. J. Med. Inform. 79(4), e18–e23 (2010)Bastien, J.M.C., Scapin, D.L.: Ergonomic Criteria for the Evaluation of Human-Computer Interfaces (1993)Blanquer, I., Hernández, V., Segrelles, D., Torres, E.: Enhancing privacy and authorization control scalability in the Grid through ontologies. IEEE Trans. Inf. Technol. Biomed. 13(1), 16–24 (2009). doi: 10.1109/TITB.2008.2003369Blanquer, I., Hernández, V., Salavert, J., Segrelles, D.: Integrating TRENCADIS components in gLite to share DICOM medical images and structured reports. Stud. Health Technol. Inform. 159, 64–75 (2010)Blanquer Espert, I., Hernández García, V., Meseguer Anastásio, F.J., Segrelles Quilis, J.D.: Content-based organisation of virtual repositories of DICOM objects. Future Gener. Comput. Syst. 25(6), 627–637 (2009). doi: 10.1016/j.future.2008.12.004Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility. Future Gener. Comput. Syst. 25(6), 599–616 (2009). doi: 10.1016/j.future.2008.12.001Clunie, D.A.: DICOM Structured Reporting. PixelMed, Bangor (2000)Cockburn, A.: Writing Effective Use Cases. Addison-Wesley Professional (2000)Costa, C., Ferreira, C., Bastião, L., Ribeiro, L., Silva, A., Oliveira, J.: Dicoogle—an open source peer-to-peer PACS. J. Digit. Imaging 24(5), 848–856 (2011). doi: 10.1007/s10278-010-9347-9Cunsolo, V.D., Distefano, S., Puliafito, A., Scarpa, M.L.: GS3: a Grid storage system with security features. J. Grid Computing 8(3), 391–418 (2010). doi: 10.1007/s10723-010-9157-9Dumas, J.S., Redish, J.: A Practical Guide to Usability Testing. Intellect Ltd (1999)European Commission: A digital agenda for Europe. Communication 5(245 final/2), 42 (2010)European Commission: Riding the wave—how Europe can gain from the rising tide of scientific data. Final report of the High Level Expert Group on Scientific Data. Tech. Rep. October, European Commission (2010)European Commission: Safeguarding privacy in a connected world a European data protection framework for the 21st century. Tech. rep., European Commission, Brussels, Belgium (2012)Freund, J., Comaniciu, D., Ioannis, Y., Liu, P., McClatchey, R., Morley-Fletcher, E., Pennec, X., Pongiglione, G., Xiang, Zhou: Health-e-child: an integrated biomedical platform for Grid-based paediatric applications. Stud. Health Technol. Inform. 120, 259–270 (2006)Fuster-Garcia, E., Navarro, C., Vicente, J., Tortajada, S., García-Gómez, J.M., Sáez, C., Calvar, J., Griffiths, J., Julià-Sapé, M., Howe, F.A., Pujol, J., Peet, A.C., Heerschap, A., Moreno-Torres, A., Martínez-Bisbal, M.C., Martínez-Granados, B., Wesseling, P., Semmler, W., Capellades, J., Majós, C., Alberich-Bayarri, A., Capdevila, A., Monleón, D., Martí-Bonmatí, L., Arús, C., Celda, B., Robles, M.: Compatibility between 3T 1H SV-MRS data and automatic brain tumour diagnosis support systems based on databases of 1.5T 1H SV-MRS spectra. Magma (N.Y. N.Y.) 24(1), 35–42 (2011). doi: 10.1007/s10334-010-0241-8Geer, L.Y., Marchler-Bauer, A., Geer, R.C., Han, L., He, J., He, S., Liu, C., Shi, W., Bryant, S.H.: The NCBI BioSystems database. Nucleic Acids Res. 38(Database issue), D492–6 (2010). doi: 10.1093/nar/gkp858Giger, M.L.: Intelligent CAD workstation for breast imaging using similarity to known lesions and multiple visual prompt aids. Proc. SPIE 4684, 768–773 (2002). doi: 10.1117/12.467222Grethe, J.S., Baru, C., Gupta, A., James, M., Ludaescher, B., Martone, M.E., Papadopoulos, P.M., Peltier, S.T., Rajasekar, A., Santini, S., Zaslavsky, I.N., Ellisman, M.H.: Biomedical informatics research network: building a national collaboratory to hasten the derivation of new understanding and treatment of disease. Stud. Health Technol. Inform. 112, 100–109 (2005)Hornbak, K.: Current practice in measuring usability: challenges to usability studies and research. Int. J. Human-Comput. Stud. 64(2), 79–102 (2006). doi: 10.1016/j.ijhcs.2005.06.002ISO—International Organization for Standardization: Ergonomic requirements for office work with visual display terminals (VDTs)—part 11: guidance on usability (1998)ISO—International Organization for Standardization: Software engineering—product quality—part 4: quality in use metrics (2004)Jaspers, M.W.M.: A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int. J. Med. Inform. 78(5), 340–353 (2009)Kranzlmüller, D., Lucas, J.M., Öster, P.: The European Grid Initiative (EGI). In: Davoli, F., Pugliese, R., Meyer, N., Zappatore, S. (eds.) Remote Instrumentation and Virtual Laboratories, pp. 61–66. Springer (2010). doi: 10.1007/978-1-4419-5597-5_6Langlotz, C.P.: RadLex: a new method for indexing online educational materials. Radiographics 26(6), 1595–1597. doi: 10.1148/rg.266065168Lee, K.P., Hu, J.: XML schema representation of DICOM structured reporting. J. Am. Med. Inform. Assoc. 10(2), 213–223 (2003)Lenert, L., Sundwall, D.N.: Public health surveillance and meaningful use regulations: a crisis of opportunity. Am. J. Publ. Health 102(3), e1–e7 (2012). doi: 10.2105/AJPH.2011.300542Lewis, J.: IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int. J. Hum.-Comput. Interact. 7(1), 57–78 (1995). doi: 10.1080/10447319509526110Lewis, J.R.: Psychometric evaluation of the PSSUQ using data from five years of usability studies. Int. J. Hum.-Comput. Interact. 14(3–4), 463–488 (2002). doi: 10.1080/10447318.2002.9669130Lindland, O.I., Sindre, G., Solvberg, A.: Understanding quality in conceptual modeling. IEEE Softw. 11(2), 42–49 (1994). doi: 10.1109/52.268955Llorente, I.M., Moreno-Vozmediano, R., Montero, R.S.: Cloud computing for on-demand resource provisioning. In: Gentzsch, W., Grandinetti, L., Joubert, G. (eds.) High Speed and Large Scale Scientific Computing, pp. 177–191. IOS Press, Amsterdam (2009). doi: 10.3233/978-1-60750-073-5-177Napel, S.A., Beaulieu, C.F., Rodriguez, C., Cui, J., Xu, J., Gupta, A., Korenblum, D., Greenspan, H., Ma, Y., Rubin, D.L.: Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. Radiology 256(1), 243–252 (2010)Nielsen, J.: Usability Engineering, vol. 44. Morgan Kaufmann (1993)OGF—Open Grid Forum: The Storage Resource Manager Interface Specification Version 2.2. Tech. rep. (2008). http://www.ogf.org/documents/GFD.129.pdfOGF—Open Grid Forum: A simple API for Grid applications (SAGA). Tech. rep. (2011). http://www.ogf.org/documents/GFD.90.pdfOGF—Open Grid Forum: Data Format Description Language (DFDL) v1.0 Specification. Tech. rep. (2011). http://www.ogf.org/documents/GFD.174.pdfOGF—Open Grid Forum: Distributed Resource Management Application API Version 2 (DRMAA). Tech. rep. (2012). http://www.ogf.org/documents/GFD.194.pdfPereira, J.A., Quach, S., Hamid, J.S., Heidebrecht, C.L., Quan, S.D., Nassif, J., Diniz, A.J., Exan, R.V., Malawski, J., Gentry, A., Finkelstein, M., Guay, M., Buckeridge, D.L., Bettinger, J.A., Kalailieff, D., Kwong, J.C.: Exploring the feasibility of integrating barcode scanning technology into vaccine inventory recording in seasonal influenza vaccination clinics. Vaccine 30(4), 794–802 (2012). doi: 10.1016/j.vaccine.2011.11.043Ramos-Pollán, R., Guevara-López, M.A., Suárez-Ortega, C., Díaz-Herrero, G., Franco-Valiente, J.M., Rubio-Del-Solar, M., González-de Posada, N., Vaz, M.A.P., Loureiro, J., Ramos, I.: Discovering mammography-based machine learning classifiers for breast cancer diagnosis. J. Med. Syst. 36(4), 2259–2269 (2012). doi: 10.1007/s10916-011-9693-2Redolfi, A., McClatchey, R., Anjum, A., Zijdenbos, A., Manset, D., Barkhof, F., Spenger, C., Legré, Y., Wahlund, L.O., Pietro, C.B.d.S., B Frisoni, G.: Grid infrastructures for computational neuroscience: the neuGRID example. Future Neurol. 4(6), 703–722 (2009). doi: 10.2217/fnl.09.53Rimal, B.P., Jukan, A., Katsaros, D., Goeleven, Y.: Architectural requirements for cloud computing cystems: an enterprise cloud approach. J. Grid Computing 9(1), 3–26 (2010). doi: 10.1007/s10723-010-9171-yRings, T., Caryer, G., Gallop, J., Grabowski, J., Kovacikova, T., Schulz, S., Stokes-Rees, I.: Grid and cloud computing: opportunities for integration with the next generation network. J. Grid Computing 7(3), 375–393 (2009). doi: 10.1007/s10723-009-9132-5Rodero-Merino, L., Vaquero, L.M., Gil, V., Galán, F., Fontán, J., Montero, R.S., Llorente, I.M.: From infrastructure delivery to service management in clouds. Future Gener. Comput. Syst. 26(8), 1226–1240 (2010). doi: 10.1016/j.future.2010.02.013Rosenthal, A., Mork, P., Li, M.H., Stanford, J., Koester, D., Reynolds, P.: Cloud computing: a new business paradigm for biomedical information sharing. Journal of Biomedical Informatics 43(2), 342–353 (2010)Ruch, P., Gobeill, J., Lovis, C., Geissbühler, A.: Automatic medical encoding with SNOMED categories. BMC Med. Inform. Decis. Mak. 8(Suppl 1), S6 (2008). doi: 10.1186/1472-6947-8-S1-S6Salavert Torres, J., Maestre Urbano, C.V., Segrelles Quilis, J.D., Blanquer Espert, I., Garcia, H.V., Martí Bonmatí, L.: Grid prototype to support cancer of breast diagnostics in clinic practice. In: IBERGRID’2010—4th Iberian Grid Infrastructure Conference. Netbiblo, Braga, pp. 285–294 (2010)Saleem, J., Haggstrom, D., Militello, L., Flanagan, M., Kiess, C., Arbuckle, N., Doebbeling, B.: Redesign of a computerized clinical reminder for colorectal cancer screening: a human-computer interaction evaluation. BMC Med. Inform. Decis. Mak. 11(1), 74 (2011). doi: 10.1186/1472-6947-11-74Schwiegelshohn, U., Badia, R.M., Bubak, M., Danelutto, M., Dustdar, S., Gagliardi, F., Geiger, A., Hluchy, L., Kranzlmüller, D., Laure, E.: Perspectives on Grid computing. Future Gener. Comput. Syst. 26(8), 1104–1115 (2010). doi: 10.1016/j.future.2010.05.010Segrelles, D., Blanquer, I., Salavert, J., Hernandez, V., Franco, J.M., Diaz, G., Ramos, R., Medina, R., Marti, L., Guevara, M.A., Gonzalez, N., Loureiro, J., Ramos, I.: Exchanging data for breast cancer diagnosis on heterogeneous Grid platforms. Comput. Inform. 31(1), 3–15 (2012)Serco Usability Services: Performance Measurement Handbook, version 3 edn. Middlesex, UK (1995)Shapiro, S.S., Wilk, M.B.: An analysis of variance test for normality (complete samples). Biometrika 52(3–4), 591–611 (1965). doi: 10.1093/biomet/52.3-4.591Silva, L., Costa, C., Oliveira, J.: A PACS archive architecture supported on cloud services. Int. J. Comput. Assist. Radiol. Surg. 7(3), 349–358 (2012). doi: 10.1007/s11548-011-0625-xTeng, C.C., Mitchell, J., Walker, C., Swan, A., Davila, C., Howard, D., Needham, T.: A medical image archive solution in the cloud. In: 2010 IEEE International Conference on Software Engineering and Service Sciences (ICSESS), pp. 431–434 (2010). doi: 10.1109/ICSESS.2010.5552343Van Rijsbergen, C.J.: Information retrieval. In: The Kluwer International Series on Information Retrieval, vol. 30. Butterworths (1979). doi: 10.1016/0020-0271(68)90016-8Weiss, D.L., Langlotz, C.P.: Structured reporting: patient care enhancement or productivity nightmare? Radiology 249(3), 739–747 (2008). doi: 10.1148/radiol.2493080988Wess, M.L., Saleem, J.J., Tsevat, J., Luckhaupt, S.E., Johnston, J.A., Wise, R.E., Kopke, J.E., Eckman, M.H.: Usability of an atrial fibrillation anticoagulation decision-support tool. J. Prim. Care Community Health 2(2), 100–106 (2011). doi: 10.1177/2150131910387608Zentner, L.K., Clark, S.M., Smith, P.M., Shivarajapura, S., Farnsworth, V., Madhavan, K.P.C., Klimeck, G.: Practical considerations in cloud utilization for the science gateway nanoHUB.org. In: 2011 Fourth IEEE International Conference on Utility and Cloud Computing (UCC), pp. 287–292 (2011). doi: 10.1109/UCC.2011.46Zhao, L., Lee, K.P., Hu, J.: Generating XML schemas for DICOM structured reporting templates. J. Am. Med. Inform. Assoc. 12(1), 72–83 (2005)Zheng, B., Mello-Thoms, C., Wang, X.H., Abrams, G.S., Sumkin, J.H., Chough, D.M., Ganott, M.A., Lu, A., Gur, D.: Interactive computer-aided diagnosis of breast masses: computerized selection of visually similar image sets from a reference library. Acad. Radiol. 14(8), 917–927 (2007

Diseño centrado en el usuario y evaluación de usabilidad de una interfaz de apoyo al diagnóstico de cáncer de mama a partir de imagen médica

Este proyecto final de carrera (PFC) presenta el diseño y evaluación de una interfaz para una aplicación informática que da soporte a la creación de informes radiológicos, los cuales presentan resultados sobre exploraciones basadas en imagen médica, con el objetivo de organizar la información para ayudar al diagnóstico y el estudio del cáncer de mama. La aplicación recibe el nombre de "Generación de informes estructurados para el cáncer de mama".Maestre Urbano, CV. (2014). Diseño centrado en el usuario y evaluación de usabilidad de una interfaz de apoyo al diagnóstico de cáncer de mama a partir de imagen médica. Universitat Politècnica de València. http://hdl.handle.net/10251/58465Archivo delegad

Utilidad de los informes estructurados basados en una plataforma web para el diagnóstico por imagen en patología mamaria

El cáncer de mama es una enfermedad con un gran impacto sobre la salud poblacional. A pesar de los avances en esta enfermedad, la mortalidad sigue siendo elevada, por lo que es importante investigar en otros campos y herramientas como por ejemplo en el ámbito de la recogida de datos de forma estandarizada, para recoger información de forma estructurada, decisiones protocolizadas y más objetivas, mejorando esto el pronóstico de la enfermedad. Con respecto a las necesidades actuales referentes fundamentalmente a los sistemas de información, los desarrollos realizados en esta Tesis Doctoral combinan la aplicación de tecnologías y estándares de la información. La hipótesis de trabajo de esta tesis es que los informes estructurados en patología mamaria optimizan los flujos de información en el proceso radiológico con respecto a los informes que emplean el texto plano. Esta optimización se reflejará en una mejor completitud, validez, efectividad y eficacia de los mismos, sin añadir un tiempo excesivo en términos de productividad al introducir la información. Los resultados obtenidos en esta tesis se centran en valorar la importancia del uso de los informes estructurados en el ámbito del cáncer de mama frente al informe convencional comúnmente realizado con texto plano. Con este objetivo se han desarrollado una serie de aportaciones que se detallan a continuación: 1. Incorporación a los flujos de trabajo ya existentes en el cáncer de mama y que afectan a su diagnóstico, tratamiento y seguimiento, de los informes estructurados organizados siguiendo la terminología BI-RADS (estándar de información radiológica) y DICOM-SR (estándar de estructura de informes), para obtener unos flujos de trabajo mejorados. 2. Desarrollo de un prototipo web que permite recoger los datos a través de los informes estandarizados al mismo tiempo que se realiza el informe radiológico estructurado. Estos datos quedan almacenados en una única base de datos para su posterior análisis y uso (como compartir con otros centros, realizar políticas de mejora de salud, facilitar la resolución de casos-problema, evaluar tendencias y extraer casos relacionados). Esta plataforma recoge toda la información necesaria para llevar a cabo un informe mamario completo, ya que incorpora una serie de plantillas consensuadas y específicas para cada técnica de imagen mamaria. 3. Creación de plantillas para los informes de mamografía, ecografía y RM, que reúnen todos los descriptores siguiendo el estándar BI-RADS. Por cada técnica de imagen se ha desarrollado una plantilla y a cada descriptor se le ha asignado su “código universal” según la ontología estándar RADLEX para el caso de la información radiológica y SNOMED-CT para la información no radiológica. Del mismo modo, se recoge en las plantillas si los datos son de obligatorio cumplimiento y la condicionalidad de los campos, para así codificar toda la información correctamente. 4. Mediante unos supuestos clínicos, realizados de manera ciega por varios profesionales de radiodiagnóstico, se ha evaluado el uso del informe estructurado, obteniéndose una mejoría en la completitud, validez, y efectividad de los informes estructurados con respecto al texto plano convencional y, por tanto, una mejora en la eficiencia. Así, empleando el informe estructurado se consiguen informes más rápidos, con una completitud superior al 90% y con una validez y efectividad entorno al 80%. En la evaluación de la mejoría del tiempo empleado y la eficiencia según las técnicas y el personal que las interpreta, se ha constatado que: 4.1. Con el sistema de informes estructurados basado en DICOM-SR se ahorra una media de 26 segundos por cada exploración informada. 4.2. De entre todas las técnicas de imagen, y de forma global, el informe estructurado es más eficiente en la mamografía (al compararla con la RM) y sobretodo en los usuarios más expertos (radiólogos adjuntos y R4+R3 al compararlos con R2). También se ha observado que para los radiólogos adjuntos el informe estructurado es más eficiente en las tres técnicas de imagen, siendo para los residentes expertos R4+R3 más eficiente para la mamografía y para los residentes de menor experiencia R2 más eficiente para la ecografía. 4.3. Si se analiza por técnicas, los informes estructurados son más completos, válidos y efectivos para la ecografía y la RM frente a la mamografía, en todos los subgrupos de usuarios, excepto en el de médicos adjuntos donde los informes para las tres técnicas son igual de completos. Si el análisis es por usuarios, las diferencias se encuentran entre los informes estructurados para los radiólogos más expertos (adjuntos) y los residentes, obteniéndose mejores resultados con una mayor experiencia. 5. No obstante y pese a lograr estandarizar la información con la creación de un prototipo web bien organizado mediante plantillas de cada una de las técnicas de imagen y su codificación siguiendo estándares universales (ontologías como RADLEX y SNOMED-CT), se ha constatado que los datos obtenidos están influenciados por factores asociados a la propia experiencia de los radiólogos. En esta tesis se plasma la creación y análisis de un sistema informático que elimina las variaciones e ineficiencias en la creación de informes radiológicos.Breast cancer is a disease with a major impact on population health. Despite major advances have been done to mitigate this disease, nowadays the mortality rate due to breast cancer still remains high. Therefore, it is crucial to open new lines of research beyond the traditional approaches based on experimental studies. In the last decade, there is an increasing interest in the analysis of large data collections due to the tremendous amount of information that is unused. New tools are needed to dig into the medical reports and to discover new knowledge that could help physicians to design more accurate protocols to manage breast cancer and to improve the prognosis of this disease. This thesis combines new-generation analytics with the use of medical standards to improve the quality of the data. We started our research from the hypothesis that the use of structured data formats to report pathological findings may improve the quality of the information allowing its use in automatic pipelines, compared with traditional free-text reports where many important details are blurred by the lack of precision and ambiguity. Our objective with this thesis is to prove this hypothesis, measuring the improvement by means of 4 quality metrics: completeness, validity, effectiveness and efficiency of the structured reports. To this end, our efforts were focused on evaluating the use of structured reports on breast cancer compared to conventional reports based on free-text. The contributions of this work are listed below: 1. Redesign of the clinical pipelines for the management of breast cancer, introducing structured reports based on standards (BI-RADS terminology and DICOM-SR data format) for diagnosis, treatment and patient outcomes monitoring. 2. Development of a Web-based prototype of a data collection system that allows physicians to introduce their findings in a common database. This prototype uses the most recent advances in security to support analytic operations on the data as well as other common operations, including data sharing among medical centres, information lookup and discovering patterns in the data. Also, the prototype allows users to download medical reports using a standardized data format. 3. Report templates based on the BI-RADS terminology were created for the different imaging techniques available, including mammography, ultrasound and MRI. Other relevant standards are included. In particular, radiological information is annotated with RADLEX ontologies and SNOMED CT terms are used for non-radiological information. All the templates share a common validation scheme where every field is assigned to a specific data type and optional fields are labelled. 4. A usability study was conducted with the participation of real radiologists to evaluate the use of structured reports and to compare the accuracy of the structured reports with the conventional free-text reports. The results of this experiment showed an improvement in the completeness (>90%), validity (>80%) and effectiveness (>80%) of the structured reports over conventional free-text reports. The experiment also measured the efficiency of the participants with the following results: 4.1. Information lookup time was reduced in 26 seconds by using structured reports. 4.2. Mammography was the imaging technique where the shortest times were achieved, especially among expert users. In contrast, introducing ultrasound reports in the system achieved the longest times. 4.3. Experts completed all the reports in the three imaging categories: mammography, ultrasound and MRI, while residents made more mistakes and create incomplete reports when they work with ultrasound and MRI. 5. Despite the improvements achieved with this thesis, the use of standard terminologies, the creation of report templates and the development of tools for the validation of the information are only one part of the solution. There are also human factors, such as the user expertise and the previous experience with computers and technology, affecting the quality of the data collected at the hospitals. 6. Finally, in this thesis we created a computer system that contributes to eliminate inter- and intra-observer variations, helping to improve both the efficiency and the quality of the medical reports

Herramienta de gestión integral en innovación en imagen médica

La implementació de les Tecnologies de la informació i la Comunicació (TIC) i la digitalització de la imatge medica ha suposat un canvi en tot el progres clínic assistencial per part de radiòlegs i metges nuclears encarregats de realitzar l'informe diagnòstic i les intervencions radiològiques. Actualment, els sistemes d'informació coneguts com Picture Archiving and Communication System (PACS), Radiology Information System (RIS), i Hospital Information System (HIS) permeten el maneig dels estudis adquirits, el seu informat i el seguiment dels processos de gestió associats als fluxos de l'activitat assistencial. Propiciat per aquest profund canvi han nascut noves oportunitats i necessitats que, desafortunadament, no han sigut resoltes i integrades en la majoria dels entorns hospitalaris. Per exemple, gracies a la digitalització de la imatge i als avanços en investigació, és possible obtindré dades quantitatives de l'exploració adquirida que reflecteixen l'estat d'una malaltia o de l'efecte d'un fàrmac sobre aquesta. Aquestes mesures es coneixen com biomarcadors d'imatge. Per altra part, actualment els informes realitzats pels metges especialistes no tenen una estructura que impedisca una variabilitat en el contingut i en conseqüència l'absència potencial d'informació rellevant pel metge peticionari de l'exploració. El disseny i desenvolupament dels informes estructurats mitjançant l'ús de lèxics normalitzats i plantilles es factible amb les TIC. Aquests informes deuen establir-se per malalties i lesions concretes, i deuen estar consensuats entre els metges especialistes de la imatge i els metges peticionaris. Degut al volum de dades generades pels sistemes d'informació citats anteriorment, es crea contínuament una font de coneixement de dades que no son explotades. L'extracció d'aquest coneixement a través d'indicadors deu de permetre visualitzar l'estat actual dels serveis de radiologia i de medicina nuclear, de tal manera que es possibilite corregir els colls de botella i realitzar accions correctores oportunes front a una situació critica. El proposit final és promoure un funcionament òptim dels serveis i facilitar la presa de decisions fonamentades a les dades d'activitat que millor s'ajusten per a una millor atenció al pacient. Aquesta Tesi Doctoral té per objectiu l'integració eficient dels biomarcadors d'imatge, els informes estructurats, i els indicadors d'activitat en la pràctica assistencial dels serveis de radiologia i de medicina nuclear. Per aconseguir aquest objectiu s'utilitzaren estàndards d'imatge medica com el Digital Imaging and Communication in Medicine (DICOM) i altres com eXtensible Language Market (XML). Mitjançant la tecnologia JAVA es desenvolupà una plataforma per a la integració dels biomarcadors d'imatge. Mitjançant l'utilització de lèxics com el Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) i Radiology Lexicon (RADLEX), més l'estàndard DICOM i l'estàndard HML5, es va implantar una aplicació que permet l'integració de l'informe estructurat. D'aquesta manera es poden realitzar estudis poblacionals, així com analitzar la relació de determinats factors en una malaltia especifica. Per últim, de l'informació obtinguda del RIS i del PACS, es va construir una plataforma d'indicadors amb tecnologia JAVA per a permetre visualitzar l'estat dels serveis respecte a l'activitat assistencial; activitat d'innovació; i activitat d'investigació i docent. Per tant, la present Tesi Doctoral aporta als servicis de radiologia i de medicina nuclear una ferramenta d'innovació amb tres camps fonamentals per a oferir un millor diagnòstic, una millor cura i atenció al pacient. A través de la quantificació dels biomarcadors d'imatge i l'informe estructurat per a una medicina personalitzada, i amb els indicadors d'activitat per a una presa de decisions i d'accions basades en l'evidència.La implantación de las Tecnologías de la Información y la Comunicación (TIC) y la digitalización de la imagen médica supuso un cambio en todo el proceso clínico asistencial por parte de los radiólogos y médicos nucleares encargados de realizar el informe diagnóstico y las intervenciones radiológicas. Actualmente, los sistemas de información conocidos como Picture Archiving and Communication System (PACS), Radiology Information System (RIS), y Hospital Information System (HIS) permiten el manejo de los estudios adquiridos, su informado y el seguimiento de los procesos de gestión asociados a los flujos de la actividad asistencial. A raíz de este profundo cambio han nacido nuevas oportunidades y necesidades que, desafortunadamente, no han sido resueltas e integradas en la mayoría de los entornos hospitalarios. Por ejemplo, gracias a la digitalización de la imagen y a los avances en investigación, es posible obtener datos cuantitativos de la exploración adquirida que reflejen el estado de una enfermedad o el efecto de un fármaco sobre ella. Estas medidas se conocen como biomarcadores de imagen. Por otra parte, actualmente los informes realizados por los médicos especialistas carecen de un esqueleto que impida la variabilidad de contenido y en consecuencia la ausencia potencial de información relevante para el médico peticionario de la exploración. El diseño y desarrollo de los informes estructurados mediante el uso de léxicos normalizados y plantillas es factible con el uso de las TIC. Estos informes deben establecerse para enfermedades y lesiones concretas, y deben estar consensuados entre los médicos especialistas en la imagen y los médicos peticionarios. Debido a los volúmenes de datos generados por los sistemas de información citados anteriormente, se crea continuamente una fuente de conocimiento con datos no explotados. La extracción de este conocimiento a través de indicadores debe permitir visualizar el estado actual de los servicios de radiología y medicina nuclear, de tal forma que se posibilite corregir los cuellos de botella y realizar las acciones correctoras oportunas ante una situación crítica. El propósito final es promover un funcionamiento óptimo de los servicios y facilitar la toma de decisiones en base a los datos de actividad que mejor se ajusten para una mejor atención al paciente. Esta Tesis Doctoral tiene por justificación la integración eficiente de los biomarcadores de imagen, los informes estructurados, y los indicadores de actividad en la práctica asistencial de los servicios de radiología y medicina nuclear. Para lograr este objetivo se utilizaron estándares de imagen médica como el Digital Imaging and Communication in Medicine (DICOM) y otros como eXtensible Language Market (XML). A través de tecnología JAVA se desarrolló una plataforma para la integración de los biomarcadores de imagen. Mediante la utilización de léxicos como el Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) y Radiology Lexicon (RADLEX), más el estándar DICOM y el estándar HTML5, se implantó una aplicación que permite la integración del informe estructurado. De este modo se podrán realizar estudios poblacionales así como analizar la relación de determinados factores con una patología determinada. Por último, de la información obtenida del RIS y del PACS, se construyó una plataforma de indicadores con tecnología JAVA para permitir visualizar el estado de los servicios respecto a la actividad asistencial; actividad de innovación; y actividad científico-docente. Por tanto, la presente Tesis Doctoral aporta a los servicios de radiología y de medicina nuclear una herramienta de innovación en tres campos fundamentales para ofrecer un mejor diagnostico, cuidado y atención al paciente. A través de cuantificación de los biomarcadores de imagen y el informe estructurado para una medicina personalizada, y con los indicadores de actividad para una toma de deThe implementation of Information and Communication Technologies (ICT) and the digitization of medical images implies a change in the entire clinical care process for radiologists and nuclear physicians responsible for the diagnostic report and the radiological interventions. Nowadays, information systems such as Picture Archiving and Communication System (PACS), Radiology Information System (RIS) and Hospital Information System (HIS) enable the management of medical imaging studies, their reports and the follow-up of the management processes associated to the workflow of healthcare activities. These changes (use of ICT and digitization in hospitals) have created new opportunities and needs. Unfortunately, they have not been solved and integrated in most hospital settings. For example, thanks to the digitization of the image and research advances, it is possible to obtain quantitative data from the acquired exploration that reflect the state of a disease or the effect of a drug on it. These measures are known as image biomarkers. Reports made by specialist physicians currently lack of a skeleton that reduces the variability of content and consequently there are potential lack of relevant information for the applicant¿s physician. The design and development of structured reports using standard lexicon and templates is feasible with the use of ICT. These reports should be set up for diseases and specific injuries, and should be agreed upon between specialists and applicant¿s physicians. Due to the volume of data generated by the information systems mentioned above, a source of knowledge is continuously growing but data remains unexploited. The extraction of this knowledge through indicators could enable us to improve the processes in radiology and nuclear medicine departments which should help to correct bottlenecks and take corrective actions in critical situations. The final purpose is to promote the optimal functioning of the services departments and to facilitate the decision-making for a better attention to the patient care based on data activity. This Doctoral Thesis aims at integrating efficiently image biomarkers, structured reports and activity indicators into the clinical practice in the radiology and nuclear medicine departments. To achieve this goal, we have used medical imaging standards such as Digital Imaging and Communication in Medicine (DICOM) and eXtensible Language Market (XML). We used JAVA technology to develop a platform for the integration of image biomarkers. An application was implemented using lexicons such as the Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT), Radiology Lexicon (RADLEX), the DICOM standard and the HTML5 standard. This application allows the integration of the structured report. In this way, it will be possible to make population studies, as well as to analyze the relation of certain factors with a certain disease. Finally, based on RIS and PACS data, a platform was developed using JAVA technology to provide the visualization of the status of key indicators of the performance of radiology departments; innovation activity; and research & teaching activities. This Doctoral Thesis provides the radiology and nuclear medicine departments with an innovative tool in three fundamental fields, offering a better diagnosis, health-care and attention to the patient. This has been done, using the quantification of image biomarkers and the structured report for a personalized medicine, and selecting those indicators of activity to make decisions based on data evidence.Ruiz Martínez, E. (2017). Herramienta de gestión integral en innovación en imagen médica [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/90429TESI