Information systems evaluation methodologies

Due to the prevalent use of Information Systems (IS) in modern organisations nowadays, evaluation research in this field is becoming more and more important. In light of this, a set of rigorous methodologies were developed and used by IS researchers and practitioners to evaluate the increasingly complex IS implementation used. Moreover, different types of IS and different focusing perspectives of the evaluation require the selection and use of different evaluation approaches and methodologies. This paper aims to identify, explore, investigate and discuss the various key methodologies that can be used in IS evaluation from different perspectives, namely in nature (e.g. summative vs. formative evaluation) and in strategy (e.g. goal-based, goal-free and criteria-based evaluation). The paper concludes that evaluation methodologies should be selected depending on the nature of the IS and the specific goals and objectives of the evaluation. Nonetheless, it is also proposed that formative criteria-based evaluation and summative criteria-based evaluation are currently among the most and more widely used in IS research. The authors suggest that the combines used of one or more of these approaches can be applied at different stages of the IS life cycle in order to generate more rigorous and reliable evaluation outcomes

Exploring Interactive Survivorship Plans: Patient Perceived Value, Acceptance and Usability Evaluation of an Online Breast Cancer Survivorship Tool

Introduction: Having recently been discharged from the hospital, several breast cancer survivors find themselves unable to adjust to the transition and take charge of their own health, away from the confines of the hospital. With the rapid advancement in treatment methods and techniques, the rate of breast cancer survivors has grown exponentially. It is crucial to provide adequate means to support cancer survivors in an active manner. This includes regular monitoring for recurrence (or occurrence of new cancers), handling any related and non-related comorbidities, provide recommendations for preventive care as well as dealing with any long term side effects from the treatment. The specific objective of this research is to design and develop a personalized web application to support breast cancer survivors after treatment (chemotherapy and/or radiation), as they deal with post-treatment challenges, such as comorbidities and side-effects of treatment. Methodology: I used an iterative design and development approach to produce a web application for breast cancer survivors that help them monitor their quality of life, provide them with personalized alerts based on their breast cancer related medical history as well as timely alerts, to remind them of follow up visits. Finally, I utilized a combination of qualitative methodology (thematic analysis), as well as user task analysis to assess the acceptability and usability of the prototype among a group of breast cancer survivors. User feedback was gathered on their perceived value of the application, and any user-interface issues that may hinder the overall usability among lay users were identified. Results: Fifteen breast cancer survivors participated in the acceptability and usability testing of the prototype. The prototype was found to be perceived as unique and valuable among the participants, in its ability to utilize personalized breast cancer related medical history. The application’s portability and capability of organizing their entire breast cancer related medical history as well as the at-home tracking of various quality of life indicators were perceived to be valuable features. The application had an overall high usability, however certain sections of the application, such as viewing observations history were not as intuitive to locate. While participants appreciated the visual and graphical elements of the website, the overall experience of the application would benefit from incorporating some sociable elements that exhibit positive re-enforcement within the end user and provide a friendlier and fun experience. Conclusion: The results of the study showcase the need to provide more personalized tools and resources to breast cancer survivors to support them in self-management after completion of treatment. It also demonstrates the ability to integrate breast cancer survivorship plans from diverse providers and paves the way to add further value-added features in consumer health applications, such as personal decision support. The feedback received from end-users will be used in order to further improve the prototype and address any existing user-interface issues. It is hoped that making such tools more accessible could help in engaging survivors to play an active role in managing their health and also encourage shared-decision making with their providers

Harnessing Information Technology to Inform Patients Facing Routine Decisions: Cancer Screening as a Test Case

PURPOSE Technology could transform routine decision making by anticipating patients’ information needs, assessing where patients are with decisions and preferences, personalizing educational experiences, facilitating patient-clinician information exchange, and supporting follow-up. This study evaluated whether patients and clinicians will use such a decision module and its impact on care, using 3 cancer screening decisions as test cases. METHODS Twelve practices with 55,453 patients using a patient portal participated in this prospective observational cohort study. Participation was open to patients who might face a cancer screening decision: women aged 40 to 49 who had not had a mammogram in 2 years, men aged 55 to 69 who had not had a prostate-specific antigen test in 2 years, and adults aged 50 to 74 overdue for colorectal cancer screening. Data sources included module responses, electronic health record data, and a postencounter survey. RESULTS In 1 year, one-fifth of the portal users (11,458 patients) faced a potential cancer screening decision. Among these patients, 20.6% started and 7.9% completed the decision module. Fully 47.2% of module completers shared responses with their clinician. After their next office visit, 57.8% of those surveyed thought their clinician had seen their responses, and many reported the module made their appointment more productive (40.7%), helped engage them in the decision (47.7%), broadened their knowledge (48.1%), and improved communication (37.5%). CONCLUSIONS Many patients face decisions that can be anticipated and proactively facilitated through technology. Although use of technology has the potential to make visits more efficient and effective, cultural, workflow, and technical changes are needed before it could be widely disseminated

Taxonomy of usage problems for improving user-centric online health information provision

Consumer health information portals (HIP) are a popular means to provide quality health information via the Web. However complex usage problems in HIPs are still a major barrier to their success. A usage-driven approach, which places emphasis on improving online services based on learnings from the data of the interactions between users and the system, is crucial to ensuring sustainable and user-centred online health provision. Inspired by this idea, we present a taxonomy of usage problems that encompasses the dimensions of the content, the systems and users, focusing on a holistic understanding of usage problems. Our taxonomy is grounded on a literature analysis empirically validated through an analysis of usage-data captured from a consumer health information portal, operational for the past five years. By exploring how usage data highlights user problems, we also present strategies for health portal improvements based on better understandings of usage data. Benefits of usage-driven health portals in terms of smart learning capabilities to improve content and user satisfaction are discussed

The implications and impact of 3 approaches to health information exchange: community, enterprise, and vendor‐mediated health information exchange

IntroductionElectronic health information exchange (HIE) is considered essential to establishing a learning health system, reducing medical errors, and improving efficiency, but establishment of widespread, high functioning HIE has been challenging. Healthcare organizations now have considerable flexibility in selecting among several HIE strategies, most prominently community HIE, enterprise HIE (led by a healthcare organization), and electronic health record vendor‐mediated HIE. Each of these strategies is characterized by different conveners, capabilities, and motivations and may have different abilities to facilitate improved patient care.MethodsI reviewed the available scholarly literature to draw conceptual distinctions between these types of HIE, to assess the current evidence on each type of HIE, and to indicate important areas of future research.ResultsWhile community HIE seems to offer the most open approach to HIE allowing for high levels of connectivity, both enterprise HIE and vendor‐mediated HIE face lower barriers to formation and sustainability. Most existing evidence is focused on community HIE and points towards low overall use, challenges to usability, and ambiguous impact. To better guide organizational leaders and policymakers in the expansion of beneficial HIE and anticipate future trends, future research should work to better capture the prevalence of other forms of HIE, and to adopt common methods to allow comparisons of rate of use, usability, and impact on patient care across studies and types of HIE.ConclusionsHealthcare organizations’ choice of HIE strategy influences the set of partners the organization is connected to and may influence the benefit that efforts supported by HIE can offer to patients. Current research is not fully capturing the diversity of approaches to HIE and their potentially varying impact on providers and patients.Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/136725/1/lrh210021_am.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/136725/2/lrh210021.pd

Source authenticity in the UMLS – A case study of the Minimal Standard Terminology

AbstractAs the UMLS integrates multiple source vocabularies, the integration process requires that certain adaptation be applied to the source. Our interest is in examining the relationship between the UMLS representation of a source vocabulary and the source vocabulary itself. We investigated the integration of the Minimal Standard Terminology (MST) into the UMLS in order to examine how close its UMLS representation is to the source MST. The MST was conceived as a “minimal” list of terms and structure intended for use within computer systems to facilitate standardized reporting of gastrointestinal endoscopic examinations. Although the MST has an overall schema and implied relationship structure, many of the UMLS integrated MST terms were found to be hierarchically orphaned, and with lateral relationships that do not closely adhere to the source MST. Thus, the MST representation within the UMLS significantly differs from that of the source MST. These representation discrepancies may affect the usability of the MST representation in the UMLS for knowledge acquisition. Furthermore, they pose a problem from the perspective of application developers. While these findings may not necessarily apply to other source terminologies, they highlight the conflict between preservation of authentic concept orientation and the UMLS overall desire to provide fully specified names for all source terms

Design Strategy for Integrated Personal Health Records: Improving the User Experience of Digital Healthcare and Wellbeing

This dissertation addresses the timely problem of designing Integrated Personal Health Records (PHR). The goal is to provide citizens with digital user experiences, sustainable and flexible enough, for gaining control over their personal health information in a seamless way. Most importantly, so that people are able to reflect and act upon their selfknowledge, towards the accomplishment of their good health and wellbeing. Towards this end, the Integrated PHR as an emerging model in the field of Health IT, was the framework that set this research forward on exploring how communication and collaboration between patients and providers can be improved, which naturally impacts the field of HCI. Acknowledging that today patients are the ones who own all that is recorded about their health data, this new model was object of a design strategy that shaped the results presented in this dissertation. These have showed how patients can have more control of their health over time, through a patient-centered, organic system, which has the ability of combining multiple sources of data both from patient and provider side. As this new type of PHR fosters the creation of integrated networks, this milestone was achieved in this research by interacting with cross-channel user experiences that took part of nationwide healthcare ecosystems. The work presented herein, has demonstrated through the analysis and development of two use cases in cooperation with organizations connected to the Portuguese Ministry of Health, how an Integrated PHR can be a powerful personal tool, to be used by the citizen with undeniable value to the demands of an aging society. The use cases structured the thesis into two parts. The first part in collaboration with the Portuguese National Patient Portal, combines an Integrated PHR and incorporates the Portuguese Data Sharing Platform (PDS), which can be used by any Portuguese citizen. This use case study led to a proposal of the portal by also creating a foundational model for designing Integrated PHRs. The second part in collaboration with the Portuguese National Senior Telehealth Program (Saúde 24 Sénior), led to another proposal for an Integrated PHR, applying the outcomes from Part 1 and the requirements that derived from the findings explored in this second use case study. The proposed solution, has the potential to be used by the Portuguese senior community in the scope of home assistive care. Both proposals applied a user experience design methodology and included the development of two prototypes. The engagement of the stakeholders during the two case studies was accomplished with participatory design methods and followed a multidisciplinary approach to create solutions that would meet the human, politics and behavior interdependencies that were inherent to the process of working with large healthcare organizations. The provided contributions from this thesis intent to be part of a transition process that is changing the behavior of the healthcare sector, which is increasingly moving towards the improvement of the patient-provider relationship, patient engagement, collaborative care and positive computing, where digital technologies play a key role

Integrated Health Information Patient Portals: A Gateway to Improved Patient-Provider Interactions

The proposed Stage II Meaningful Use (MU) objectives, would require healthcare organizations to offer patients the ability to access their health related information through a web-based portal. However, MU criteria is just one of the underlying reasons patient portals will become an essential component of thriving healthcare organizations. As the Internet technology permanently altered the way in which we shop, bank, communicate and distribute information, these capabilities have also entered healthcare arena with the promise of improving patient-provider communications and healthcare outcomes. This paper addresses the role of patient portals in meeting the MU objectives, evaluates interest and factors influencing healthcare consumers' utilization of portals, and identifies barriers to adoption of portal technologies by healthcare organizations

MyStay – Development of nurse-facilitated condition-specific multimedia resources to facilitate patient participation in postoperative care

Improved postoperative outcomes and the global drive toward the provision of patient-centred care underpins efforts to enhance the nature and capacity of patient participation in acute postoperative hospital care. In this paper, we describe the design, framework and processes used to develop a modular, procedure-specific, digital health intervention platform aimed at improving the patient experience and patient participation in care following surgery. The intervention, a multimedia application MyStay, uses bedside delivery of audio-visual and text-based information to engage postoperative patients to better participate in their care. MyStay modules are developed using an iterative, multi-method approach intended to balance procedure-specific best evidence, current clinical practice, and patient preferences. Development involves six key elements: (1) Empathise with target users, (2) Ground in evidence and behavioural theory, (3) Specify target behaviours, (4) Integration of health service standards and clinical care pathways/guidelines, (5) Build and refine the multimedia product and, (6) Pilot implementation to assess potential effectiveness and usability. To-date, we have developed four procedure-specific MyStay modules and an additional three are under development. Initial patient usage data for the Total Knee Replacement (TKR) and cardiac surgery applications indicated that users accessed a wide range of text-based and audio-visual information, most frequently recovery goals and exercises following TKR, and postoperative recovery information for the intensive care unit following cardiac surgery. As previous research that tested MyStay indicated its efficacy in optimising clinical postoperative outcomes, this framework may be useful in the development of other digital health innovations. Further research is required to assess patient and clinician engagement and determine whether MyStay is associated with improved patient outcomes across varied clinical contexts. Experience Framework This article is associated with the Innovation & Technology lens of The Beryl Institute Experience Framework (https://www.theberylinstitute.org/ExperienceFramework). Access other PXJ articles related to this lens. Access other resources related to this len