Special Theme of Research in Information Systems Analysis and Design-IV Evaluation Criteria for Information Systems Development Methodologies

The evaluation of information systems development methodologies is becoming increasingly important. Some researchers propose their own criteria for conceptual evaluations. But such criteria are often constrained by the limitation of the researcher\u27s view toward and experience with development methodologies. Furthermore, existing evaluation criteria are either not practical for direct measurement or not tested for validity and reliability. The lack of a generally acceptable, practically valid, and reliable set of criteria for evaluation hinders the development of knowledge in this area. Our study is a step toward developing a systematic process to evaluate information systems development methodologies. We captured the opinions of a group of twenty-eight researchers and practitioners who are experienced in information systems development methodologies. Through a systematic content analysis, the authors classified these criteria into three categories: (1) methodology design, (2) methodology use, and (3) methodology deliverables. The three categories of criteria are not totally separate in that they represent the means and goals of information systems development methodologies. Appropriate criteria should be applied depending on the stages when evaluation is performed

An Information Security Control Assessment Methodology for Organizations

In an era where use and dependence of information systems is significantly high, the threat of incidents related to information security that could jeopardize the information held by organizations is more and more serious. Alarming facts within the literature point to inadequacies in information security practices, particularly the evaluation of information security controls in organizations. Research efforts have resulted in various methodologies developed to deal with the information security controls assessment problem. A closer look at these traditional methodologies highlights various weaknesses that can prevent an effective information security controls assessment in organizations. This dissertation develops a methodology that addresses such weaknesses when evaluating information security controls in organizations. The methodology, created using the Fuzzy Logic Toolbox of MATLAB based on fuzzy theory and fuzzy logic, uses fuzzy set theory which allows for a more accurate assessment of imprecise criteria than traditional methodologies. It is argued and evidenced that evaluating information security controls using fuzzy set theory addresses existing weaknesses found in the literature for traditional evaluation methodologies and, thus, leads to a more thorough and precise assessment. This, in turn, results in a more effective selection of information security controls and enhanced information security in organizations. The main contribution of this research to the information security literature is the development of a fuzzy set theory-based assessment methodology that provides for a thorough evaluation of ISC in organizations. The methodology just created addresses the weaknesses or limitations identified in existing information security control assessment methodologies, resulting in an enhanced information security in organizations. The methodology can also be implemented in a spreadsheet or software tool, and promote usage in practical scenarios where highly complex methodologies for ISC selection are impractical. Moreover, the methodology fuses multiple evaluation criteria to provide a holistic view of the overall quality of information security controls, and it is easily extended to include additional evaluation criteria factor not considered within this dissertation. This is one of the most meaningful contributions from this dissertation. Finally, the methodology provides a mechanism to evaluate the quality of information security controls in various domains. Overall, the methodology presented in this dissertation proved to be a feasible technique for evaluating information security controls in organizations

Cross disciplinary evaluation framework for e-health services

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel UniversityE-health is an emerging field in the intersection of information systems, healthcare and business management, referring mainly to healthcare services delivered and enhanced through the use of information and communication technologies (ICT). In a broader sense, the term characterizes not only a technical development, but also a wider way of thinking, an attitude, and a commitment for a network to improve and connect provider, patients and governments. Such a network will be used to educate and inform healthcare professionals, managers and healthcare users; to stimulate innovation in care delivery and health system management; and to improve the healthcare system locally, regionally, and globally. The evaluation of e-health services in both theory and practice has proved to be important and complex. E-health evaluation will help achieve better user services utilization, justify the enormous investments of governments on delivering e-health services, and address the aspects that are hampering healthcare services from embracing the full potential of ICT towards successful e-health initiatives. The complexity of evaluation is mostly due to the challenges faced at the intersection of three areas, each well-known for its complexity; healthcare services, information systems, and evaluation methodologies. However, despite the importance of the evaluation of e-health services, literature shows that e-health evaluation is still in its infancy in terms of development and management. The aim of this research study is to develop, and assess a cross disciplinary evaluation framework for e-health services and to propose evaluation criteria for better user’s utilization and satisfaction of e-health services. The evaluation framework is criteria based, while the criteria are determined by an evaluation matrix of three elements, the evaluation rationales, the evaluation timeframes, and the evaluation stakeholders. The evaluation criteria have to be multi-dimensional as well as grounded in, or derived from, one or more specific perspectives or theories. The framework is designed to deal effectively with the challenges of e-health evaluation and overcome the limitation of existing evaluation frameworks. The cross disciplinary evaluation framework has been examined and validated by adopting an interpretive case study methodology. The chosen case study is NHS direct which is currently one of the largest e-health services in the world. The data collection process has been carried out by using three research methods; archival records, documentation analysis and semi-structured interviews. The use of multiple methods is essential to generate comparable data patterns and structures, and enhance the reliability of conclusions through data triangulation. The contribution of the research study is in bridging the gap between the theory and practice in the evaluation of e-health services by providing an efficient evaluation framework that can be applied to a wide range of e-health application and able to answer real-world concerns. The study also offers three sets of well-argued and balanced hierarchies of evaluation criteria that influence user’s utilization and satisfaction of e-health services. The evaluation criteria can be used to help achieve better user services utilization, to serve as part of e-health evaluation framework, and to address areas that require further attention in the development of future e-health initiatives

Requirements engineering for computer integrated environments in construction

A Computer Integrated Environment (CIE) is the type of innovative integrated information system that helps to reduce fragmentation and enables the stakeholders to collaborate together in business. Researchers have observed that the concept of CIE has been the subject of research for many years but the uptake of this technology has been very limited because of the development of the technology and its effective implementation. Although CIE is very much valued by both industrialists and academics, the answers to the question of how to develop and how to implement it are still not clear. The industrialists and researchers conveyed that networking, collaboration, information sharing and communication will become popular and critical issues in the future, which can be managed through CIE systems. In order for successful development of the technology, successful delivery, and effective implementation of user and industry-oriented CIE systems, requirements engineering seems a key parameter. Therefore, through experiences and lessons learnt in various case studies of CIE systems developments, this book explains the development of a requirements engineering framework specific to the CIE system. The requirements engineering process that has been developed in the research is targeted at computer integrated environments with a particular interest in the construction industry as the implementation field. The key features of the requirements engineering framework are the following: (1) ready-to-use, (2) simple, (3) domain specific, (4) adaptable and (5) systematic, (6) integrated with the legacy systems. The method has three key constructs: i) techniques for requirements development, which includes the requirement elicitation, requirements analysis/modelling and requirements validation, ii) requirements documentation and iii) facilitating the requirements management. It focuses on system development methodologies for the human driven ICT solutions that provide communication, collaboration, information sharing and exchange through computer integrated environments for professionals situated in discrete locations but working in a multidisciplinary and interdisciplinary environment. The overview for each chapter of the book is as follows; Chapter 1 provides an overview by setting the scene and presents the issues involved in requirements engineering and CIE (Computer Integrated Environments). Furthermore, it makes an introduction to the necessity for requirements engineering for CIE system development, experiences and lessons learnt cumulatively from CIE systems developments that the authors have been involved in, and the process of the development of an ideal requirements engineering framework for CIE systems development, based on the experiences and lessons learnt from the multi-case studies. Chapter 2 aims at building up contextual knowledge to acquire a deeper understanding of the topic area. This includes a detailed definition of the requirements engineering discipline and the importance and principles of requirements engineering and its process. In addition, state of the art techniques and approaches, including contextual design approach, the use case modelling, and the agile requirements engineering processes, are explained to provide contextual knowledge and understanding about requirements engineering to the readers. After building contextual knowledge and understanding about requirements engineering in chapter 2, chapter 3 attempts to identify a scope and contextual knowledge and understanding about computer integrated environments and Building Information Modelling (BIM). In doing so, previous experiences of the authors about systems developments for computer integrated environments are explained in detail as the CIE/BIM case studies. In the light of contextual knowledge gained about requirements engineering in chapter 2, in order to realize the critical necessity of requirements engineering to combine technology, process and people issues in the right balance, chapter 4 will critically evaluate the requirements engineering activities of CIE systems developments that are explained in chapter 3. Furthermore, to support the necessity of requirements engineering for human centred CIE systems development, the findings from semi-structured interviews are shown in a concept map that is also explained in this chapter. In chapter 5, requirements engineering is investigated from different angles to pick up the key issues from discrete research studies and practice such as traceability through process and product modelling, goal-oriented requirements engineering, the essential and incidental complexities in requirements models, the measurability of quality requirements, the fundamentals of requirements engineering, identifying and involving the stakeholders, reconciling software requirements and system architectures and barriers to the industrial uptake of requirements engineering. In addition, a comprehensive research study measuring the success of requirements engineering processes through a set of evaluation criteria is introduced. Finally, the key issues and the criteria are comparatively analyzed and evaluated in order to match each other and confirm the validity of the criteria for the evaluation and assessment of the requirements engineering implementation in the CIE case study projects in chapter 7 and the key issues will be used in chapter 9 to support the CMM (Capability Maturity Model) for acceptance and wider implications of the requirements engineering framework to be proposed in chapter 8. Chapter 6 explains and particularly focuses on how the requirements engineering activities in the case study projects were handled by highlighting strengths and weaknesses. This will also include the experiences and lessons learnt from these system development practices. The findings from these developments will also be utilized to support the justification of the necessity of a requirements engineering framework for the CIE systems developments. In particular, the following are addressed. • common and shared understanding in requirements engineering efforts, • continuous improvement, • outputs of requirement engineering • reflections and the critical analysis of the requirements engineering approaches in these practices. The premise of chapter 7 is to evaluate and assess the requirements engineering approaches in the CIE case study developments from multiple viewpoints in order to find out the strengths and the weaknesses in these requirements engineering processes. This evaluation will be mainly based on the set of criteria developed by the researchers and developers in the requirements engineering community in order to measure the success rate of the requirements engineering techniques after their implementation in the various system development projects. This set of criteria has already been introduced in chapter 5. This critical assessment includes conducting a questionnaire based survey and descriptive statistical analysis. In chapter 8, the requirements engineering techniques tested in the CIE case study developments are composed and compiled into a requirements engineering process in the light of the strengths and the weaknesses identified in the previous chapter through benchmarking with a Capability Maturity Model (CMM) to ensure that it has the required level of maturity for implementation in the CIE systems developments. As a result of this chapter, a framework for a generic requirements engineering process for CIE systems development will be proposed. In chapter 9, the authors will discuss the acceptance and the wider implications of the proposed framework of requirements engineering process using the CMM from chapter 8 and the key issues from chapter 5. Chapter 10 is the concluding chapter and it summarizes the findings and brings the book to a close with recommendations for the implementation of the Proposed RE framework and also prescribes a guideline as a way forward for better implementation of requirements engineering for successful developments of the CIE systems in the future

Python Library for Consumer Decision Support System with Automatic Identification of Preferences

The development of information systems (IS) has increased in the e-commerce field. The need for continuous improvement of decision support systems implies the integration of multiple methodologies such as expert knowledge, data mining, big data, artificial intelligence, and multicriteria decision analysis (MCDA) methods. Artificial intelligence algorithms have proven their effectiveness as an engine for data-driven information systems. MCDA methods demonstrated usefulness in domains dealing with multiple dimensions. One of the most critical points of any MCDA procedure is criteria weighting using subjective or objective methods. However, both approaches have several limitations when there is a need to map the preferences of unavailable experts. EVO-SPOTIS library integrating a stochastic evolutionary algorithm with the MCDA method, introduced in this paper, attempts to address this problem. In this approach, the Differential Evolution (DE) algorithm is used to identify decision-makers’ preferences based on datasets evaluated by experts in the past. The Stable Preference Ordering Towards Ideal Solution (SPOTIS) method is used to compute the DE objective function’s values and perform the final evaluation of alternatives using the identified weights. Results confirm the high potential of the library for identification preferences and modeling customer behavior

Information systems evaluation methodologies

Due to the prevalent use of Information Systems (IS) in modern organisations nowadays, evaluation research in this field is becoming more and more important. In light of this, a set of rigorous methodologies were developed and used by IS researchers and practitioners to evaluate the increasingly complex IS implementation used. Moreover, different types of IS and different focusing perspectives of the evaluation require the selection and use of different evaluation approaches and methodologies. This paper aims to identify, explore, investigate and discuss the various key methodologies that can be used in IS evaluation from different perspectives, namely in nature (e.g. summative vs. formative evaluation) and in strategy (e.g. goal-based, goal-free and criteria-based evaluation). The paper concludes that evaluation methodologies should be selected depending on the nature of the IS and the specific goals and objectives of the evaluation. Nonetheless, it is also proposed that formative criteria-based evaluation and summative criteria-based evaluation are currently among the most and more widely used in IS research. The authors suggest that the combines used of one or more of these approaches can be applied at different stages of the IS life cycle in order to generate more rigorous and reliable evaluation outcomes

Iterative criteria-based approach to engineering the requirements of software development methodologies

Software engineering endeavours are typically based on and governed by the requirements of the target software; requirements identification is therefore an integral part of software development methodologies. Similarly, engineering a software development methodology (SDM) involves the identification of the requirements of the target methodology. Methodology engineering approaches pay special attention to this issue; however, they make little use of existing methodologies as sources of insight into methodology requirements. The authors propose an iterative method for eliciting and specifying the requirements of a SDM using existing methodologies as supplementary resources. The method is performed as the analysis phase of a methodology engineering process aimed at the ultimate design and implementation of a target methodology. An initial set of requirements is first identified through analysing the characteristics of the development situation at hand and/or via delineating the general features desirable in the target methodology. These initial requirements are used as evaluation criteria; refined through iterative application to a select set of relevant methodologies. The finalised criteria highlight the qualities that the target methodology is expected to possess, and are therefore used as a basis for de. ning the final set of requirements. In an example, the authors demonstrate how the proposed elicitation process can be used for identifying the requirements of a general object-oriented SDM. Owing to its basis in knowledge gained from existing methodologies and practices, the proposed method can help methodology engineers produce a set of requirements that is not only more complete in span, but also more concrete and rigorous

An In-Depth International Comparison of Major Donor Agencies: How Do They Systematically Conduct Country Program Evaluation?

Background: This paper presents an in-depth international comparison of systems and procedures of aid evaluation, focusing on Country Program Evaluation among major donor agencies. The original client of this study is Ministry of Foreign Affairs, Japan (MOFAJ).   Purpose: The purposes of this paper are set as follows: (1) to understand how aid agencies conduct Country Program Evaluation; and (2) to make recommendations for improvement of the current practice of Country Program Evaluation in the aid evaluation community.   Setting: The examined donors include: the World Bank (WB), the Asian Development Bank (ADB), the Inter-American Development Bank (IADB), the United Nations Development Programme (UNDP), the U.S. (USAID), Canada (CIDA), the U.K. (DFID), the Netherlands (IOB), Germany (BMZ), France (Foreign Ministry), and Japan (Ministry of Foreign Affairs (MOFAJ)). In addition, aid agencies conducting respective project evaluation are also examined, and they are JICA (Japan), GTZ and KfW (Germany) and AFD (France). Intervention: This study presents the result of comparative analysis among those donor agencies in terms of the following viewpoints: (1) evaluation criteria employed; (2) approaches to evaluate “effectiveness” and “impact”; (3) attribution issue; (4) the use of a rating system;  and (5) overall evaluative conclusion and integrating methods. All viewpoints are focusing on Country Program Evaluation. One conclusion is that most agencies have been struggling with how to judge the degree and value of their country programs.   Data Collection and Analysis: Mixed methodologies were employed to collect data from the said donor agencies. The analysis was conducted by a systematic procedure consisting of: (i) summarizing information in a comparative table; (ii) trying to make groups/categories based on common characteristics if possible; and (iii) examining and concluding basic thoughts/philosophy which make their differences.   Findings: This study made some new knowledge about how aid agencies conduct Country Program Evaluation and identified several issues remained. Varieties of their practices are observed and it is far from the unified methods agreed. Some remarkable points identified in this study are:(1) Most aid agencies invoke the DAC five evaluation criteria for Country Program Evaluation. (Major exception was USAID); (2) “Strategic relevance” and “coherence/complementarity” are the emerging new criteria; (3) Attribution is still the issue that aid agencies have struggled; and (4) The attitude for introduction of rating system is clearly divided among aid agencies.  &nbsp