Applying FAHP to Improve the Performance Evaluation Reliability and Validity of Software Defect Classifiers

Today’s Software complexity makes developing defect-free software almost impossible. On an average, billions of dollars are lost every year because of software defects in the United States alone, while the global loss is much higher. Consequently, developing classifiers to classify software modules into defective and non-defective before software releases, has attracted a great interest in academia and the software industry alike. Although many classifiers have been proposed, none has been proven superior to others. The major reason is that while a research shows that classifier-A is better than classifier-B, we can find other research coming to a diametrically opposite conclusion. These conflicts are usually triggered when researchers report results using their preferred performance quality measures such as recall and precision. Although this approach is valid, it does not examine all possible facets of classifiers’ performance characteristics. Thus, performance evaluation might improve or deteriorate if researchers choose other performance measures. As a result, software developers usually struggle to select the most suitable classifier to use in their projects. The goal of this dissertation is to apply the Fuzzy Analytical Hierarchy Process (FAHP) as a popular multi-criteria decision-making technique to overcome these inconsistencies in research outcomes. This evaluation framework incorporates a wider spectrum of performance measures to evaluate classifiers’ performance, rather than relying on selected, preferred measures. The results show that this approach will increase software developers’ confidence in research outcomes, help them in avoiding false conclusions and indicate reasonable boundaries for them. We utilized 22 popular performance measures and 11 software defect classifiers. The analysis was carried out using KNIME data mining platform and 12 software defect data sets provided by NASA Metrics Data Program (MDP) repository

Development of a multi-criteria collaborative decision model for performance management in networks of organisations

Páginas numeradas; I-XIII, 14-117Tese de mestrado. Gestão de Informação. Faculdade de Engenharia. Universidade do Porto. 200

Improving the process of analysis and comparison of results in dependability benchmarks for computer systems

Tesis por compendioLos dependability benchmarks (o benchmarks de confiabilidad en español), están diseñados para evaluar, mediante la categorización cuantitativa de atributos de confiabilidad y prestaciones, el comportamiento de sistemas en presencia de fallos. En este tipo de benchmarks, donde los sistemas se evalúan en presencia de perturbaciones, no ser capaces de elegir el sistema que mejor se adapta a nuestras necesidades puede, en ocasiones, conllevar graves consecuencias (económicas, de reputación, o incluso de pérdida de vidas). Por esa razón, estos benchmarks deben cumplir ciertas propiedades, como son la no-intrusión, la representatividad, la repetibilidad o la reproducibilidad, que garantizan la robustez y precisión de sus procesos. Sin embargo, a pesar de la importancia que tiene la comparación de sistemas o componentes, existe un problema en el ámbito del dependability benchmarking relacionado con el análisis y la comparación de resultados. Mientras que el principal foco de investigación se ha centrado en el desarrollo y la mejora de procesos para obtener medidas en presencia de fallos, los aspectos relacionados con el análisis y la comparación de resultados quedaron mayormente desatendidos. Esto ha dado lugar a diversos trabajos en este ámbito donde el proceso de análisis y la comparación de resultados entre sistemas se realiza de forma ambigua, mediante argumentación, o ni siquiera queda reflejado. Bajo estas circunstancias, a los usuarios de los benchmarks se les presenta una dificultad a la hora de utilizar estos benchmarks y comparar sus resultados con los obtenidos por otros usuarios. Por tanto, extender la aplicación de los benchmarks de confiabilidad y realizar la explotación cruzada de resultados es una tarea actualmente poco viable. Esta tesis se ha centrado en el desarrollo de una metodología para dar soporte a los desarrolladores y usuarios de benchmarks de confiabilidad a la hora de afrontar los problemas existentes en el análisis y comparación de resultados. Diseñada para asegurar el cumplimiento de las propiedades de estos benchmarks, la metodología integra el proceso de análisis de resultados en el flujo procedimental de los benchmarks de confiabilidad. Inspirada en procedimientos propios del ámbito de la investigación operativa, esta metodología proporciona a los evaluadores los medios necesarios para hacer su proceso de análisis explícito, y más representativo para el contexto dado. Los resultados obtenidos de aplicar esta metodología en varios casos de estudio de distintos dominios de aplicación, mostrará las contribuciones de este trabajo a mejorar el proceso de análisis y comparación de resultados en procesos de evaluación de la confiabilidad para sistemas basados en computador.Dependability benchmarks are designed to assess, by quantifying through quantitative performance and dependability attributes, the behavior of systems in presence of faults. In this type of benchmarks, where systems are assessed in presence of perturbations, not being able to select the most suitable system may have serious implications (economical, reputation or even lost of lives). For that reason, dependability benchmarks are expected to meet certain properties, such as non-intrusiveness, representativeness, repeatability or reproducibility, that guarantee the robustness and accuracy of their process. However, despite the importance that comparing systems or components has, there is a problem present in the field of dependability benchmarking regarding the analysis and comparison of results. While the main focus in this field of research has been on developing and improving experimental procedures to obtain the required measures in presence of faults, the processes involving the analysis and comparison of results were mostly unattended. This has caused many works in this field to analyze and compare results of different systems in an ambiguous way, as the process followed in the analysis is based on argumentation, or not even present. Hence, under these circumstances, benchmark users will have it difficult to use these benchmarks and compare their results with those from others. Therefore extending the application of these dependability benchmarks and perform cross-exploitation of results among works is not likely to happen. This thesis has focused on developing a methodology to assist dependability benchmark performers to tackle the problems present in the analysis and comparison of results of dependability benchmarks. Designed to guarantee the fulfillment of dependability benchmark's properties, this methodology seamlessly integrates the process of analysis of results within the procedural flow of a dependability benchmark. Inspired on procedures taken from the field of operational research, this methodology provides evaluators with the means not only to make their process of analysis explicit to anyone, but also more representative for the context being. The results obtained from the application of this methodology to several case studies in different domains, will show the actual contributions of this work to improving the process of analysis and comparison of results in dependability benchmarking for computer systems.Els dependability benchmarks (o benchmarks de confiabilitat, en valencià), són dissenyats per avaluar, mitjançant la categorització quantitativa d'atributs de confiabilitat i prestacions, el comportament de sistemes en presència de fallades. En aquest tipus de benchmarks, on els sistemes són avaluats en presència de pertorbacions, el no ser capaços de triar el sistema que millor s'adapta a les nostres necessitats pot tenir, de vegades, greus conseqüències (econòmiques, de reputació, o fins i tot pèrdua de vides). Per aquesta raó, aquests benchmarks han de complir certes propietats, com són la no-intrusió, la representativitat, la repetibilitat o la reproductibilitat, que garanteixen la robustesa i precisió dels seus processos. Així i tot, malgrat la importància que té la comparació de sistemes o components, existeix un problema a l'àmbit del dependability benchmarking relacionat amb l'anàlisi i la comparació de resultats. Mentre que el principal focus d'investigació s'ha centrat en el desenvolupament i la millora de processos per a obtenir mesures en presència de fallades, aquells aspectes relacionats amb l'anàlisi i la comparació de resultats es van desatendre majoritàriament. Açò ha donat lloc a diversos treballs en aquest àmbit on els processos d'anàlisi i comparació es realitzen de forma ambigua, mitjançant argumentació, o ni tan sols queden reflectits. Sota aquestes circumstàncies, als usuaris dels benchmarks se'ls presenta una dificultat a l'hora d'utilitzar aquests benchmarks i comparar els seus resultats amb els obtinguts per altres usuaris. Per tant, estendre l'aplicació dels benchmarks de confiabilitat i realitzar l'explotació creuada de resultats és una tasca actualment poc viable. Aquesta tesi s'ha centrat en el desenvolupament d'una metodologia per a donar suport als desenvolupadors i usuaris de benchmarks de confiabilitat a l'hora d'afrontar els problemes existents a l'anàlisi i comparació de resultats. Dissenyada per a assegurar el compliment de les propietats d'aquests benchmarks, la metodologia integra el procés d'anàlisi de resultats en el flux procedimental dels benchmarks de confiabilitat. Inspirada en procediments propis de l'àmbit de la investigació operativa, aquesta metodologia proporciona als avaluadors els mitjans necessaris per a fer el seu procés d'anàlisi explícit, i més representatiu per al context donat. Els resultats obtinguts d'aplicar aquesta metodologia en diversos casos d'estudi de distints dominis d'aplicació, mostrarà les contribucions d'aquest treball a millorar el procés d'anàlisi i comparació de resultats en processos d'avaluació de la confiabilitat per a sistemes basats en computador.Martínez Raga, M. (2018). Improving the process of analysis and comparison of results in dependability benchmarks for computer systems [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/111945TESISCompendi

A technique for the evaluation of free and open source e-learning systems

Philosophiae Doctor - PhDEvaluating software is a universal and complex problem. The question is: how should software be selected and adopted, or rather, which of the software packages is the most suitable for a specific environment? Extensive research on the evaluation of software has been done, but only a few researchers have considered evaluation of e-learning systems based on three software quality characteristics (i.e. usability, maintainability and deployability) for implementation in third world countries. In this thesis, it will be considered how to use a mixed research methods for the evaluation of free and open source e-learning systems in a developing country. The scope of this investigation is the evaluation of two free and open source e-learning systems at the Open University of Tanzania using 33 stakeholders (some with more and others with less computer expertise).South Afric

Development of a project complexity assessment method for energy megaprojects

Megaprojects are characterised by their large-scale capital expenditure, long duration and significant levels of technical and process complexity. Empirical data show that megaprojects in the energy sector experience alarming rates of failure, such as cost overruns, delays in completion and production shortfalls. One of the main causes of failure is their high level of complexity and the absence of effective tools to assess and manage it. Project complexity has received increasing attention in recent years, both in academia and the industry. However, there is still a lack of consensus on a clear definition for ‘project complexity’ or a comprehensive list of complexity indicators, specifically for energy megaprojects. Furthermore, there is also a lack of a widely accepted assessment method to measure project complexity in a quantitative manner. This study is carried out in response to these problems. First, it develops a taxonomy of project complexity indicators on the basis of a comprehensive review and synthesis of existing literature. It includes 51 internal and external Project Complexity Indicators (PCIs) in a logical hierarchical structure; these indicators specify the aspects that need to be measured when assessing project complexity. Second, weights for all indicators are established through an integrated Delphi-AHP method, with the participation of 20 international experts. Finally, the study specifies Numerical Scoring Criteria (NSCs) for all indicators based on a synthesis of existing knowledge about megaprojects. The criteria specify the scoring thresholds, on a 1-5 scale, for each indicator. These three components constitute a new Project Complexity Assessment (PCA) method, which is implemented as a spreadsheet PCA tool. The developed tool allows a project team to assess and score their project in each of the PCIs against the defined criteria. It then calculates two separate complexity indices for internal and external factors; the results indicate the complexity level of the project. Complexity profiles are also produced to illustrate the complexity scores of different categories of PCIs. The PCA method is tested using an energy megaproject case study. The results demonstrate not only that the tool can help a project team understand the complexity of their project, but also it can help the team to develop appropriate complexity management strategies by comparing the assessment results of different projects

Evaluating the performance of airports using an integrated AHP/DEA-AR technique

Airport efficiency is an area of increasing interest to academics, policy makers and practitioners. This has resulted in a body of literature applying various econometric techniques to compare efficiency between different samples of airports. This paper uses the multi-criteria decision making method Analytic Hierarchy Process (AHP) to incorporate the weightings of input and output variables into Data Envelopment Analysis (DEA) and Assurance Region DEA (DEA-AR) models, with 24 major international airports in the empirical analysis. The paper concludes the discriminatory power in the proposed AHP/DEA-AR model is greater than in the basic DEA model when measuring the efficiency of airports. By applying this approach, policy makers and practitioners can effectively compare operational efficiency between airports, and therefore generate more informed decisions

An Investigation into Factors Affecting the Chilled Food Industry

With the advent of Industry 4.0, many new approaches towards process monitoring, benchmarking and traceability are becoming available, and these techniques have the potential to radically transform the agri-food sector. In particular, the chilled food supply chain (CFSC) contains a number of unique challenges by virtue of it being thought of as a temperature controlled supply chain. Therefore, once the key issues affecting the CFSC have been identified, algorithms can be proposed, which would allow realistic thresholds to be established for managing these problems on the micro, meso and macro scales. Hence, a study is required into factors affecting the CFSC within the scope of Industry 4.0. The study itself has been broken down into four main topics: identifying the key issues within the CFSC; implementing a philosophy of continuous improvement within the CFSC; identifying uncertainty within the CFSC; improving and measuring the performance of the supply chain. However, as a consequence of this study two further topics were added: a discussion of some of the issues surrounding information sharing between retailers and suppliers; some of the wider issues affecting food losses and wastage (FLW) on the micro, meso and macro scales. A hybrid algorithm is developed, which incorporates the analytic hierarchical process (AHP) for qualitative issues and data envelopment analysis (DEA) for quantitative issues. The hybrid algorithm itself is a development of the internal auditing algorithm proposed by Sueyoshi et al (2009), which in turn was developed following corporate scandals such as Tyco, Enron, and WorldCom, which have led to a decline in public trust. However, the advantage of the proposed solution is that all of the key issues within the CFSC identified can be managed from a single computer terminal, whilst the risk of food contamination such as the 2013 horsemeat scandal can be avoided via improved traceability

A Customer Value Assessment Process (CVAP) for Ballistic Missile Defense

A systematic customer value assessment process (CVAP) was developed to give system engineering teams the capability to qualitatively and quantitatively assess customer values. It also provides processes and techniques used to create and identify alternatives, evaluate alternatives in terms of effectiveness, cost, and risk. The ultimate goal is to provide customers (or decision makers) with objective and traceable procurement recommendations. The creation of CVAP was driven by an industry need to provide ballistic missile defense (BMD) customers with a value proposition of contractors’ BMD systems. The information that outputs from CVAP can be used to guide BMD contractors in formulating a value proposition, which is used to steer customers to procure their BMD system(s) instead of competing system(s). The outputs from CVAP also illuminate areas where systems can be improved to stay relevant with customer values by identifying capability gaps. CVAP incorporates proven approaches and techniques appropriate for military applications. However, CVAP is adaptable and may be applied to business, engineering, and even personal every-day decision problems and opportunities. CVAP is based on the systems decision process (SDP) developed by Gregory S. Parnell and other systems engineering faculty at the Unites States Military Academy (USMA). SDP combines Value-Focused Thinking (VFT) decision analysis philosophy with Multi-Objective Decision Analysis (MODA) quantitative analysis of alternatives. CVAP improves SDP’s qualitative value model by implementing Quality Function Deployment (QFD), solution design implements creative problem solving techniques, and the qualitative value model by adding cost analysis and risk assessment processes practiced by the U.S DoD and industry. CVAP and SDP fundamentally differ from other decision making approaches, like the Analytic Hierarchy Process (AHP) and the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), by distinctly separating the value/utility function assessment process with the ranking of alternatives. This explicit value assessment allows for straightforward traceability of the specific factors that influence decisions, which illuminates the tradeoffs involved in making decisions with multiple objectives. CVAP is intended to be a decision support tool with the ultimate purpose of helping decision makers attain the best solution and understanding the differences between the alternatives. CVAP does not include any processes for implementation of the alternative that the customer selects. CVAP is applied to ballistic missile defense (BMD) to give contractors ideas on how to use it. An introduction of BMD, unique BMD challenges, and how CVAP can improve the BMD decision making process is presented. Each phase of CVAP is applied to the BMD decision environment. CVAP is applied to a fictitious BMD example

Analysing Stakeholder Consensus for a Sustainable Transport Development Decision by the Fuzzy AHP and Interval AHP

In any public service development decision, it is essential to reach the stakeholders&rsquo agreement to gain a sustainable result, which is accepted by all involved groups. In case this criterion is violated, the impact of the development will be less than expected due to the resistance of one group or another. Concerning public urban transport decisions, the lack of consensus might cause lower utilisation of public vehicles, thus more severe environmental damage, traffic problems and negative economic impacts. This paper aims to introduce a decision support procedure (applying the current MCDM techniques Fuzzy and Interval AHP) which is capable of analysing and creating consensus among different stakeholder participants in a transport development problem. The combined application of FAHP and IAHP ensures that the consensus creation is not only based on an automated computation process (just as in IAHP) but also on the consideration of specific group interests. Thus, the decision makers have the liberty to express their preferences in urban planning, along with the consideration of numerical results. The procedure has been tested in a real public transport improvement decision as a follow-up project, in an emerging city, Mersin, Turkey. Results show that by the application of the proposed techniques, decision-makers can be more aware of the conflicts of interests among the involved groups, and they can pay more attention to possible violations. Document type: Articl