An Evaluation Framework for Data Quality Tools

International audienceData Quality is a major stake for large organizations and software companies are proposing increasing numbers of tools focusing on these issues. The scope of these tools is moving from specific applications (deduplication, address normalization etc ...) to a more global perspective integrating all areas of data quality (profiling, rule-detection...). A framework is needed to help managers to choose this type of tool. In this article, we focus on tool-functionalities which aim to measure the quality of data(bases). We explain what one can expect of such functionalities in a CRM context, and we propose a general matrix which can be used for the evaluation and comparison of these tools

Integrated Framework for Data Quality and Security Evaluation on Mobile Devices

Data quality (DQ) is an important concept that is used in the design and employment of information, data management, decision making, and engineering systems with multiple applications already available for solving specific problems. Unfortunately, conventional approaches to DQ evaluation commonly do not pay enough attention or even ignore the security and privacy of the evaluated data. In this research, we develop a framework for the DQ evaluation of the sensor originated data acquired from smartphones, that incorporates security and privacy aspects into the DQ evaluation pipeline. The framework provides support for selecting the DQ metrics and implementing their calculus by integrating diverse sensor data quality and security metrics. The framework employs a knowledge graph to facilitate its adaptation in new applications development and enables knowledge accumulation. Privacy aspects evaluation is demonstrated by the detection of novel and sophisticated attacks on data privacy on the example of colluded applications attack recognition. We develop multiple calculi for DQ and security evaluation, such as a hierarchical fuzzy rules expert system, neural networks, and an algebraic function. Case studies that demonstrate the framework\u27s performance in solving real-life tasks are presented, and the achieved results are analyzed. These case studies confirm the framework\u27s capability of performing comprehensive DQ evaluations. The framework development resulted in producing multiple products, and tools such as datasets and Android OS applications. The datasets include the knowledge base of sensors embedded in modern mobile devices and their quality analysis, technological signals recordings of smartphones during the normal usage, and attacks on users\u27 privacy. These datasets are made available for public use and can be used for future research in the field of data quality and security. We also released under an open-source license a set of Android OS tools that can be used for data quality and security evaluation

Electronic immunization data collection systems: application of an evaluation framework

BACKGROUND: Evaluating the features and performance of health information systems can serve to strengthen the systems themselves as well as to guide other organizations in the process of designing and implementing surveillance tools. We adapted an evaluation framework in order to assess electronic immunization data collection systems, and applied it in two Ontario public health units. METHODS: The Centers for Disease Control and Prevention’s Guidelines for Evaluating Public Health Surveillance Systems are broad in nature and serve as an organizational tool to guide the development of comprehensive evaluation materials. Based on these Guidelines, and informed by other evaluation resources and input from stakeholders in the public health community, we applied an evaluation framework to two examples of immunization data collection and examined several system attributes: simplicity, flexibility, data quality, timeliness, and acceptability. Data collection approaches included key informant interviews, logic and completeness assessments, client surveys, and on-site observations. RESULTS: Both evaluated systems allow high-quality immunization data to be collected, analyzed, and applied in a rapid fashion. However, neither system is currently able to link to other providers’ immunization data or provincial data sources, limiting the comprehensiveness of coverage assessments. We recommended that both organizations explore possibilities for external data linkage and collaborate with other jurisdictions to promote a provincial immunization repository or data sharing platform. CONCLUSIONS: Electronic systems such as the ones described in this paper allow immunization data to be collected, analyzed, and applied in a rapid fashion, and represent the infostructure required to establish a population-based immunization registry, critical for comprehensively assessing vaccine coverage

Introduction of the Methods and Models of University Website Evaluation

Among different tools, university website is one of the cost-effective and convenient methods for communicating with users; it is also a platform for institutions or organizations to achieve their goals. Universities should work on their websites to keep positive images in their public minds. This study aimed to introduce overlapping various types of university website evaluation methods and models. Therefore, content analysis was adopted as the research methodology. Then, scientific databases were searched using the related keywords and extracting the published articles in the context of the university website evaluation. The results of this study showed that university website evaluation models and methods included Quality Web Model Development, Webqem, WAI, Webometrics, E-Qual, University Websites Evaluation and Credibility (UWEAC), University Website Evaluation Framework (UWEF), Development and Validation of the University Website Evaluation Scale (UWES), a Web Usability Evaluation Model for Higher Education (WUEM), University Websites Quality Web Model Development, University Portals Data Quality assessment Framework (UPDQAF), and the University Website Usability Evaluation checklist (UWUE): An exploratory study. In fact, webometrics is the only method that is distinct from other assessment methods. Moreover, UWEAC model with the highest average (0.194) has the most overlap with others. Since the University Websites Evaluation and Credibility (UWEAC) are the comprehensive model, university website designers and evaluators could consider this approach as a preferable model in the university website evaluation but they should ensure that most of the evaluation criteria are considered

Call to action for global access to and harmonization of quality information of individual earth science datasets

Knowledge about the quality of data and metadata is important to support informed decisions on the (re)use of individual datasets and is an essential part of the ecosystem that supports open science. Quality assessments reflect the reliability and usability of data. They need to be consistently curated, fully traceable, and adequately documented, as these are crucial for sound decision- and policy-making efforts that rely on data. Quality assessments also need to be consistently represented and readily integrated across systems and tools to allow for improved sharing of information on quality at the dataset level for individual quality attribute or dimension. Although the need for assessing the quality of data and associated information is well recognized, methodologies for an evaluation framework and presentation of resultant quality information to end users may not have been comprehensively addressed within and across disciplines. Global interdisciplinary domain experts have come together to systematically explore needs, challenges and impacts of consistently curating and representing quality information through the entire lifecycle of a dataset. This paper describes the findings of that effort, argues the importance of sharing dataset quality information, calls for community action to develop practical guidelines, and outlines community recommendations for developing such guidelines. Practical guidelines will allow for global access to and harmonization of quality information at the level of individual Earth science datasets, which in turn will support open science

An Integrated Approach for Characterizing Aerosol Climate Impacts and Environmental Interactions

Aerosols exert myriad influences on the earth's environment and climate, and on human health. The complexity of aerosol-related processes requires that information gathered to improve our understanding of climate change must originate from multiple sources, and that effective strategies for data integration need to be established. While a vast array of observed and modeled data are becoming available, the aerosol research community currently lacks the necessary tools and infrastructure to reap maximum scientific benefit from these data. Spatial and temporal sampling differences among a diverse set of sensors, nonuniform data qualities, aerosol mesoscale variabilities, and difficulties in separating cloud effects are some of the challenges that need to be addressed. Maximizing the long-term benefit from these data also requires maintaining consistently well-understood accuracies as measurement approaches evolve and improve. Achieving a comprehensive understanding of how aerosol physical, chemical, and radiative processes impact the earth system can be achieved only through a multidisciplinary, inter-agency, and international initiative capable of dealing with these issues. A systematic approach, capitalizing on modern measurement and modeling techniques, geospatial statistics methodologies, and high-performance information technologies, can provide the necessary machinery to support this objective. We outline a framework for integrating and interpreting observations and models, and establishing an accurate, consistent, and cohesive long-term record, following a strategy whereby information and tools of progressively greater sophistication are incorporated as problems of increasing complexity are tackled. This concept is named the Progressive Aerosol Retrieval and Assimilation Global Observing Network (PARAGON). To encompass the breadth of the effort required, we present a set of recommendations dealing with data interoperability; measurement and model integration; multisensor synergy; data summarization and mining; model evaluation; calibration and validation; augmentation of surface and in situ measurements; advances in passive and active remote sensing; and design of satellite missions. Without an initiative of this nature, the scientific and policy communities will continue to struggle with understanding the quantitative impact of complex aerosol processes on regional and global climate change and air quality

An Integrative and Applicable Phylogenetic Footprinting Framework for \u3cem\u3ecis\u3c/em\u3e-regulatory Motifs Identification in Prokaryotic Genomes

Background: Phylogenetic footprinting is an important computational technique for identifying cis-regulatory motifs in orthologous regulatory regions from multiple genomes, as motifs tend to evolve slower than their surrounding non-functional sequences. Its application, however, has several difficulties for optimizing the selection of orthologous data and reducing the false positives in motif prediction. Results: Here we present an integrative phylogenetic footprinting framework for accurate motif predictions in prokaryotic genomes (MP3 ). The framework includes a new orthologous data preparation procedure, an additional promoter scoring and pruning method and an integration of six existing motif finding algorithms as basic motif search engines. Specifically, we collected orthologous genes from available prokaryotic genomes and built the orthologous regulatory regions based on sequence similarity of promoter regions. This procedure made full use of the large-scale genomic data and taxonomy information and filtered out the promoters with limited contribution to produce a high quality orthologous promoter set. The promoter scoring and pruning is implemented through motif voting by a set of complementary predicting tools that mine as many motif candidates as possible and simultaneously eliminate the effect of random noise. We have applied the framework to Escherichia coli k12 genome and evaluated the prediction performance through comparison with seven existing programs. This evaluation was systematically carried out at the nucleotide and binding site level, and the results showed that MP3 consistently outperformed other popular motif finding tools. We have integrated MP3 into our motif identification and analysis server DMINDA, allowing users to efficiently identify and analyze motifs in 2,072 completely sequenced prokaryotic genomes. Conclusion: The performance evaluation indicated that MP3 is effective for predicting regulatory motifs in prokaryotic genomes. Its application may enhance progress in elucidating transcription regulation mechanism, thus provide benefit to the genomic research community and prokaryotic genome researchers in particular

Why Government and Non-Governmental Policies and Projects Fail Despite ‘Evaluations’: An Indicator to Measure whether Evaluation Systems Incorporate the Rules of Good Governance

Background: While much has been written in the evaluation literature on the theory of evaluations and on specific cases, there is still no comprehensive and easy to use indicator that can be used to hold organizations to the principles of effective evaluation, to score their quality in several areas, and to offer an immediate diagnostic for improvements. Purpose: The article offers an easy-to-use indicator for measuring whether organizational evaluation policies for government and non-governmental organization spending actually protect the public interest in line with good governance and management principles or whether they serve, instead, to protect bureaucracies and hide wrongdoing under the cover of an “outside” evaluation. Setting: The primary focus of the piece is on international development evaluations but the author shows how the same indicator and model can be used for other government agencies as well as businesses, with modifications. Intervention: The article examines failures of evaluation systems in light of the principles for quality and shows how an indicator can be used to measure and prevent those failures. Research Design: The piece defines the principles of evaluation systems and accountability using both the frameworks of international agencies, themselves, and professional texts, places them in a model framework for effective evaluations, and then turns this framework into a set of questions to derive an indicator. Data Collection and Analysis: The article offers a sample detailed test of the indicator using the United Nations Development Programme (UNDP) as a case study. Several other organizations of different types are also tested briefly to yield a variety of results on the quality of their evaluation systems. Findings: Use of this indicator on several organizations including those of the UN system, the EC, U.S.A.I.D., reveals that a number of governmental agencies and contractors (and particularly in the field of international development) are actually failing to protect the public interest and are using evaluation processes as tools to cover up abuses and mistakes and to advocate or advertise for more funding.  Keywords: evaluation, international development, transparency, accountability, monitoring, governanc

CASE 11: Evaluating a Public Health Program for Continuous Quality Improvement: Options and Methods in a Time of Pandemic

In 2018, a review of the public health inspection program by the Middlesex-London Health Unit (MLHU) identified a gap in how its Environmental Health and Infectious Disease division evaluated mandatory public health services. To meet the requirements of the Ontario Public Health Standards, the MLHU must implement important recommendations from the review to ensure the most effective and efficient service model is delivered. One recommendation under consideration is to initiate activities that will entrench quality assurance and promote continuous quality improvement through monitoring and integration of findings from client experience into MLHU’s mandatory public health services. Rex Paul, the Manager of the Food Safety and Healthy Environment team at the MLHU, wants to initiate an evaluation framework for mandatory public health inspection programs. He is interested in exploring best practices and/or approaches for evaluating experiences of public pool and spa operators. This will serve as a pilot to assess client experiences with other mandatory public health services. Additionally, Rex wants to know the best data collection methods for the assessment of mandatory public health services in the context of COVID-19 pandemic. This will take a different approach as it is focused on improving the performance of staff who conduct regulatory work that enforces rules as opposed to previous client experience surveys where clients seek services from Public Health (vaccines, sexual health checks, smoking cessation, etc.). This case outlines the procedures and approaches for the evaluation of a public health program. It discusses relevant tools including logic models useful for clarifying the purpose of evaluation, mapping out an evaluation plan, identifying data collection tools, and collecting important information to address evaluation needs. The case also describes the importance of organizational standards and stakeholder consultations towards effective data collection and analysis for overall improvement in the program outcome. This process will help Paul’s team understand best approaches for collecting client feedback and incorporating findings to improve delivery of mandatory public health services. It will eventually provide baseline information for the planned monitoring of all regulated clients’ experiences of the MLHU’s mandatory programs

Dissecting the Workforce and Workplace for Clinical Endocrinology, and the Work of Endocrinologists Early in Their Careers

[Excerpt] No national mechanism is in place for an informed, penetrating, and systematic assessment of the physician workforce such as that achieved by the National Science Foundation (NSF) for the periodic evaluation of the nation’s scientists and engineers. Likewise, knowledge of the workforce for clinical research is enigmatic and fragmentary despite the serial recommendations of “blue-ribbon” panels to establish a protocol for the recurrent assessment of clinical investigators early in their careers. Failure to adopt a national system for producing timely, high-quality data on the professional activities of physicians limits the application of improvement tools for advancing clinical investigation and ultimately improving clinical practice. The present study was designed as a pilot project to test the feasibility of using Web-based surveys to estimate the administrative, clinical, didactic, and research work of subspecialty physicians employed in academic, clinical, federal, and pharmaceutical workplaces. Physician members of The Endocrine Society (TES) were used as surrogate prototypes of a subspecialty workforce because of their manageable number and investigative tradition. The results establish that Web-based surveys provide a tool to assess the activities of a decentralized workforce employed in disparate workplaces and underscore the value of focusing on physician work within the context of particular workplaces within a subspecialty. Our report also provides a new and timely snapshot of the amount and types of research performed by clinically trained endocrinologists and offers an evidenced-based framework for improving the investigative workforce in this medical subspecialty