The value of source data verification in a cancer clinical trial

Background Source data verification (SDV) is a resource intensive method of quality assurance frequently used in clinical trials. There is no empirical evidence to suggest that SDV would impact on comparative treatment effect results from a clinical trial. Methods Data discrepancies and comparative treatment effects obtained following 100% SDV were compared to those based on data without SDV. Overall survival (OS) and Progression-free survival (PFS) were compared using Kaplan-Meier curves, log-rank tests and Cox models. Tumour response classifications and comparative treatment Odds Ratios (ORs) for the outcome objective response rate, and number of Serious Adverse Events (SAEs) were compared. OS estimates based on SDV data were compared against estimates obtained from centrally monitored data. Findings Data discrepancies were identified between different monitoring procedures for the majority of variables examined, with some variation in discrepancy rates. There were no systematic patterns to discrepancies and their impact was negligible on OS, the primary outcome of the trial (HR (95% CI): 1.18(0.99 to 1.41), p = 0.064 with 100% SDV; 1.18(0.99 to 1.42), p = 0.068 without SDV; 1.18(0.99 to 1.40), p = 0.073 with central monitoring). Results were similar for PFS. More extreme discrepancies were found for the subjective outcome overall objective response (OR (95% CI): 1.67(1.04 to 2.68), p = 0.03 with 100% SDV; 2.45(1.49 to 4.04), p = 0.0003 without any SDV) which was mostly due to differing CT scans. Interpretation Quality assurance methods used in clinical trials should be informed by empirical evidence. In this empirical comparison, SDV was expensive and identified random errors that made little impact on results and clinical conclusions of the trial. Central monitoring using an external data source was a more efficient approach for the primary outcome of OS. For the subjective outcome objective response, an independent blinded review committee and tracking system to monitor missing scan data could be more efficient than SDV

Comparing "challenge-based" and "code-based" internet voting verification implementations

Internet-enabled voting introduces an element of invisibility and unfamiliarity into the voting process, which makes it very different from traditional voting. Voters might be concerned about their vote being recorded correctly and included in the final tally. To mitigate mistrust, many Internet-enabled voting systems build verifiability into their systems. This allows voters to verify that their votes have been cast as intended, stored as cast and tallied as stored at the conclusion of the voting period. Verification implementations have not been universally successful, mostly due to voter difficulties using them. Here, we evaluate two cast as intended verification approaches in a lab study: (1) "Challenge-Based" and (2) "Code-Based". We assessed cast-as-intended vote verification efficacy, and identified usability issues related to verifying and/or vote casting. We also explored acceptance issues post-verification, to see whether our participants were willing to engage with Internet voting in a real election. Our study revealed the superiority of the code-based approach, in terms of ability to verify effectively. In terms of real-life Internet voting acceptance, convenience encourages acceptance, while security concerns and complexity might lead to rejection

Mutation testing on an object-oriented framework: An experience report

This is the preprint version of the article - Copyright @ 2011 ElsevierContext The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners. Objective This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective. Method In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT. Results Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field. Conclusion We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.This work has been partially supported by the European Commission (FEDER) and Spanish Government under CICYT Project SETI (TIN2009-07366) and the Andalusian Government Projects ISABEL (TIC-2533) and THEOS (TIC-5906)

Empirical validity of the evaluation of public policies: models of evaluation and quality of evidence

There is a wide range of evaluation methods. On the basis of which criteria should one method be chosen? On what scientific foundations are the results of numerous evaluations based? How can reliability and empirical validity be tested? The relevance of such questions is heightened in new fields of action such as agri-environmental policy. This paper aims to demonstrate that theoretical advances on level and types of evidence (existence, causality, effectiveness) can help resolve these issues. The main evaluation methods are classified into three main categories, according to their main goal (to learn, measure, understand) and linked to the debate on types of evidence. The analysis is illustrated by comparing evaluation methods in the field of agroenvironmental policies and farm advisory services. Attention is drawn to the shortcomings of each method with respect to corroborating facts as well as existing complementarities and trade-offs between methods in terms of empirical validity.evaluation, evidence, agricultural extension, agri-environment, Agricultural and Food Policy, B49 H83 Q18 Q58,

Generating citizen trust in e-government using a trust verification agent: A research note

Generating Citizen Trust in e-Government using a Trust Verification AgentThis is an eGISE network paper. It is motivated by a concern about the extent to which trust issues inhibit a citizen’s take-up of online public sector services or engagement with public decision and policy making. A citizen’s decision to use online systems is influenced by their willingness to trust the environment and agency involved. This project addresses one aspect of individual “trust” decisions by providing support for citizens trying to evaluate the implications of the security infrastructure provided by the agency. Based on studies of the way both groups (citizens and agencies) express their concerns and concepts in the security area, the project will develop a software tool – a trust verification agent (TVA) - that can take an agency’s security statements (or security audit) and infer how effectively this meets the security concerns of a particular citizen. This will enable citizens to state their concerns and obtain an evaluation of the agency’s provision in appropriate “citizen friendly” language. Further, by employing rule-based expert systems techniques the TVA will also be able to explain its evaluation.Engineering and Physical Sciences Research Council, UK (grant GR/T27020/01

Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing

Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project whilst also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation

Operator-based approaches to harm minimisation in gambling: summary, review and future directions

In this report we give critical consideration to the nature and effectiveness of harm minimisation in gambling. We identify gambling-related harm as both personal (e.g., health, wellbeing, relationships) and economic (e.g., financial) harm that occurs from exceeding one’s disposable income or disposable leisure time. We have elected to use the term ‘harm minimisation’ as the most appropriate term for reducing the impact of problem gambling, given its breadth in regard to the range of goals it seeks to achieve, and the range of means by which they may be achieved. The extent to which an employee can proactively identify a problem gambler in a gambling venue is uncertain. Research suggests that indicators do exist, such as sessional information (e.g., duration or frequency of play) and negative emotional responses to gambling losses. However, the practical implications of requiring employees to identify and interact with customers suspected of experiencing harm are questionable, particularly as the employees may not possess the clinical intervention skills which may be necessary. Based on emerging evidence, behavioural indicators identifiable in industryheld data, could be used to identify customers experiencing harm. A programme of research is underway in Great Britain and in other jurisdiction

Generating citizen trust in e-government using a trust verification agent: A research note

Generating Citizen Trust in e-Government using a Trust Verification AgentThis is an eGISE network paper. It is motivated by a concern about the extent to which trust issues inhibit a citizen’s take-up of online public sector services or engagement with public decision and policy making. A citizen’s decision to use online systems is influenced by their willingness to trust the environment and agency involved. This project addresses one aspect of individual “trust” decisions by providing support for citizens trying to evaluate the implications of the security infrastructure provided by the agency. Based on studies of the way both groups (citizens and agencies) express their concerns and concepts in the security area, the project will develop a software tool – a trust verification agent (TVA) - that can take an agency’s security statements (or security audit) and infer how effectively this meets the security concerns of a particular citizen. This will enable citizens to state their concerns and obtain an evaluation of the agency’s provision in appropriate “citizen friendly” language. Further, by employing rule-based expert systems techniques the TVA will also be able to explain its evaluation.Engineering and Physical Sciences Research Council-UK (grant GR/T27020/01