The metric tide: report of the independent review of the role of metrics in research assessment and management

This report presents the findings and recommendations of the Independent Review of the Role of Metrics in Research Assessment and Management. The review was chaired by Professor James Wilsdon, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and administration. This review has gone beyond earlier studies to take a deeper look at potential uses and limitations of research metrics and indicators. It has explored the use of metrics across different disciplines, and assessed their potential contribution to the development of research excellence and impact. It has analysed their role in processes of research assessment, including the next cycle of the Research Excellence Framework (REF). It has considered the changing ways in which universities are using quantitative indicators in their management systems, and the growing power of league tables and rankings. And it has considered the negative or unintended effects of metrics on various aspects of research culture. The report starts by tracing the history of metrics in research management and assessment, in the UK and internationally. It looks at the applicability of metrics within different research cultures, compares the peer review system with metric-based alternatives, and considers what balance might be struck between the two. It charts the development of research management systems within institutions, and examines the effects of the growing use of quantitative indicators on different aspects of research culture, including performance management, equality, diversity, interdisciplinarity, and the ‘gaming’ of assessment systems. The review looks at how different funders are using quantitative indicators, and considers their potential role in research and innovation policy. Finally, it examines the role that metrics played in REF2014, and outlines scenarios for their contribution to future exercises

The case for cloud service trustmarks and assurance-as-a-service

Cloud computing represents a significant economic opportunity for Europe. However, this growth is threatened by adoption barriers largely related to trust. This position paper examines trust and confidence issues in cloud computing and advances a case for addressing them through the implementation of a novel trustmark scheme for cloud service providers. The proposed trustmark would be both active and dynamic featuring multi-modal information about the performance of the underlying cloud service. The trustmarks would be informed by live performance data from the cloud service provider, or ideally an independent third-party accountability and assurance service that would communicate up-to-date information relating to service performance and dependability. By combining assurance measures with a remediation scheme, cloud service providers could both signal dependability to customers and the wider marketplace and provide customers, auditors and regulators with a mechanism for determining accountability in the event of failure or non-compliance. As a result, the trustmarks would convey to consumers of cloud services and other stakeholders that strong assurance and accountability measures are in place for the service in question and thereby address trust and confidence issues in cloud computing

A high-level semiotic trust agent scoring model for collaborative virtual organisations

In this paper, we describe how a semiotic ladder, together with a supportive trust agent, can be used to address “soft” trust issues in the context of collaborative Virtual Organisations (VO). The intention is to offer all parties better support for trust (as reputation) management including the reduction of risk and improved reliability of VO e-services. The semiotic ladder is intended to support the VO e-service lifecycle through the articulation of e-trust at various levels of system abstraction, including trust as measurable confidence. At the social level, reputation and reliability measures of e-trust are the relevant dimensions as regards choice of VO partner and are also relevant to the negotiation of service level agreements between the VO partners. By contrast, at the lower levels of the trust ladder, e-trust measures typically address the degree to which secure sign on and message level security conforms to various tangible technological security protocols. The novel trust agent provides the e-service consumer with an objective measure of the trustworthiness of the e-service at run-time, just prior to its actual consumption. Specifically, VO e-service consumer confidence level is informed, by leveraging third party objective evidence. This evidence comprises a set of Corporate Governance (CG) scores. These scores are used as a trust proxy for the "real" owner of the VO. There are also inherent limitations associated with the use of CG scores. These are duly acknowledged

Academic digital libraries of the future : an environment scan

Libraries are attempting to face a future in which almost every fixed point has disappeared. Users are changing; content is changing; research is taking new forms. Indeed the very need for libraries is being questioned in some quarters. This paper explores the nature of the changes and challenges facing higher education libraries and suggests key areas of strength and core activities which should be exploited to secure their future

Computing word-of-mouth trust relationships in social networks from Semantic Web and Web 2.0 data sources

Social networks can serve as both a rich source of new information and as a filter to identify the information most relevant to our specific needs. In this paper we present a methodology and algorithms that, by exploiting existing Semantic Web and Web2.0 data sources, help individuals identify who in their social network knows what, and who is the most trustworthy source of information on that topic. Our approach improves upon previous work in a number of ways, such as incorporating topic-specific rather than global trust metrics. This is achieved by generating topic experience profiles for each network member, based on data from Revyu and del.icio.us, to indicate who knows what. Identification of the most trustworthy sources is enabled by a rich trust model of information and recommendation seeking in social networks. Reviews and ratings created on Revyu provide source data for algorithms that generate topic expertise and person to person affinity metrics. Combining these metrics, we are implementing a user-oriented application for searching and automated ranking of information sources within social networks

Machine-Readable Privacy Certificates for Services

Privacy-aware processing of personal data on the web of services requires managing a number of issues arising both from the technical and the legal domain. Several approaches have been proposed to matching privacy requirements (on the clients side) and privacy guarantees (on the service provider side). Still, the assurance of effective data protection (when possible) relies on substantial human effort and exposes organizations to significant (non-)compliance risks. In this paper we put forward the idea that a privacy certification scheme producing and managing machine-readable artifacts in the form of privacy certificates can play an important role towards the solution of this problem. Digital privacy certificates represent the reasons why a privacy property holds for a service and describe the privacy measures supporting it. Also, privacy certificates can be used to automatically select services whose certificates match the client policies (privacy requirements). Our proposal relies on an evolution of the conceptual model developed in the Assert4Soa project and on a certificate format specifically tailored to represent privacy properties. To validate our approach, we present a worked-out instance showing how privacy property Retention-based unlinkability can be certified for a banking financial service.Comment: 20 pages, 6 figure

Reconsidering online reputation systems

Social and socioeconomic interactions and transactions often require trust. In digital spaces, the main approach to facilitating trust has effectively been to try to reduce or even remove the need for it through the implementation of reputation systems. These generate metrics based on digital data such as ratings and reviews submitted by users, interaction histories, and so on, that are intended to label individuals as more or less reliable or trustworthy in a particular interaction context. We suggest that conventional approaches to the design of such systems are rooted in a capitalist, competitive paradigm, relying on methodological individualism, and that the reputation technologies themselves thus embody and enact this paradigm in whatever space they operate in. We question whether the politics, ethics and philosophy that contribute to this paradigm align with those of some of the contexts in which reputation systems are now being used, and suggest that alternative approaches to the establishment of trust and reputation in digital spaces need to be considered for alternative contexts

Provenance-based trust for grid computing: Position Paper

Current evolutions of Internet technology such as Web Services, ebXML, peer-to-peer and Grid computing all point to the development of large-scale open networks of diverse computing systems interacting with one another to perform tasks. Grid systems (and Web Services) are exemplary in this respect and are perhaps some of the first large-scale open computing systems to see widespread use - making them an important testing ground for problems in trust management which are likely to arise. From this perspective, today's grid architectures suffer from limitations, such as lack of a mechanism to trace results and lack of infrastructure to build up trust networks. These are important concerns in open grids, in which "community resources" are owned and managed by multiple stakeholders, and are dynamically organised in virtual organisations. Provenance enables users to trace how a particular result has been arrived at by identifying the individual services and the aggregation of services that produced such a particular output. Against this background, we present a research agenda to design, conceive and implement an industrial-strength open provenance architecture for grid systems. We motivate its use with three complex grid applications, namely aerospace engineering, organ transplant management and bioinformatics. Industrial-strength provenance support includes a scalable and secure architecture, an open proposal for standardising the protocols and data structures, a set of tools for configuring and using the provenance architecture, an open source reference implementation, and a deployment and validation in industrial context. The provision of such facilities will enrich grid capabilities by including new functionalities required for solving complex problems such as provenance data to provide complete audit trails of process execution and third-party analysis and auditing. As a result, we anticipate that a larger uptake of grid technology is likely to occur, since unprecedented possibilities will be offered to users and will give them a competitive edge