The Diversity of European Research Evaluation Systems

Universities have undergone profound changes in the last decades. A shift towards more accountability and to “new public management” practices in the administration of universities took place and led to an increase of the share of project funds in some countries and to the introduction of performance-based funding systems (PRFSs) in others. In all countries, research evaluation’s importance increases. However, while research evaluation is centralized in some countries, evaluation is organized at the institutional level only in others. Thus, the importance of research evaluation and how it is organised varies across countries. In this paper, we present a typology of national research evaluation systems in Europe that sheds light on the complex issue of national differences in the organisation of research evaluation. We use the data of a two-round Delphi survey among specialists in research evaluation as a basis of our analysis and apply Multiple Correspondence Analysis to create a typology of European research evaluation systems that includes a broad range of countries, including countries, for which not much information is available to the English-speaking research community and takes into account that during the last years also the SSH are more concerned with evaluations but the commonly applied evaluation instruments do not fit their research practices. We identify five types of research evaluation: “non-metric, non-SSH” (with Iceland as the best representor), “non-metric, SSH-specific” (Switzerland, “funding, non-metric” (Norway), “funding, metric” (Denmark) and “metric, English” (Estonia). The main result of this analysis is that the national organization of research evaluation system is a complex issue and the research evaluation landscape in Europe is diverse. Yet some components can be identified that define main types of research evaluation. A secondary result is that different types of research evaluation are linked to different conditions in countries. We therefore recommend, that designers of evaluation systems make a conscious link between the goals to achieve, the incentives to promote and the design of the evaluation system, rather than to strive to the unification of evaluation systems

Peer review and citation data in predicting university rankings, a large-scale analysis

Most Performance-based Research Funding Systems (PRFS) draw on peer review and bibliometric indicators, two different method- ologies which are sometimes combined. A common argument against the use of indicators in such research evaluation exercises is their low corre- lation at the article level with peer review judgments. In this study, we analyse 191,000 papers from 154 higher education institutes which were peer reviewed in a national research evaluation exercise. We combine these data with 6.95 million citations to the original papers. We show that when citation-based indicators are applied at the institutional or departmental level, rather than at the level of individual papers, surpris- ingly large correlations with peer review judgments can be observed, up to r <= 0.802, n = 37, p < 0.001 for some disciplines. In our evaluation of ranking prediction performance based on citation data, we show we can reduce the mean rank prediction error by 25% compared to previous work. This suggests that citation-based indicators are sufficiently aligned with peer review results at the institutional level to be used to lessen the overall burden of peer review on national evaluation exercises leading to considerable cost savings

STATISTICAL MODELS USED IN PROJECT MANAGEMENT

“Statistical Models used in Project Management” is approaching major interest issues within the actual research context, when research activities are grouped per project. An observer of the phenomena taking place today in the higher education and research can not fail to notice the importance of scientific research have achieved through programs or projects as part of European higher education and the need to establish closer links between Common European Higher Education Area and European Research Area in the context of both the organic substrate defining building a Europe of knowledge In this context we can define the major role they play research conducted by research programs and projects with national or international funding and training resources for scientific research, along with promoting interdisciplinary, in the process of maintaining and improving the quality of European higher education oriented performance and competitiveness.performance in scientific research, projects, performance indicators, statistical models based on indicators systems, performance evaluation.

Implementing a national health research for development platform in a low-income country – a review of Malawi’s Health Research Capacity Strengthening Initiative

Background National health research for development (R4D) platforms in lower income countries (LICs) are few. The Health Research Capacity Strengthening Initiative (HRCSI, 2008–2013) was a national systems-strengthening programme in Malawi involved in national priority setting, decision-making on funding, and health research actor mobilization. Methods We adopted a retrospective mixed-methods evaluation approach, starting with information gleaned from reports (HRCSI and Malawian) and databases (HRCSI). A framework of a health research system (actors and components) guided report review and interview guide development. From a list of 173 individuals involved in HRCSI, 30 interviewees were selected within categories of stakeholders. Interviews were conducted face-to-face or via telephone/Skype over 1 month, documented with extensive notes. Analysis of emerging themes was iterative among co-evaluators, with synthesis according to the implementation stage. Results Major HRCSI outputs included (1) National research priority-setting: through the production of themed background papers by Malawian health researchers and broad consultation, HRCSI led the development of a National Health Research Agenda (2012–2016), widely regarded as one of HRCSI’s foremost achievements. (2) Institutional research capacity: there was an overwhelming view that HRCSI had produced a step-change in the number of high calibre scientists in Malawi and in fostering research interest among young Malawians, providing support for around 56 MSc and PhD students, and over 400 undergraduate health-related projects. (3) Knowledge sharing: HRCSI supported research dissemination through national and institutional meetings by sponsoring attendance at conferences and through close relationships with individuals in the print media for disseminating information. (4) Sustainability: From 2011–2013, HRCSI significantly improved research systems, processes and leadership in Malawi, but further strengthening was needed for HRCSI to be effectively integrated into government structures and sustained long-term. Overall, HRCSI carried out many components relevant to a national health research system coordinating platform, and became competent at managing over half of 12 areas of performance for research councils. Debate about its location and challenges to sustainability remain open questions. Conclusions More experimentation in the setting-up of national health R4D platforms to promote country ‘ownership’ is needed, accompanied by evaluation processes that facilitate learning and knowledge exchange of better practices among key actors in health R4D systems

Impact response of advance combat helmet pad systems

Combat helmets are designed to protect against ballistic threats and fragments of explosive devices. There are numerous types of helmet comfort foams available. However, pad systems have not been evaluated in combat helmets to understand to what extent they mitigate head accelerations. In this work, different pad systems are studied to analyze the ballistic performance of combat helmets using a Hybrid III dummy equipped with longitudinal accelerometers at the head and a neck simulator. The tests are conducted with 9 mm Full Metal Jacket (FMJ) projectiles according to the performance requirements III-A of the NIJ 0106.01 standard. This experimental methodology allows the evaluation of brain and neck injuries. The thicker bicomponent polyurethane foams and the honeycomb configuration provided the best results in terms of mitigating brain damage due to accelerations applying different criteria (PLA, WSTC, HIC). However, it was concluded that there is no cervical injury or cranial fracture risk for any of the cases studied.The authors acknowledge the Ministry of Economy and Competitiveness of Spain and the FEDER program under Projects DPI2017-88166-R and PID2020-118946RB-I00 for the financial support of the work. M. Rodriguez-Millan acknowledges the Spanish Ministry of Universities, the National Program for the Promotion of Talent, and its Employability in Research and Development and Innovation (R&D&I), National Mobility Subprogram of the National Plan for Scientific and Technical Research and Innovation 2021–2023, for the professor’s mobility program (PRX21/00329). It has to thank the funding for APC: Universidad Carlos III de Madrid (Read and Publish Agreement CRUECSIC 2023).Publicad

Towards a Cooperative Security System for Mobile-Health Applications

[EN] Mobile Health (m-Health) system architectures are typically based on mobile and wireless communications, and use mobile devices with data exchange supported by Web Services (WS). Although m-Health systems offer mobility as a potential and precious resource they also present several challenged issues and constraints, such as, battery and storage capacity, broadcast constraints, interferences, disconnections, noises, limited bandwidths, and network delays. Furthermore, constant mobility and often-required Internet connectivity also exposes and compromises the privacy and confidentiality of the m-Health system information. This paper proposes a novel data encryption solution for mobile health systems, considering a novel and early-proposed cooperation strategy. This encryption solution, called data encryption for mobile health applications (DE4MHA), tries to guarantee the best confidentiality, integrity, and authenticity of m-health systems users data. The paper also presents a performance evaluation study comparing the performance an m-Health application with and without the DE4MHA.This work has been partially supported by National Funding from the FCT - Fundacao para a Ciencia e a Tecnologia through the PEst-OE/EEI/LA0008/2013 Project; by the AAL4ALL (Ambient Assisted Living for All), project co-funded by COMPETE under FEDER via QREN Programme; by Brazilian National Council for Research and Development (CNPq) via Grant No. 309335/2017-5; and by FINEP, with resources from Funttel, Grant No. 01.14.0231.00, under the Centro de Referencia em Radiocomunicacoes - CRR project of the Instituto Nacional de Telecomunicacoes (Inatel), Brazil.Silva, BM.; Rodrigues, JJPC.; Canelo, F.; Lopes, IMC.; Lloret, J. (2019). Towards a Cooperative Security System for Mobile-Health Applications. Electronic Commerce Research and Applications. 19(3):629-654. https://doi.org/10.1007/s10660-014-9154-362965419

Health policy and systems research agendas in developing countries

BACKGROUND: Health policy and systems research (HPSR) is an international public good with potential to orient investments and performance at national level. Identifying research trends and priorities at international level is therefore important. This paper offers a conceptual framework and defines the HPSR portfolio as a set of research projects under implementation. The research portfolio is influenced by factors external to the research system as well as internal to it. These last include the capacity of research institutions, the momentum of research programs, funding opportunities and the influence of stakeholder priorities and public opinion. These dimensions can vary in their degree of coordination, leading to a complementary or a fragmented research portfolio. OBJECTIVE: The main objective is to identify the themes currently being pursued in the research portfolio and agendas within developing countries and to quantify their frequency in an effort to identify current research topics and their underlying influences. METHODS: HPSR topics being pursued by developing country producer institutions and their perceived priorities were identified through a survey between 2000 and 2002. The response to a call for letters of intent issued by the Alliance in 2000 for a broad range of topics was also analyzed. The institutions that were the universe of this study consisted of the 176 institutional partners of the Alliance for Health Policy and Systems Research producing research in low and middle income countries outside Europe. HPSR topics as well as the beneficiaries or issues and the health problems addressed were content analyzed. Topics were classified into 19 categories and their frequency analyzed across groups of countries with similar per capita income. Agendas were identified by analyzing the source of funding and of project initiation for projects under implementation. RESULTS: The highest ranking topic at the aggregate level is "Sector analysis", followed by "Disease burden" and "Management and organization". Categories at the bottom of this ranking are "Equity", "Policy process", "Economic policy and health" and "Information systems". "Disease burden" is more often funded than other topics for which there is more demand or perceived priority. Analysis suggests few although important differences across priorities, demand for funding and actual project funding. The donors' agenda coincides most with the ranking of research topics overall. Ranking across country income groups shows important differences. Topics that gain prominence in low income countries are "Disease burden" and "Accessibility". In lower middle income countries "Insurance" gains prominence. In upper middle income countries "Decentralization/local health systems", "Equity" and "Policy process" are more prominent. "Program evaluation" is the most consistently ranked topic across income regions, showing a neutral influence by donors, governments or researchers. CONCLUSIONS: The framework proposed offers a basis to identify and contrast research needs, projects and products at the international level and to identify the actor agendas and their influence. Research gaps are suggested when comparing topic ranking against the challenges to health system strengthening and scaling up of disease control programs. Differences across per capita income groups suggests the need for differentiated priority setting mechanisms guiding international support. Data suggests that stakeholders have different agendas, and that donors predominate in determining the research portfolio. High-level consensus building at the national and international levels is necessary to ensure that the diverse agendas play a complementary role in support of health system objectives. The Ministerial Summit for Health Research to be held in Mexico in November 2004 should be an opportunity to analyze further data and to commit funding for priorities identified through sharing and discussion of agendas