Measuring quality and outcomes of research collaborations: An integrative review

Introduction: Although the science of team science is no longer a new field, the measurement of team science and its standardization remain in relatively early stages of development. To describe the current state of team science assessment, we conducted an integrative review of measures of research collaboration quality and outcomes. Methods: Collaboration measures were identified using both a literature review based on specific keywords and an environmental scan. Raters abstracted details about the measures using a standard tool. Measures related to collaborations with clinical care, education, and program delivery were excluded from this review. Results: We identified 44 measures of research collaboration quality, which included 35 measures with reliability and some form of statistical validity reported. Most scales focused on group dynamics. We identified 89 measures of research collaboration outcomes; 16 had reliability and 15 had a validity statistic. Outcome measures often only included simple counts of products; publications rarely defined how counts were delimited, obtained, or assessed for reliability. Most measures were tested in only one venue. Conclusions: Although models of collaboration have been developed, in general, strong, reliable, and valid measurements of such collaborations have not been conducted or accepted into practice. This limitation makes it difficult to compare the characteristics and impacts of research teams across studies or to identify the most important areas for intervention. To advance the science of team science, we provide recommendations regarding the development and psychometric testing of measures of collaboration quality and outcomes that can be replicated and broadly applied across studies

Interdisciplinary Fall Risk Screening and Assessment: An Evidence-Based Practice Project

This project sought to answer the following Evidence-Based Practice (EBP) question: What occupational therapy and interdisciplinary assessments have the best psychometric characteristics and are most effective for screening or evaluating person and environment fall risk factors and measuring the outcomes of fall prevention programs

A Systematic Review of International Clinical Guidelines for Rehabilitation of People With Neurological Conditions: What Recommendations Are Made for Upper Limb Assessment?

Conclusions: We present a comprehensive, critical, and original summary of current recommendations. Defining a core set of measures and agreed protocols requires international consensus between experts representing the diverse and multi-disciplinary field of neurorehabilitation including clinical researchers and practitioners, rehabilitation technology researchers, and commercial developers. Current lack of guidance may hold-back progress in understanding function and recovery. Together with a Delphi consensus study and an overview of systematic reviews of outcome measures it will contribute to the development of international guidelines for upper limb assessment in neurological conditions.This review formed part of the COST Action TD 1006A European Network on Robotics for Neurorehabilitation. It was an interdisciplinary EU-funded research network concentrating on the coordination of European research in the area of rehabilitation robotics

Comparing Social Science and Computer Science Workflow Processes for Studying Group Interactions

In this article, a team of authors from the Geeks and Groupies workshop, in Leiden, the Netherlands, compare prototypical approaches to studying group interaction in social science and computer science disciplines, which we call workflows. To help social and computer science scholars understand and manage these differences, we organize workflow into three major stages: research design, data collection, and analysis. For each stage, we offer a brief overview on how scholars from each discipline work. We then compare those approaches and identify potential synergies and challenges. We conclude our article by discussing potential directions for more integrated and mutually beneficial collaboration that go beyond the producer–consumer model

A best evidence medical education (BEME) systematic review on the feasibility, reliability and validity of the objective structured clinical examination (OSCE) in undergraduate medical studies

Tese de doutoramento, Ciências e Tecnologias da Saúde (Educação e Comunicação em Ciências da Saúde), Universidade de Lisboa, Faculdade de Medicina, 201

An analysis of rater effects in reviews of scientific manuscripts

In the peer review process used by scientific journals, ratings of manuscripts are obtained and used to make publication decisions. Though concerns have been raised about reviews given to scientific manuscripts, little has been done to address the effects of reviewer severity bias on decision making. In other settings, the methods of Generalizability Theory and Many-Facet Rasch Measurement often have been used to investigate and address such effects. The purpose of this study is to use Generalizability Theory and Many-Facet Rasch Measurement to examine the effects of reviewer severity on the ratings and decisions made during the peer review of scientific manuscripts. The merits of each method and their utility in this novel context also are assessed. Deidentified peer reviews (N = 635) that used a five-item rating scale were included in a two-facet, partially nested Generalizability Theory analysis and subsequent Decision Studies. Many-Facet Rasch Measurement analysis of the data produced reviewer severity measures and manuscript publishability measures corrected for reviewer severity. Multinomial logistic regression analysis was used to compare manuscript decision categories predicted by average raw scores and Many-Facet Rasch Measurement corrected scores. Reviewer severity rankings also were compared using raw and adjusted methods. The results of the Generalizability Theory analysis revealed that reviewers nested within manuscripts account for 35.48% of the variance in publishability scores. Manuscripts accounted for 12.21% of the total variance, and items accounted for 15.22% of the total variance. Decision Studies indicated that an unrealistic number of reviewers and items would be needed to increase the generalizability coefficient and index of dependability to acceptable levels and that other methods of improving reliability should be employed. When the average raw total score was used to predict manuscript decision category, the overall percentage of manuscripts that were correctly classified using the average raw total score was 55.15%. Using the manuscript publishability measure (theta), the percentage of manuscripts that were correctly classified when the publishability measure was used was 52.49%, suggesting differences in classification, if a manuscript publishability measures corrected for reviewer severity were used. The reviewers’ average raw ratings and the reviewers’ severity measures had a Spearman rank-order correlation of -0.6083, which demonstrates differences likely attributable to the adjustment for manuscript quality in the severity measure. These findings indicate that reviewers are inconsistent in their reviews of manuscripts. Reviewer severity bias can be addressed with Many-Facet Rasch Measurement adjustments, but additional reviewer training may be needed to improve the reliability of manuscript scores. Both Generalizability Theory and Many-Facet Rasch Measurement contributed to the findings of the study and to understanding reviewer behavior. These methods show potential for increasing the capacity for more fair and accurate rating methods in the peer review of scientific manuscripts

Built environment assessment: Multidisciplinary perspectives.

Context:As obesity has become increasingly widespread, scientists seek better ways to assess and modify built and social environments to positively impact health. The applicable methods and concepts draw on multiple disciplines and require collaboration and cross-learning. This paper describes the results of an expert team׳s analysis of how key disciplinary perspectives contribute to environmental context-based assessment related to obesity, identifies gaps, and suggests opportunities to encourage effective advances in this arena. Evidence acquisition:A team of experts representing diverse disciplines convened in 2013 to discuss the contributions of their respective disciplines to assessing built environments relevant to obesity prevention. The disciplines include urban planning, public health nutrition, exercise science, physical activity research, public health and epidemiology, behavioral and social sciences, and economics. Each expert identified key concepts and measures from their discipline, and applications to built environment assessment and action. A selective review of published literature and internet-based information was conducted in 2013 and 2014. Evidence synthesis:The key points that are highlighted in this article were identified in 2014-2015 through discussion, debate and consensus-building among the team of experts. Results focus on the various disciplines׳ perspectives and tools, recommendations, progress and gaps. Conclusions:There has been significant progress in collaboration across key disciplines that contribute to studies of built environments and obesity, but important gaps remain. Using lessons from interprofessional education and team science, along with appreciation of and attention to other disciplines׳ contributions, can promote more effective cross-disciplinary collaboration in obesity prevention

Knowledge Elicitation Methods for Affect Modelling in Education

Research on the relationship between affect and cognition in Artificial Intelligence in Education (AIEd) brings an important dimension to our understanding of how learning occurs and how it can be facilitated. Emotions are crucial to learning, but their nature, the conditions under which they occur, and their exact impact on learning for different learners in diverse contexts still needs to be mapped out. The study of affect during learning can be challenging, because emotions are subjective, fleeting phenomena that are often difficult for learners to report accurately and for observers to perceive reliably. Context forms an integral part of learners’ affect and the study thereof. This review provides a synthesis of the current knowledge elicitation methods that are used to aid the study of learners’ affect and to inform the design of intelligent technologies for learning. Advantages and disadvantages of the specific methods are discussed along with their respective potential for enhancing research in this area, and issues related to the interpretation of data that emerges as the result of their use. References to related research are also provided together with illustrative examples of where the individual methods have been used in the past. Therefore, this review is intended as a resource for methodological decision making for those who want to study emotions and their antecedents in AIEd contexts, i.e. where the aim is to inform the design and implementation of an intelligent learning environment or to evaluate its use and educational efficacy