Ranking Meets Distance Education: Defining Relevant Criteria and Indicators for Online Universities

University ranking systems are being implemented with the aim of assessing and comparing higher education institutions at a global level. Despite their being increasingly used, rankings are often strongly criticized for their social and economic implications, as well as for limitations in their technical implementation. One of these limitations is that they do not consider the specific characteristics of online education. This study used a participatory approach to define a set of criteria and indicators suitable to reflect the specific nature of distance education. This endeavour will help evaluate and rank online higher education institutions more appropriately than in current practice, where indicators are devised for traditional universities. To this end, several stakeholders and informants were involved in a Delphi study in an attempt to reach the broader higher education institutions (HEI) community. According to the study participants, apart from students’ achievements and general quantitative measures of HEI performance, which are quite common in traditional ranking systems, teaching and student learning experience turned out to be the most important criteria. Student support, teacher support, technological infrastructure, research and organization were deemed middle ground criteria, while sustainability and reputation were regarded as the least important criteria

Efficience de la recherche dans les écoles de gestion au Canada : modélisation par des approches paramétriques et non-paramétriques

La production des connaissances revêt une grande valeur pour les gouvernements, les universités et les chercheurs. Pour ces derniers, aujourd’hui plus que jamais, la recherche prend de plus en plus d’importance dans leurs portefeuilles d’activités. Plusieurs facteurs motivent cette nouvelle tendance, notamment l’adoption, dans plusieurs pays, de systèmes de financement de la recherche axée sur la productivité, la concurrence en matière de recherche qui est devenue mondiale, la prolifération des outils uniformisés de mesure de l’excellence scientifique, et l’importance de la performance et de la productivité en recherche dans la réussite de la trajectoire de carrière des chercheurs. Cependant, la performance des chercheurs universitaires est très variable, tant entre les disciplines qu’au sein d’une même discipline. Ces constats renforcent la pertinence de questionner l’allocation et la gestion des ressources dans le domaine de la recherche universitaire. La présente étude s’attaque à cette problématique dans le cadre spécifique de la recherche dans les Écoles de gestion au Canada. Son objectif général est, d’une part, de dresser un portrait détaillé de l’avancement des connaissances sur le concept de l’efficience de la recherche académique et d’identifier les principaux jalons qui ont marqué son évolution au cours des deux dernières décennies et, d’autre part, d’évaluer l’efficience en matière de publications et de citations des chercheurs dans les Écoles de gestion au Canada et d’identifier les déterminants susceptibles d’expliquer les écarts d’efficience entre eux. La thèse est structurée en trois articles. Le premier article a préconisé la méthode de la revue systématique de la littérature et celle du « vote counting » pour édifier un cadre conceptuel intégrateur des intrants, des extrants et des déterminants de l’efficience de la recherche académique. Il a également permis d’identifier plusieurs opportunités de recherche pour contribuer à l’avancement des connaissances dans ce champ d’étude. Le deuxième article a utilisé une nouvelle méthode, « The Reference Publication Year Spectroscopy » pour étudier en profondeur le concept de l’efficience de la recherche académique en identifiant ses racines historiques ainsi que les contributions qui ont conditionné son évolution. Ces deux premiers articles ont permis de satisfaire à la première partie de l’objectif général de cette recherche : « dresser un portrait détaillé de l’avancement des connaissances sur le concept de l’efficience de la recherche académique et d’identifier les principaux jalons qui ont marqué son évolution au cours des deux dernières décennies ». Tirant profit des constats et des contributions potentielles à l’avancement des connaissances identifiés dans les deux premiers articles, le troisième article de la thèse a estimé des frontières paramétriques et nonparamétriques de l’efficience en matière de publications et de citations des chercheurs dans huit disciplines de recherche dans les Écoles de gestion au Canada. Il a également identifié plusieurs déterminants des écarts d’efficience entre les chercheurs affiliés à ces Écoles. Entre autres, les résultats de ce troisième article ont montré que les niveaux d’efficience diffèrent d’une manière significative d’un champ disciplinaire à un autre, et au sein même des champs disciplinaires, et que l'accréditation AACSB, l'affiliation à des universités prestigieuses, la taille de l'institution, les sources de financement et la séniorité sont positivement associées à des niveaux d’efficience élevés. Les résultats des trois articles ont permis de suggérer quelques pistes de réflexion et d’intervention pouvant améliorer l’efficience de la recherche académique des chercheurs en général, et de ceux affiliés aux Écoles de gestion au Canada, en particulier.The production of knowledge is of great importance to governments, universities and researchers. For the latter, today more than ever, research is becoming more and more important in their portfolios of activities. A number of factors are driving this new trend, including the introduction of productivity-driven research funding systems in a number of countries, global competition for research, the proliferation of standardized tools for assessing scientific excellence, and the importance of research performance and productivity in researchers’ career path success. However, the performance of university researchers varies greatly between disciplines and within the same discipline. These findings reinforce the relevance of questioning the allocation and management of resources in the field of university research. This study addresses this issue in the specific context of research in Canadian Business Schools. Its aims, on the one hand, to draw a detailed portrait of the advancement of knowledge on the concept of the efficiency of academic research and to identify the main milestones that have marked its evolution over the last two decades and, on the other hand, to evaluate the efficiency of academic research of Canadian Business Schools’ scholars and to identify the determinants that may explain the differences in efficiency between them. The thesis allowed the production of three articles. The first one used the systematic review of the literature and the method of vote counting to build an integrative conceptual framework of inputs, outputs, and determinants of the efficiency of academic research. It has also identified several research opportunities to contribute to the advancement of knowledge in this field of study. The second article used a new method, The Reference Publication Year Spectroscopy, to study in depth the concept of the efficiency of academic research by identifying its historical roots as well as the contributions that marked its evolution. These first two articles made it possible to satisfy the first part of the general objective of this research: “to draw a detailed portrait of the advancement of knowledge on the concept of the efficiency of academic research and to identify the main milestones that have marked its evolution over the last two decades”. Taking advantage of the findings and potential contributions to the advancement of knowledge identified in the first two articles, the third article estimated parametric and non-parametric frontiers of efficiency of scholars’ publications and citations in eight research disciplines in Canadian Business schools. It also allowed to identify several levers of efficiency gaps among researchers affiliated with these schools. Among other things, the results of this third article showed that efficiency scores differ significantly from one disciplinary field to another, and even within disciplinary fields, and that AACSB accreditation, affiliation to prestigious universities, size of institution, sources of funding and seniority are positively associated with high levels of efficiency. The findings of the three articles devised some lines of action that might improve the efficiency of academic research for researchers in general and those affiliated with Canadian Business schools, in particular

Nondeterministic ranking of university departments

Rankings in higher education are largely used to summarize a huge amount of information into easily understandable numbers. They are also used by governments in order to allocate funding. Nevertheless, they are often criticized. One stream of criticism refers to the fact that rankings build up an ordinal order by considering only the mean of the distribution of indicators and not their variability. Using the micro-data from the Italian evaluation of the quality of research (VQR, Valutazione della Qualita della Ricerca), we examine whether difference in performance between departments with different position in the ranking are distinguishable from random effects. We obtain a robust clustering of departments in a limited number of groups. The number of groups is in the range 3-7, while in most cases it is 4-6. The implications of these findings for evaluation and research policy are explored