Sustainable development goals research in higher education institutions: An interdisciplinarity assessment through an entropy-based indicator

publishedVersionPaid Open Acces

Development of academic ranking model in Croatia

Kvalitetno obrazovanje i doprinos svjetskom fondu znanja glavni su strateški ciljevi sustava obrazovanja i znanosti Republike Hrvatske. Ukoliko je kvalitetno obrazovanje poslanje hrvatskog obrazovnog sustava, potrebno je usuglasiti sve dionike oko definiranja kvalitete i modela njezina mjerenja kako bi sva visoka učilišta mogla prilagoditi svoje djelatnosti tome zajedničkome cilju. S druge strane, budući da „nacionalna“ znanost i visoko obrazovanje nisu izolirani sustavi, važno je kvalitetu promatrati i u globalnom svjetlu. Sustavi vrednovanja kvalitete – svjetska i nacionalna rangiranja te vrednovanja u hrvatskom sustavu visokog obrazovanja i znanosti su, u svojoj suštini, kombinacija odabranih pokazatelja koji mjere ulazne podatke (input), procese (process) i ishode (output) (Westerheijden, 1991) – jedna matrica ili poučavanje i učenje, istraživanje i društvena uloga – druga matrica. Pomoću ove dvije matrice moguće je opisati sustave vrednovanja na globalnoj i nacionalnoj razini te ih usporediti prema vrijednosti udjela pokazatelja u svakoj od kategorija. Analizom udjela svake od šest varijabli unutar ove dvije za svaki od šest analiziranih tipologija vrednovanja, dobiveni su rezultati, odnosno, koncept kvalitete koja se kroz svaki od tipologija na nacionalnoj i svjetskoj razini „nagrađuje“ kao poželjan. Sustav visokog obrazovanja u Hrvatskoj jedan je od rijetkih u Europi koji ne provodi nacionalno, niti globalno rangiranje institucija i(li) programa visokog obrazovanja, stoga potreba ujednačavanja definiranja i vrednovanja kvalitete na nacionalnoj razini treba obuhvatiti i njezinu usklađenost s međunarodnom perspektivom. Ona je, kako je ovo istraživanje potvrdilo, za sada deklarativne prirode, i kreće se od obrazovanja temeljenog na ulaznim podatcima (input), procesima (process) i poučavanju i učenju u smjeru obrazovanja i znanosti temeljenih na ishodima (output) i istraživanju. Ukoliko bi se željela potaći vidljivost te bolja kvaliteta ishoda nastavnog i istraživačkog procesa u Hrvatskoj, što je temeljna kategorija vrednovanja na međunarodnoj razini (rangiranjima), tada bi pokazatelji kvalitete ishoda (output) i istraživanja svakako trebali biti jače kapacitirani (u smislu broja i težinskih udjela) u sustavima vrednovanja i uspoređivanja kvalitete. Kao uravnotežen skup pokazatelja i težinskih udjela, predloženi model rangiranja sveučilišta u Hrvatskoj jedan je od prvih sustavno obrazloženih prijedloga, čija primjena zahtjeva konsenzus svih dionika u sustavu visokog obrazovanja i, naravno, detaljniju razradu u smislu realizacije.The quality and relevance of higher education is in the focus of European policy documents (European Commission, 2013). As a part of that environment, Croatia is also facing challenges of adjustment, dealing with and reviewing the status and role of the higher education institutions. Due to the increasing restrictions of state funding, the transformation of the education system into the student-oriented service, the need to prove the accountability and quality of the service offered by the higher education institutions, the increasing competitiveness among higher education institutions, the massification and internationalization of higher education (Avralev and Efimova, 2014), the academic ranking systems are becoming an increasingly common phenomenon. The information regarding the quality of the study programs and/or institutions and their status in comparison with other universities and programs (Altbach, 2010) becomes essential when it comes to choosing between a large number of higher education institutions and study programs (Blanco-Ramirez and Berger, 2014). In recent times the results of external quality assessment do not apply only to inform students and their parents about the quality of higher education institutions, but also to the geopolitical positioning of each country (EUA, 2014). The ranking of national universities and the number of universities that occupy high places on those rankings are increasingly considered to be one of the important elements of the country's economic competitiveness (Kishkovsky, 2012). Given that education has not the characteristics of a "product" (Majumdar, 1983; Winch, 2010), its quality is considered to be a multifaceted and multidimensional category difficult to be measured closely (Chattopadhyay, 2012). For this reason, the concept of quality should be analysed and defined in relation to the stakeholders in the higher education and science - prospective students, students, institutions, staff at higher education institutions, employers and financiers. Westerheijden (2007) gives priority to the importance of achieving a broad consensus on what the term quality encompasses in a particular education system over the attempts to define this ambiguous contextual concept. Each one of the approaches to defining the quality has implications for the quality assurance system and the policies adopted in each higher education system. Behind the context of the quality in a higher education system usually lies the idea of higher education, and given that the meaning and context of higher education changes over time, in that sense the notion of the quality refers to the values, goals, desired actions, experiences and results of higher education (Boyle and Bowden, 1997). In most cases, it is about approaching the quality as fitness for purpose, which allows the higher education institutions to evaluate and define the quality in line with their missions and goals (Woodhouse, 1999; Nicholson, 2011). The methodological approach to quality assessment and qualitative or quantitative procedures depends on the definition of quality requirements. Some types of assessment require the use of quantitative methods (e.g., ranking, thematic evaluations), while for others are much more acceptable qualitative methods (accreditation). The selection of indicators will depend on the choice of evaluation methods (qualitative or quantitative) and their possible complementary use in the same evaluation. The performance indicators are considered to provide clear, objective and measurable information that may serve as a solid basis for some (political) decision (Van Vught and Westerheijden, 1994). This clearly links the procedures of quality assurance with the intended results of the quality ensuring policy. The problem of the use of performance indicators is their applicability in the field of higher education and education in general considering the complexity, including the comparability of the data collected. In fact, it is sometimes difficult to obtain data that are mutually comparable. The assessment models are a combination of indicators that measure input, process and output, while the choice of the quality assessment model is based on the particularities of the national system, the current quality assessment model at international level, and finally, the availability of information. The latter can have negative consequences in cases when the emphasis is placed on measurable, not relevant indicators. The external quality assessment procedures vary within the national higher education systems and vary in their scope and focus. In today's systems of the external quality assessment of universities there are basically four models: accreditation, auditing, benchmarking and classification (including ranking). They further vary according to the subject and emphasis of the evaluation. The focus of the evaluation also changes, sometimes emphasizing the management and regulation or financial sustainability, and sometimes the student experiences, learning, curriculum development, curriculum design and competence of teachers. In the context of this research, a systematic review includes all assessment models existing in Croatia, while the analytical part is focused on the contents of the two procedures of the university quality assessment: re-accreditation and university rankings. Due to their importance, the university rankings are divided into national systems and ranking leagues, with particular emphasis placed on their contents. According to the purpose and methodology, the accreditation and ranking are different quality assessment tools. Both systems allow a certain way of description and comparison of the quality of institution and (or) program, based on the indicators used for the quality measuring and assessment. The main problems to which this research seeks to provide answers are related to an ambivalent understanding of the concept of quality which in Croatian system of higher education is not clearly defined and varies depending on the stakeholders, and to a problem of poor visibility of Croatian higher education institutions in the world rankings (Jokić et al., 2012). Among other, the reasons for that can be found in the specifics and large mutual differences of national institutions with respect to the size and profile (public / private universities, non-integrated / integrated universities; public / private universities of applied sciences and colleges, public / private research institutes). In addition to the specific structure of the Croatian higher education, the reason also lies in an increasing number and scope of external quality assessment procedures which still do not have a comparative element, and finally, in the absence of ranking system at national level that would enable a comparison of Croatian higher education institutions. On the one hand, using a matrix analysis of the performance indicators of higher education institutions, the research seeks to identify relevant indicators in ranking systems according to their share in the following categories: input, process and output, as well as teaching and learning, research and university third mission, and to determine differences between those and the indicators used as a quality measure in the national framework of its measurements. The objective of this paper is, on the basis of the methodology application and the results used in the process of re-accreditation of Croatian universities, as well as the results of the analysis of the methodological approach elaboration, the most commonly used global and national rankings, to propose a model of ranking of the higher education institutions in Croatia. The dissertation analyses the differences and specificities of the international and national evaluation models and understanding of quality as a basis for modelling. In order to determine areas of success of Croatian universities in different scientific fields have been used data and quality grades of public universities realized in existing national model of quality assessment – the re-accreditation. A comparison with international models of quality assessment will be based on the five most influential global rankings. In order to be able to mutually compare different models of the quality assessment of programs and (or) universities that have a different purpose, but use the same instruments - adequate indicators, documents that describe them - in this research they are divided into six analytical sections: 1. World academic rankings ARWU (Academic Ranking of World Universities), THE (Times Higher Education World University Rankings), QS (Quaquarelli Simmonds Top Universities), Webometrics Ranking of World Universities, CWTS Leiden Ranking and SJR (SCImago Institutions Rankings) 2. National academic rankings (Macedonian University Rankings, Macedonia, Bulgarian University Ranking System, Bulgaria, Perspektywy Ranking, Poland, Akademická rankingová a ratingová agentúra (ARRA), Slovakia) 3. Policy documents in the Croatian system of higher education and science (Strategy for Science, education and Technology, Three year funding agreements between the state and public HE institutions, Criteria for academic promotion) 4. Principles and criteria for the quality assessment of institutions and (or) programs in the external assessment procedures (Ordinance on Conditions for Issuing Licence for Scientific Activity, Conditions for Re-accreditation of Scientific Organisations and Content of Licence, Thematic evaluation Criteria), Higher education audit criteria, Principles and criteria for evaluation of scientific institution in Croatia, Reaccreditation of doctoral study programmes) 5. Criteria for the quality assessment of higher education institutions within universities 6. Quality grades of higher education institutions within public universities (technology, science and humanities) It is possible to analyse the indicators of quality in the framework of samples divided in this manner with regard to their content (according to Neuendorf, 2002) in two matrices: input - process - output, and teaching and learning - research - university third mission, and determine towards which group the results within these matrices deflect. With each pattern it is also described a process of its equalization with other samples, and as well the preparations for the most relevant comparison. The methodological limitations regarding the use of this approach are basically related to the size of the sample which, although comprising the entire population, represents a small sample. On such sample it possible to conduct the correspondence analysis only by using the descriptive statistical methods. Due to above limitations of the sample, it is not possible to confirm it by inferential statistics. By analytical elaboration and comparison of these six units are established differences between the Croatian and world models of the quality assessment, while their common features are the starting point for drafting a model of ranking of the higher education institutions in Croatia. The results of this research have confirmed all three hypotheses: 1. Hypothesis: Global systems of academic ranking include the indicators which are divided into three groups: input, process and output, and focused mainly on the output. -- The analysis of the world's most famous university rankings shows obvious deflection towards the measurement of the output (at least 66.7% of the overall assessment). The input is measured only in three world rankings - Webometrics, QS ranking and SCImago, but its share does not exceed one third of the overall assessment (max. 33.3% in SCImago ranking). Among the analysed rankings, THE ranking places emphasis on the process (32.5%). CWTS Leiden ranking evaluates exclusively the output. -- According to the second matrix, in the academic rankings most of the indicators and their weights refer to the research (Chart 2). Another interesting fact is that in the QS and Webometrics rankings half of the overall assessment refers to university third mission. CWTS Leiden and SCImago Institutions rankings do not include any indicators of the quality of teaching component, while the lowest share of indicators in the world rankings refers to teaching and learning (10- 30%). -- The national rankings according to the input-process-output matrix included in the analysis in the most cases rank the institutions on the basis of the output. Concerning the input, unlike the world rankings, the national rankings are taking it into account in the range of 23-11%, wherein only the Bulgarian University Ranking System does not take into account the indicators of input. Furthermore, all national rankings also include the indicators of process in the overall ranking, but their share in the overall assessment does not exceed 35% (Bulgarian University Ranking System). The output indicators in all analysed national rankings have a significant impact on the overall assessment and their share does not drop below 43%. -- In the teaching and learning - research – university third mission matrix, an overall assessment in the most of the national rankings is based on the indicators of the quality of teaching and learning. The indicators of the quality of research component are essential in ARRA ranking (50%) and Perspektywy (44%), and the lowest in Bulgaria's ranking system. Interestingly, in overall assessments that form national rankings, the indicators of university third mission have the least weight (Chart 4). -- The comparison of world and selected national rankings according to the inputprocess-output and teaching and learning-research-university third mission matrices showed similarities in two categories: the share of indicators of the output quality which comprises an average of more than two-thirds of all indicators, and the share of indicators of the quality of the university research component. An analysis according to individual ranking methodologies shows that among all world rankings CWTS Leiden Ranking places the highest emphasis on research and output, while the QS ranking is focused on the quality of teaching and learning, and the input. In average, the national rankings slightly more value the input and the quality of teaching and learning, but in this typology as well, as in the world rankings, the highest share have the qualities of research and output as categories primarily used for measuring the scientific component. 2. Hypothesis: Concept of quality in Croatian higher education system is not clearly defined, varies depending on stakeholders and it is focused on measuring the input and process. -- Observing the policy documents cumulatively, the distribution of categories and indicators of the quality assessment between two matrices is uneven. Despite the fact that some of the documents tend towards input (Ordinance on the Content of Licence and Conditions for Issuing Licence for Performing Higher Education Activity, carrying out a Study Programme and Re-accreditation of Higher Education Institutions), or output (Three year funding agreements between the state and public HE institutions (scientific activity), Principles and criteria for evaluation of scientific institution in Croatia, and Criteria for academic promotion), each of the analysed documents includes the evaluation of the process. The process (in more than half of the indicators) are evaluated by Strategy for Science, education and Technology, Three year funding agreements between the state and public HE institutions (higher education) and Criteria for the assessment of quality of higher education institutions within universities. These results point to the inconsistency of defining and measuring the quality in the evaluations carried out in the national system of higher education and science. -- The analysis of indicators according to the teaching and learning, research and university third mission matrix shows that elements of higher education institutions related to teaching and learning, as well as research have a similar weight, while the university third mission is included in each of the analysed documents, with the exception of Three year funding agreements between the state and public HE institutions (higher education). -- In the process of re-accreditation the input is measured in a slightly larger ratio than the output, which can be explained by the fact that the quality assessment in the context of re-accreditation includes the evaluation of the minimum conditions which is, in its essence, a check of the teaching and physical resources (input). The average share of the quality of teaching and learning, as well as research, in the overall quality assessment was 62.4%. The share of indicators and quality assessment of research in the process of re-accreditation is represented by less than a third (28.1%). -- The results of the comparative analysis of existing systems of the external assessment of universities in Croatia and selected world ranking systems show significant differences in shares of quality indicators considering the input- process- output and teaching and learning-research-university third mission. While models of the world and national rankings do not differ in placing the emphasis on output, Croatian systems of the university quality assessment are based on the quality of process, i.e., the activities taking place at the institutions of higher education. This indicates that the national assessment systems differ from the models relevant at global level. -- The comparative analysis of data from the teaching and learning-research-university third mission matrix shows that in both ranking typologies, national and world, the research is most valued, while Croatian assessment systems are focused on the third mission and teaching and learning. The category of university third mission in the world rankings is represented with an average share of 18.46%, while in the selected national university rankings is represented with significantly higher share of 25.25%. 3. Hypothesis: Croatian institutions of higher education have the elements of excellence at the level of individual segments of their activities. -- This research includes Croatian universities from the fields of technology, science and humanities, a total of 39. Despite the fact that the sample is not large enough to draw statistically validated conclusions, using the descriptive statistical analysis it is possible to encompass the characteristics of quality grades for the fields of technology, science and humanities. The process and the input, as well as the quality of teaching and learning are best rated elements in all analysed universities. In the category of research that has a significant share in academic rankings, the higher education institutions in the field of technology are the highest rated (55.40% rated as "mostly implemented") and science (the indicators

Propuesta de un modelo de análisis redinformétrico multinivel para el estudio sistémico de las universidades españolas (2010)

La universidad, en tanto que institución milenaria, tiene una influencia y peso en la sociedad actual incuestionable. Una influencia tanto activa (en su vertiente formadora de futuros profesionales y ciudadanos, y de generación de nuevo conocimiento e investigación) como pasiva (debido a sus enormes necesidades de financiación). Este peso e influencia de las universidades en la sociedad marcan la necesidad de establecer mecanismos y procedimientos para analizar su rendimiento, eficiencia y eficacia como institución, así como de instrumentos para visualizar adecuadamente este rendimiento, todo ello en el contexto de una sociedad marcada por la gestión y transferencia masiva de información a través de las redes de comunicación. Se vislumbran por tanto 3 líneas de investigación complementarias: el análisis de la universidad, su rendimiento (explicitado por el rastro digital que ésta genera y las técnicas existentes para cuantificarlo), y la visualización de este rendimiento, donde la técnica de ranking es la más extendida dado el impacto que su formato genera en los usuarios. Además, la naturaleza sistémica de la universidad determina, complica y acota cada una de estas áreas. La presente tesis doctoral pretende por tanto explorar las capacidades que la cibermetría (renombrada en este trabajo como redinformetría) proporciona para analizar, desde un punto de vista sistémico, las universidades del sistema español, con el propósito de obtener nuevo conocimiento acerca del rendimiento de éstas que permita la construcción futura de rankings sistémicos de universidades. Para ello, en primer lugar se ofrece un estado de la cuestión enfocado en las 3 principales líneas de trabajo (universidad, ranking y cibermetría), con el objetivo de ofrecer un marco de trabajo exhaustivo y crítico. Tras la parte introductoria, se propone un modelo de análisis redinformétrico multinivel de universidades que facilite la obtención de información estructurada, y que permita su posterior utilización en el diseño y elaboración de rankings web de universidades. Este modelo de análisis se basa en el establecimiento de 3 niveles (institucional, externo y satélite) y dos subniveles (contorno e interno). Los resultados obtenidos muestran que el modelo de análisis propuesto, basado en niveles (institucional, externo y satélite) y subniveles (contorno y unidad) es sencillo, independiente de técnica y proporciona información estructurada que permite un análisis completo de cada institución.Orduña Malea, E. (2012). Propuesta de un modelo de análisis redinformétrico multinivel para el estudio sistémico de las universidades españolas (2010) [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/14420Palanci