Quantitative Analysis of the Italian National Scientific Qualification

The Italian National Scientific Qualification (ASN) was introduced in 2010 as part of a major reform of the national university system. Under the new regulation, the scientific qualification for a specific role (associate or full professor) and field of study is required to apply to a permanent professor position. The ASN is peculiar since it makes use of bibliometric indicators with associated thresholds as one of the parameters used to assess applicants. Overall, more than 59000 applications were submitted, and the results have been made publicly available for a short period of time, including the values of the quantitative indicators for each applicant. The availability of this wealth of information provides an opportunity to draw a fairly detailed picture of a nation-wide evaluation exercise, and to study the impact of the bibliometric indicators on the qualification results. In this paper we provide a first account of the Italian ASN from a quantitative point of view. We show that significant differences exist among scientific disciplines, in particular with respect to the fraction of qualified applicants, that can not be easily explained. Furthermore, we describe some issues related to the definition and use of the bibliometric indicators and thresholds. Our analysis aims at drawing attention to potential problems that should be addressed by decision-makers in future ASN rounds.Comment: ISSN 1751-157

Assessing evaluation procedures for individual researchers: the case of the Italian National Scientific Qualification

The Italian National Scientific Qualification (ASN) was introduced as a prerequisite for applying for tenured associate or full professor positions at state-recognized universities. The ASN is meant to attest that an individual has reached a suitable level of scientific maturity to apply for professorship positions. A five member panel, appointed for each scientific discipline, is in charge of evaluating applicants by means of quantitative indicators of impact and productivity, and through an assessment of their research profile. Many concerns were raised on the appropriateness of the evaluation criteria, and in particular on the use of bibliometrics for the evaluation of individual researchers. Additional concerns were related to the perceived poor quality of the final evaluation reports. In this paper we assess the ASN in terms of appropriateness of the applied methodology, and the quality of the feedback provided to the applicants. We argue that the ASN is not fully compliant with the best practices for the use of bibliometric indicators for the evaluation of individual researchers; moreover, the quality of final reports varies considerably across the panels, suggesting that measures should be put in place to prevent sloppy practices in future ASN rounds

Reformism and Evaluation in the Field of Social and Political Sciences. Consequences for the Academic Community, Projects, People

After the introduction of the most recent University evaluation reforms, this article aims at presenting the first findings of a research project on these reforms and the evaluation processes carried out so far in the field of social and political sciences. In particular, the research wishes to investigate the relational dynamics, the behaviors and the values of the academic profession with special attention to the scientific activity, but without putting aside the implications for the teaching, management and the “third mission” activities. Inserted in the line of studies on higher education and evaluation well consolidated in Italy, the research is characterized by the added value  of  connecting the criticaltheoretical level to the empirical one

Bayesian binary quantile regression for the analysis of Bachelor-Master transition

The multi-cycle organization of modern university systems stimulates the interest in studying the progression to higher level degree courses during the academic career. In particular, after the achievement of the first level qualification (Bachelor degree), students have to decide whether to continue their university studies, by enrolling in a second level (Master) programme, or to conclude their training experience. In this work we propose a binary quantile regression approach to analyze the Bachelor-Master transition phenomenon with the adoption of the Bayesian inferential perspective. In addition to the traditional predictors of academic outcomes, such as the personal characteristics and the field of study, different aspects of the student's performance are considered. Moreover, a new contextual variable, indicating the type of university regulations, is taken into account in the model specification. The utility of the Bayesian binary quantile regression to characterize the non-continuation decision after the first cycle studies is illustrated with an application to administrative data of Bachelor graduates at the School of Economics of Sapienza University of Rome and compared with a more conventional logistic regression approach.Comment: 24 pages, 7 figures and 3 table

Studying the emergence of a new social representation: Changes in thinking about nanotechnologies in early 21st-century Italy

We investigated the emergence of a new social representation (SR) of a techno\u2010scientific innovation\u2014nanotechnologies\u2014among the Italian public. We reviewed how nanotechnologies entered parliamentary debates and the media agenda in the early third millennium. We conducted cross\u2010sectional surveys in 2006 (N = 246) and 2011 (N = 486) to examine the emerging SR of nanotechnologies. We sought to observe processes of anchoring and objectification \u2018in action\u2019, by analyzing roles of (i) social groups, and (ii) neighboring SRs of science and of technology, over time. Several changes from 2006 to 2011 were identified: From a \u2018descriptive\u2019 to an \u2018evaluative\u2019 approach; from a \u2018neutral\u2019 to a \u2018controversial\u2019 issue; from a \u2018concrete\u2019 to an \u2018abstract\u2019 object; and from a \u2018technological\u2019 to a \u2018scientific\u2019 phenomenon. We conclude that nanotechnologies finally became \u2018relevant enough\u2019 by 2011 to be considered a proper object of SR, and an emerging SR can be observed

Public initiatives of settlement transformation. A theoretical-methodological approach to selecting tools of multi-criteria decision analysis

In Europe, the operating context in which initiatives of settlement transformation are currently initiated is characterized by a complex, elaborate combination of technical, regulatory and governance-related factors. A similar set of considerations makes it necessary to address the complex decision-making problems to be resolved through multidisciplinary, comparative approaches designed to rationalize the process and treat the elements to be considered in systematic fashion with respect to the range of alternatives available as solutions. Within a context defined in this manner, decision-making processes must often be used to obtain multidisciplinary and multidimensional analyses to support the choices made by the decision-makers. Such analyses are carried out using multi-criteria tools designed to arrive at syntheses of the numerous forms of input data needed to describe decision-making problems of similar complexity, so that one or more outcomes of the synthesis make possible informed, well thought-out, strategic decisions. The technical literature on the topic proposes numerous tools of multi-criteria analysis for application in different decision-making contexts. Still, no specific contributions have been drawn up to date on the approach to take in selecting the tool best suited to providing adequate responses to the queries of evaluation that arise most frequently in the various fields of application, and especially in the settlement sector. The objective of this paper is to propose, by formulating a taxonomy of the endogenous and exogenous variables of tools of multi-criteria analysis, a methodology capable of selecting the tool best suited to the queries of evaluation which arise regarding the chief categories of decision-making problems, and particularly in the settlement sector

The role of citation networks to explain academic promotions: an empirical analysis of the Italian national scientific qualification

The aim of this paper is to study the role of citation network measures in the assessment of scientific maturity. Referring to the case of the Italian national scientific qualification (ASN), we investigate if there is a relationship between citation network indices and the results of the researchers’ evaluation procedures. In particular, we want to understand if network measures can enhance the prediction accuracy of the results of the evaluation procedures beyond basic performance indices. Moreover, we want to highlight which citation network indices prove to be more relevant in explaining the ASN results, and if quantitative indices used in the citation-based disciplines assessment can replace the citation network measures in non-citation-based disciplines. Data concerning Statistics and Computer Science disciplines are collected from different sources (ASN, Italian Ministry of University and Research, and Scopus) and processed in order to calculate the citation-based measures used in this study. Then, we apply logistic regression models to estimate the effects of network variables. We find that network measures are strongly related to the results of the ASN and significantly improve the explanatory power of the models, especially for the research fields of Statistics. Additionally, citation networks in the specific sub-disciplines are far more relevant than those in the general disciplines. Finally, results show that the citation network measures are not a substitute of the citation-based bibliometric indices