Exploring a prototype framework of web-based and peer-reviewed “European Educational Research Quality Indicators” (EERQI)

Digitization, the Internet, and information or webometric interdisciplinary approaches are affecting the fields of Scientometrics and Library and Information Science (LIS). These new approaches can be used to improve citation-only procedures to estimate the quality and impact of research. A European pilot to explore this potential was called “European Educational Research Quality Indicators” (EERQI, FP7 # 217549). An interdisciplinary consortium was involved from 2008-2011. Different types of indicators were developed to score 171 educational research documents. Extrinsic bibliometric and citation indicators were collected from the Internet for each document; intrinsic indicators reflecting content-based quality were developed and relevant data gathered by peer review. Exploratory and confirmatory factor analysis and structural modeling were used to explore statistical relationships among latent factors or concepts and their indicators. Three intrinsic and two extrinsic latent factors were found to be relevant. Moreover, the more a document was related to a reviewer’s own area of research, the higher the score the reviewer gave concerning 1) significance, originality, and consistency, and 2) methodological adequacy. The conclusions are that a prototype EERQI framework has been constructed: intrinsic quality indicators add specific information to extrinsic quality or impact indicators, and vice versa. Also, a problem of “objective” impact scores is that they are based on “subjective” or biased peer-review scores. Peer-review, which is foundational to having a work cited, seems biased and this bias should be controlled or improved by more refined estimates of quality and impact of research. Some suggestions are given and limitations of the pilot are discussed. As the EERQI development approach, instruments, and tools are new, they should be developed further

A prototype empirical framework of intrinsic and extrinsic EERQI indicators

The research question is: What do statistical analyses show us about the relationships between intrinsic and extrinsic indicators of quality and what does this mean when constructing a prototype EERQI framework? The pilot study involved the scoring on both intrinsic and extrinsic indica-tors for 177 research documents or articles written by 268 authors. Intrinsic data were gathered by peer review and extrinsic data were collected from the Internet. Confirmatory Factor Analysis (CFA) resulted in a measurement model contain-ing three intrinsic and two extrinsic latent factors. There are significant correlations between the intrinsic latent factors and between the extrinsic latent factors; however, no significant correlations have been found between intrinsic and ex-trinsic factors. This outcome underlines the notion that intrinsic indicators may add specific quality information to an EERQI prototype that consists solely of extrinsic indicators, and vice versa. Testing by means of a structural model re-vealed that the more a reviewed document is related to the reviewer’s own area of research, the higher the score the reviewer gives the document with respect to 1) significance, originality and consistency and 2) methodological adequacy. No relationships were found between the reviewer’s own area of research and the extrinsic latent factors. These effects on the two intrinsic latent factors indicate that there may be some subjective evaluation bias in peer reviewing. The conclusion is that the outcomes of the statistical analyses seem plausible and support the validity of the conceptual framework. An initial prototype EERQI framework has been constructed, which is in line with the main goal of the EERQI project. Although the pilot had some methodological limitations, the present empirical outcomes are promising for future EERQI developmental and research activities, which could, for example, also integrate semantic latent factors and indicators.FP7 Collaborative project "European Educational Research Quality Indicators" (EERQI). (Project 217549)