Gogolin, Ingrid

Gradmann, Stefan

Åström, Fredrik

English

Lund University Publications

LUND UNIVERSITYPO Box 117221 00 Lund+46 46-222 00 00European Educational Research Quality Indicators (EERQI)Gogolin, Ingrid; Gradmann, Stefan; Åström, FredrikPublished in:Proceedings of ISSI 2009: The 12th International Conference of the International Society for Scientometrics andInformetrics: Volume 22009Link to publicationCitation for published version (APA):Gogolin, I., Gradmann, S., & Åström, F. (2009). European Educational Research Quality Indicators (EERQI).Proceedings of ISSI 2009: The 12th International Conference of the International Society for Scientometrics andInformetrics: Volume 2, 934-935.Total number of authors:3General rightsUnless other specific re-use rights are stated the following general rights apply:Copyright and moral rights for the publications made accessible in the public portal are retained by the authorsand/or other copyright owners and it is a condition of accessing publications that users recognise and abide by thelegal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private studyor research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portalRead more about Creative commons licenses: https://creativecommons.org/licenses/Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will removeaccess to the work immediately and investigate your claim.European Educational Research Quality Indicators (EERQI) Ingrid Gogolin1, Stefan Gradmann2 and Fredrik Åström3 1 gogolin@uni-hamburg.de University of Hamburg, Department for International, Comparative and Multicultural Education, Von-Melle-Park 8, D-20146 Hamburg (Germany) 2 stefan.gradmann@ibi.hu-berlin.de Humboldt-Universität zu Berlin, Berlin School of Library and Information Science, Unter den Linden 6, 10099 Berlin (Germany) 3 fredrik.astrom@lub.lu.se Lund University Libraries, Head Office, P.O. Box 134, SE-221 00 Lund (Sweden) Introduction Metrics-based quality assessment of scientific publications have depended heavily on ranking methods involving citation frequencies, largely based on data from the ISI Thomson databases. Within the ISI databases, differences in terms of coverage between fields are well known, as is the dominance of publications in English, affecting e.g. the social sciences and the humanities. Issues concerning analyzing the social sciences and humanities have been approached by e.g. Butler and Visser (2006), analyzing non-source item citation; and by van Leeuwen (2006), analyzing publication and citation practices in the social sciences. However, both articles are based on ISI data; and since much of e.g. the European social sciences publish in non-journal media and in their respective domestic languages: we know little about how representative these analyses actually are. Furthermore, we still know relatively little about how varying citations practices affect the citation analyses, as well as of the nature of these citation practices in different fields. Objectives These issues are approached in the European Educational Research Quality Indicators (EERQI) project, whose broadly defined aim is to develop a prototype framework for relevance assessment of research publications in European educational research. Together with traditional indicators, new methods and indicators will also be tested, such as semantic-based full text analysis for tracking evidence of quality in text and references.  More specifically, the goals of the project are to: - Develop new indicators and methodologies for quality assessment. - Make this framework operational on a multilingual basis (starting with English, German, French and Swedish). - Produce a search and query engine for resource harvesting and text analysis. - Test transferability of the EERQI indicators into other fields in social sciences and the humanities. - Develop a sustainability plan for quality assessment of European educational research publications.  Analyzing educational research as an example of academic activities in the social sciences and humanities is largely because of its nature as an interdisciplinary field of research, covering the whole range of methodologies and epistemological orientations found in social science and humanities research. Using European educational research as case for the analyses is motivated by there being a joint institutional research infrastructure as expressed in e.g. the European Educational Research Association and the European Association for Learning and Instruction, at the same time as much of the research is performed in the domestic languages of the member countries. Tentative citation analysis To get an overview of publication and citation patterns in the field, research articles proper from the 20 highest cited journals in the ‘Education and Educational Research’ JCR category was downloaded from the Web of Science (omitting specialized journals and journals with a domestic focus). Out of 4,386 research articles citing a total of 179,240 references, 30% of the references were to other journal articles. 48% of the author addresses were from the U.S and 18% from the U.K (46% of the European total). The most cited authors are dominated by citation classics in the field, such as Vygotsky and Piaget; and general theorists such as Foucault and Bourdieu. Furthermore, when applying a citation among documents analysis (Persson, 2001), only 2% of the total amount of citations goes back to source item articles; and the majority of these are limited to one research specialization within the larger field. Thus, we have a large dominance of Anglo-American articles; and also, an indication that ‘citation traffic’ to active peers within the field is relatively slow. The question whether this information is representative to the field as a whole remains, but having access to data from European research publications in different languages and in different media of publication, we have the opportunity to test this. Content base and prototype search engine The content base is built by gathering research documents including books and book chapters, as well as journal articles, supplied by the academic publishers being part of the project. The data comes in form of pdf-files with xml-files containing document metadata. In addition to building the local document corpus, a search and query engine is being developed for resource harvesting and data analysis; as well as for finding new educational research documents publicly available at the World Wide Web. To deal with the documents, two indices have been created: a content index and a metadata index building on Lucene, an open-source Java library, whereas the the WWW index builds on Nutch for crawling and indexing URLs. Quality indicators One basic idea in the EERQI project is to combine different indicators for document quality assessment. One type of metrics is different usage indicators, e.g. download frequencies and citation analysis. Another type of metrics to investigate is analyses of other forms of metadata, e.g. information on authors and institutions, as well as keywords and their relation to concepts of relevance. In addition to these kinds of quantitative analyses, semantic processing tools are used to identify evidence of quality by combining quality indicators identified by e.g. peer reviewers with semantic analyses of research documents. Results Semantic analyses A main theme in the EERQI project is to utilize automated semantic analyses of documents and references to identify evidence of quality by using the natural language processing tool Xerox Incremental Parser (XIP) (Aït et.al., 2001; 2002), e.g. for identifying different kinds of citations. In relation to this, tentative analyses on citation practices in e.g. education research has been done to see to what extent research texts follow the normative idea of research being cumulative, one of the main assumptions for using citations as an indicator of quality. Based on science studies theories and the result of co-citation based research maps, together with semantic analyses, Åström and Sándor (2009) found evidence of research organized and communicated in a negotiating and distinctive way, in addition to being cumulative. Further more, the semantic analyses are also used to identify evidence of quality in the documents per se. Based on a set of quality criteria and how peer reviewers applies these, the XIP is ‘trained’ to identify conceptual, factual and referential expressions in the text reflecting aspects of research quality. In addition to this, the semantic analyses can also be used for extracting metadata out of the documents, which might not necessarily be reflecting research quality per se, but that can be of great interest for conceptual mapping of the research field, as well as for developing relevance criteria for ranking search results in the EERQI prototype search engine. Classification of indicators In addition to this, there is also important conceptual work going on, identifying and categorizing quality indicators. There are the indicators themselves: e.g. rigour and significance, as well as usage indicators; but also the distinction between intrinsic indicators, being directly related to the text itself, and extrinsic indicators, not inherently constituting quality with a probabilistic relation to quality. Acknowledgments The project is funded by the European Union’s 7th Framework Programme for Research in the Socio-economic Sciences and Humanities Theme (SSH). References Aït-Mokhtar, S., Chanod, J-P. & Roux, C. (2001). A multi-input dependency parser. In: Proceedings of the Seventh IWPT (International Workshop on Parsing Technologies), Beijing, China, 17-19 October, 2001. Beijing: Tsinghua University Press. Aït-Mokhtar, S., Chanod, J-P. & Roux, C. (2002). Robustness beyond shallowness: Incremental dependency parsing. Natural Language Engineering, 8(2/3), 121-144. Åström, F. & Sándor, Á. (2009). Models of scholarly communication and citation analysis. In B. Larsen & J. Leta (Eds.), Proceedings of the 12th International Conference on Scientometrics and Informetrics (ISSI 2009). Rio de Janeiro, Brazil, 14-17 July 2009. Butler, L. & Visser, M.S. (2006). Extending citation analysis to non-source items. Scientometrics. 66(2), 327-343. Persson, O. (2001). All author co-citations versus first author co-citations. Scientometrics, 50(2), 339-344. van Leeuwen, T. (2006). The application of bibliometrics analyses in the evaluation of social science research: Who benefits from it, and why it is still feasible. Scientometrics, 66(1), 133-154. 

European Educational Research Quality Indicators (EERQI)

https://lup.lub.lu.se/search/files/3653120/1883085.pdf

European Educational Research Quality Indicators (EERQI)

Abstract

Similar works

Full text

Available Versions

Lund University Publications