Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media

We present an overview of the third edition of the CheckThat! Lab at CLEF 2020. The lab featured five tasks in two different languages: English and Arabic. The first four tasks compose the full pipeline of claim verification in social media: Task 1 on check-worthiness estimation, Task 2 on retrieving previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on claim verification. The lab is completed with Task 5 on check-worthiness estimation in political debates and speeches. A total of 67 teams registered to participate in the lab (up from 47 at CLEF 2019), and 23 of them actually submitted runs (compared to 14 at CLEF 2019). Most teams used deep neural networks based on BERT, LSTMs, or CNNs, and achieved sizable improvements over the baselines on all tasks. Here we describe the tasks setup, the evaluation results, and a summary of the approaches used by the participants, and we discuss some lessons learned. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.Comment: Check-Worthiness Estimation, Fact-Checking, Veracity, Evidence-based Verification, Detecting Previously Fact-Checked Claims, Social Media Verification, Computational Journalism, COVID-1

ImageCLEF 2013: The vision, the data and the open challenges

This paper presents an overview of the ImageCLEF 2013 lab. Since its first edition in 2003, ImageCLEF has become one of the key initiatives promoting the benchmark evaluation of algorithms for the cross-language annotation and retrieval of images in various domains, such as public and personal images, to data acquired by mobile robot platforms and botanic collections. Over the years, by providing new data collections and challenging tasks to the community of interest, the ImageCLEF lab has achieved an unique position in the multi lingual image annotation and retrieval research landscape. The 2013 edition consisted of three tasks: the photo annotation and retrieval task, the plant identification task and the robot vision task. Furthermore, the medical annotation task, that traditionally has been under the ImageCLEF umbrella and that this year celebrates its tenth anniversary, has been organized in conjunction with AMIA for the first time. The paper describes the tasks and the 2013 competition, giving an unifying perspective of the present activities of the lab while discussion the future challenges and opportunities.This work has been partially supported by the Halser Foundation (B. C.),by the LiMoSINe FP7 project under grant # 288024 (B. T.), by the Khresmoi (grant# 257528) and PROMISE ( grant # 258191) FP 7 projects (H.M.) and by the tranScriptorium FP7 project under grant # 600707 (M. V., R. P.)Caputo ., B.; Muller ., H.; Thomee ., B.; Villegas, M.; Paredes Palacios, R.; Zellhofer ., D.; Goeau ., H.... (2013). ImageCLEF 2013: The vision, the data and the open challenges. En Information Access Evaluation. Multilinguality, Multimodality, and Visualization. Springer Verlag (Germany). 8138:250-268. https://doi.org/10.1007/978-3-642-40802-1_26S2502688138Muller, H., Clough, P., Deselaers, T., Caputo, B.: ImageCLEF: experimental evaluation in visual information retrieval. Springer (2010)Tsikrika, T., Seco de Herrera, A.G., Müller, H.: Assessing the scholarly impact of imageCLEF. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 95–106. Springer, Heidelberg (2011)Huiskes, M., Lew, M.: The MIR Flickr retrieval evaluation. In: Proceedings of the 10th ACM Conference on Multimedia Information Retrieval, Vancouver, BC, Canada, pp. 39–43 (2008)Huiskes, M., Thomee, B., Lew, M.: New trends and ideas in visual concept detection. In: Proceedings of the 11th ACM Conference on Multimedia Information Retrieval, Philadelphia, PA, USA, pp. 527–536 (2010)Villegas, M., Paredes, R.: Overview of the ImageCLEF 2012 Scalable Web Image Annotation Task. In: CLEF 2012 Evaluation Labs and Workshop, Online Working Notes, Rome, Italy (2012)Zellhöfer, D.: Overview of the Personal Photo Retrieval Pilot Task at ImageCLEF 2012. In: CLEF 2012 Evaluation Labs and Workshop, Online Working Notes, Rome, Italy (2012)Villegas, M., Paredes, R., Thomee, B.: Overview of the ImageCLEF 2013 Scalable Concept Image Annotation Subtask. In: CLEF 2013 Evaluation Labs and Workshop, Online Working Notes, Valencia, Spain (2013)Zellhöfer, D.: Overview of the ImageCLEF 2013 Personal Photo Retrieval Subtask. In: CLEF 2013 Evaluation Labs and Workshop, Online Working Notes, Valencia, Spain (2013)Leafsnap (2011)Plantnet (2013)Mobile flora (2013)Folia (2012)Goëau, H., Bonnet, P., Joly, A., Bakic, V., Boujemaa, N., Barthelemy, D., Molino, J.F.: The imageclef 2013 plant identification task. In: ImageCLEF 2013 Working Notes (2013)Pronobis, A., Xing, L., Caputo, B.: Overview of the CLEF 2009 robot vision track. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009. LNCS, vol. 6242, pp. 110–119. Springer, Heidelberg (2010)Pronobis, A., Caputo, B.: The robot vision task. In: Muller, H., Clough, P., Deselaers, T., Caputo, B. (eds.) ImageCLEF. The Information Retrieval Series, vol. 32, pp. 185–198. Springer, Heidelberg (2010)Pronobis, A., Christensen, H.I., Caputo, B.: Overview of the imageCLEF@ICPR 2010 robot vision track. In: Ünay, D., Çataltepe, Z., Aksoy, S. (eds.) ICPR 2010. LNCS, vol. 6388, pp. 171–179. Springer, Heidelberg (2010)Martinez-Gomez, J., Garcia-Varea, I., Caputo, B.: Overview of the imageclef 2012 robot vision task. In: CLEF 2012 Working Notes (2012)Rusu, R., Cousins, S.: 3d is here: Point cloud library (pcl). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–4. IEEE (2011)Bosch, A., Zisserman, A., Munoz, X.: Image classification using random forests and ferns. In: International Conference on Computer Vision, pp. 1–8. Citeseer (2007)Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 1, pp. 886–893. IEEE (2005)Linde, O., Lindeberg, T.: Object recognition using composed receptive field histograms of higher dimensionality. In: Proc. ICPR. Citeseer (2004)Orabona, F., Castellini, C., Caputo, B., Luo, J., Sandini, G.: Indoor place recognition using online independent support vector machines. In: Proc. BMVC, vol. 7 (2007)Orabona, F., Castellini, C., Caputo, B., Jie, L., Sandini, G.: On-line independent support vector machines. Pattern Recognition 43, 1402–1412 (2010)Orabona, F., Jie, L., Caputo, B.: Online-Batch Strongly Convex Multi Kernel Learning. In: Proc. of Computer Vision and Pattern Recognition, CVPR (2010)Orabona, F., Jie, L., Caputo, B.: Multi kernel learning with online-batch optimization. Journal of Machine Learning Research 13, 165–191 (2012)Clough, P., Müller, H., Sanderson, M.: The CLEF 2004 cross-language image retrieval track. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 597–613. Springer, Heidelberg (2005)Clough, P., Müller, H., Deselaers, T., Grubinger, M., Lehmann, T.M., Jensen, J., Hersh, W.: The CLEF 2005 cross–language image retrieval track. In: Peters, C., Gey, F.C., Gonzalo, J., Müller, H., Jones, G.J.F., Kluck, M., Magnini, B., de Rijke, M., Giampiccolo, D. (eds.) CLEF 2005. LNCS, vol. 4022, pp. 535–557. Springer, Heidelberg (2006)Müller, H., Deselaers, T., Deserno, T., Clough, P., Kim, E., Hersh, W.: Overview of the imageCLEFmed 2006 medical retrieval and medical annotation tasks. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 595–608. Springer, Heidelberg (2007)Müller, H., Deselaers, T., Deserno, T., Kalpathy–Cramer, J., Kim, E., Hersh, W.: Overview of the imageCLEFmed 2007 medical retrieval and medical annotation tasks. In: Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D.W., Peñas, A., Petras, V., Santos, D. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 472–491. Springer, Heidelberg (2008)Müller, H., Kalpathy–Cramer, J., Eggel, I., Bedrick, S., Radhouani, S., Bakke, B., Kahn Jr., C.E., Hersh, W.: Overview of the CLEF 2009 medical image retrieval track. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009, Part II. LNCS, vol. 6242, pp. 72–84. Springer, Heidelberg (2010)Tommasi, T., Caputo, B., Welter, P., Güld, M.O., Deserno, T.M.: Overview of the CLEF 2009 medical image annotation track. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009. LNCS, vol. 6242, pp. 85–93. Springer, Heidelberg (2010)Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds.): ImageCLEF – Experimental Evaluation in Visual Information Retrieval. The Springer International Series on Information Retrieval, vol. 32. Springer, Heidelberg (2010)Kalpathy-Cramer, J., Müller, H., Bedrick, S., Eggel, I., García Seco de Herrera, A., Tsikrika, T.: The CLEF 2011 medical image retrieval and classification tasks. In: Working Notes of CLEF 2011 (Cross Language Evaluation Forum) (2011)Müller, H., García Seco de Herrera, A., Kalpathy-Cramer, J., Demner Fushman, D., Antani, S., Eggel, I.: Overview of the ImageCLEF 2012 medical image retrieval and classification tasks. In: Working Notes of CLEF 2012 (Cross Language Evaluation Forum) (2012)García Seco de Herrera, A., Kalpathy-Cramer, J., Demner Fushman, D., Antani, S., Müller, H.: Overview of the ImageCLEF 2013 medical tasks. In: Working Notes of CLEF 2013 (Cross Language Evaluation Forum) (2013

Developing sustainable business models for institutions’ provision of open educational resources: Learning from OpenLearn users’ motivations and experiences

Universities across the globe have, for some time, been exploring the possibilities for achieving public benefit and generating business and visibility through releasing and sharing open educational resources (OER). Many have written about the need to develop sustainable and profitable business models around the production and release of OER. Downes (2006), for example, has questioned the financial sustainability of OER production at scale. Many of the proposed business models focus on OER’s value in generating revenue and detractors of OER have questioned whether they are in competition with formal education. This paper reports on a study intended to broaden the conversation about OER business models to consider the motivations and experiences of OER users as the basis for making a better informed decision about whether OER and formal learning are competitive or complementary with each other. The study focused on OpenLearn - the Open University’s (OU) web-based platform for OER, which hosts hundreds of online courses and videos and is accessed by over 3,000,000 users a year. A large scale survey and follow-up interviews with OpenLearn users worldwide revealed that university provided OER can offer learners a bridge to formal education, allowing them to try out a subject before registering on a formal course and to build confidence in their abilities as learners. In addition, it was found that using OER during formal paid-for study can improve learners’ performance and self-reliance, leading to increased retention and satisfaction with the learning experience

Open educational resources for all? Comparing user motivations and characteristics across The Open University’s iTunes U channel and OpenLearn platform.

With the rise in access to mobile multimedia devices, educational institutions have exploited the iTunes U platform as an additional channel to provide free educational resources with the aim of profile-raising and breaking down barriers to education. For those prepared to invest in content preparation, it is possible to produce interactive, portable material that can be made available globally. Commentators have questioned both the financial implications for platform-specific content production, and the availability of devices for learners to access it (Osborne, 2012). The Open University (OU) makes its free educational resources available on iTunes U and via its web-based open educational resources (OER) platform, OpenLearn. The OU’s OER on iTunes U reached the 60 million download mark in 2013; its OpenLearn platform boasts 27 million unique visitors since 2006. This paper reports the results of a large-scale study of users of the OU’s iTunes U channel and OpenLearn platform. A survey of several thousand users revealed key differences in demographics between those accessing OER via the web and via iTunes U. In addition, the data allowed comparison between three groups: formal learners, informal learners and educators. The study raises questions about whether university-provided OER meet the needs of users and makes recommendations for how content can be modified to suit their needs. As the publishing of OER becomes core to business, we reflect on reasons why understanding users’ motivations and demographics is vital, allowing for needs-led resource provision and content that is adapted to best achieve learner satisfaction, and to deliver institutions’ social mission

CHORUS Deliverable 2.2: Second report - identification of multi-disciplinary key issues for gap analysis toward EU multimedia search engines roadmap

After addressing the state-of-the-art during the first year of Chorus and establishing the existing landscape in multimedia search engines, we have identified and analyzed gaps within European research effort during our second year. In this period we focused on three directions, notably technological issues, user-centred issues and use-cases and socio- economic and legal aspects. These were assessed by two central studies: firstly, a concerted vision of functional breakdown of generic multimedia search engine, and secondly, a representative use-cases descriptions with the related discussion on requirement for technological challenges. Both studies have been carried out in cooperation and consultation with the community at large through EC concertation meetings (multimedia search engines cluster), several meetings with our Think-Tank, presentations in international conferences, and surveys addressed to EU projects coordinators as well as National initiatives coordinators. Based on the obtained feedback we identified two types of gaps, namely core technological gaps that involve research challenges, and “enablers”, which are not necessarily technical research challenges, but have impact on innovation progress. New socio-economic trends are presented as well as emerging legal challenges

Designing MOOC:a shared view on didactical principles

The innovative impact of the paper can be highlighted by the following statements: 1. Applying the Group Concept Mapping, a non-traditional and power research methodology for objectively identifying the shared vision of a group of experts on MOOC didactical principles. 2. Defining MOOC didactical principles and their operationalisations in more concrete guidelines. 3. Formulating suggestions for combining xMOOC and cMOOC.Supported by European Commission, DG EAC, under the Erasmus+ Programm

Qualitative research in psychology

"The first 'Workshop Qualitative Research in Psychology' took place in Blaubeuren, Germany from October 20-22, 2000. The meeting was organized by the Center for Qualitative Psychology of the University of Tübingen, Germany. The purpose of the meeting was to begin a network of qualitative psychologists. Thirty-two participants got to know each other, presented and discussed their research, discussed potential further developments within the field of qualitative psychology, and inspired each other with plans for the future. There were psychologists from Germany, Spain, Latvia, Finland, and the United States, most of whom were working as researchers within university contexts. The workshop took place at the retreat house of the University of Tübingen in a small village called Blaubeuren. A comfortable place with beautiful landscape, welcoming staff at the house and delicious food created a friendly atmosphere for the meeting from the start. The meeting started with an evening opening session in which all participants briefly introduced themselves and their interests in qualitative psychology. In order to communicate with each other, all of the participants spoke English. The introductions helped people to seek each other out afterwards in more informal conversations during the following two days." (author's abstract). Contents: Group I: Examples of Applications of Qualitative Methods, Part I - Discussion (summarized by Leo Gürtler) (17-20); Irmentraud Ertel: Categorizing the Content of Everyday Family Communication: What Do Families Talk About in Everyday Life? (21-31); Michaela Gläser-Zikuda: Emotions and Learning Strategies at School – Opportunities of Qualitative Content Analysis (32-50); Leo Gürtler: The role of subjective theories on love (51-65); Inge M. Lutz: Deciding which Kinds of Data to Collect in an Evaluative Study and Selecting a Setting for Data Collection and Analysis (66-76); Thomas Irion: Dynamics of a qualitative research design. An interactive approach to interactive reception (78-89); Ilze Plaude and Josef Held: Cross-cultural youth research as an international and interdisciplinary cooperation project: "International Learning" (90-98). Group II: Examples of Applications of Qualitative Methods, Part II - Discussion (summarized by Mechthild Kiegelmann) (99-101); Silke-Birgitta Gahleitner: Ways of combining qualitative and quantitative procedures exemplified in a study on the gender-specifics of coping with sexual violence (102-116); Mechthild Kiegelmann: Qualitative Research With a Genuine Psychological Approach: The Method of Voice Analysis (117-134); Tamara Beauboeuf: Toward a method of ideological becoming (135-142); Carlos Kölbl: Methods which are accommodated to their research object: On the adequate investigation of historical consciousness at youth age (143-149); Stephan Marks: Research Project 'History and Memory' (Geschichte und Erinnerung) (150-154); Antonio Medina Rivilla, M. Concepción Domínguez Garrido, Ramón Pérez Pérez, Tiberio Feliz Murias: Research Organization and Word Analysis from Discussion Groups about In-Practice Training (155-173); Antonio Medina Rivilla, M. Concepción Domínguez Garrido, Ramón Pérez Pérez, Tiberio Feliz Murias: Coding, Inquiring, and Analysis of Data from Discussion Groups about In-Practice Training (174-201). Group III: Specific Methodological Questions - Discussion (summarized by Günter L. Huber) (201-205); Günter L. Huber: The Analysis of Qualitative Data as Process of Classification (206-216); Gerhard Kleining, Thomas Burkart: Group-Based Dialogic Introspection and its Use in Qualitative Media Research (217-239); Julia Nentwich: The Process of Understanding in Qualitative Social Research (241-245); Bernd Reinhoffer: Forming Categories in Qualitative Data Analysis. The Teaching Research Project "Teachers' Attitude and Practice concerning Elementary Science in Primary School" (246-261); Hannu Soini: The Contribution of Qualitative Approaches to Learning Research: A Critical Incident Technique as a Research Method for Studying Student Learning (262-273); Leo Gürtler, Josef Held, Günter L. Huber, Mechthild Kiegelmann: Contributions of Qualitative Approaches to Psychological Inquiry (274-282)