CLEF 2017 NewsREEL Overview: Offline and Online Evaluation of Stream-based News Recommender Systems

The CLEF NewsREEL challenge allows researchers to evaluate news recommendation algorithms both online (NewsREEL Live) and offline (News- REEL Replay). Compared with the previous year NewsREEL challenged participants with a higher volume of messages and new news portals. In the 2017 edition of the CLEF NewsREEL challenge a wide variety of new approaches have been implemented ranging from the use of existing machine learning frameworks, to ensemble methods to the use of deep neural networks. This paper gives an overview over the implemented approaches and discusses the evaluation results. In addition, the main results of Living Lab and the Replay task are explained

NewsImages : Addressing the Depiction Gap with an Online News Dataset for Text-Image Rematching

We present NewsImages, a dataset of online news items, and the related NewsImages rematching task. The goal of NewsImages is to provide researchers with a means of studying the depiction gap, which we define to be the difference between what an image literally depicts and the way in which it is connected to the text that it accompanies. Online news is a domain in which the image-text connection is known to be indirect: The news article does not describe what is literally depicted in the image. We validate NewsImages with experiments that show the dataset's and the task's use for studying occurring connections between image and text, as well as addressing the depiction gap, which include sparse data, diversity of content, and importance of background knowledge.Peer reviewe

A stream-based resource for multi-dimensional evaluation of recommender algorithms

Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today's researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests. We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by "replaying" it locally, and ORP makes it possible to take this evaluation "live" in a living lab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in the development of the dataset and ORP

Idomaar : a framework for multi-dimensional benchmarking of recommender algorithms

In real-world scenarios, recommenders face non-functional requirements of technical nature and must handle dynamic data in the form of sequential streams. Evaluation of recommender systems must take these issues into account in order to be maximally informative. In this paper, we present Idomaar—a framework that enables the efficient multi-dimensional benchmarking of recommender algorithms. Idomaar goes beyond current academic research practices by creating a realistic evaluation environment and computing both effectiveness and technical metrics for stream-based as well as set-based evaluation. A scenario focussing on “research to prototyping to productization” cycle at a company illustrates Idomaar’s potential. We show that Idomaar simplifies testing with varying configurations and supports flexible integration of different data

CLEF 2017 NewsREEL overview: A stream-based recommender task for evaluation and education

News recommender systems provide users with access to news stories that they find interesting and relevant. As other online, stream-based recommender systems, they face particular challenges, including limited information on users’ preferences and also rapidly fluctuating item collections. In addition, technical aspects, such as response time and scalability, must be considered. Both algorithmic and technical considerations shape working requirements for real-world recommender systems in businesses. NewsREEL represents a unique opportunity to evaluate recommendation algorithms and for students to experience realistic conditions and to enlarge their skill sets. The NewsREEL Challenge requires participants to conduct data-driven experiments in NewsREEL Replay as well as deploy their best models into NewsREEL Live’s ‘living lab’. This paper presents NewsREEL 2017 and also provides insights into the effectiveness of NewsREEL to support the goals of instructors teaching recommender systems to students. We discuss the experiences of NewsREEL participants as well as those of instructors teaching recommender systems to students, and in this way, we showcase NewsREEL’s ability to support the education of future data scientists

Continuous evaluation of large-scale information access systems : a case for living labs

A/B testing is currently being increasingly adopted for the evaluation of commercial information access systems with a large user base since it provides the advantage of observing the efficiency and effectiveness of information access systems under real conditions. Unfortunately, unless university-based researchers closely collaborate with industry or develop their own infrastructure or user base, they cannot validate their ideas in live settings with real users. Without online testing opportunities open to the research communities, academic researchers are unable to employ online evaluation on a larger scale. This means that they do not get feedback for their ideas and cannot advance their research further. Businesses, on the other hand, miss the opportunity to have higher customer satisfaction due to improved systems. In addition, users miss the chance to benefit from an improved information access system. In this chapter, we introduce two evaluation initiatives at CLEF, NewsREEL and Living Labs for IR (LL4IR), that aim to address this growing “evaluation gap” between academia and industry. We explain the challenges and discuss the experiences organizing these living labs

Optimization of atmospheric plasma treatment of LDPE films: Influence on adhesive properties and ageing behavior

One of the major disadvantages of low density polyethylene (LDPE) films is their poor adhesive properties. Therefore, LDPE films have been treated with atmospheric pressure air plasma in order to improve their surface properties. So as to simulate the possible conditions in an industrial process, the samples have been treated with two different sample distances (6 and 10 mm), and treatment rates between 100 and 1000 mm s-1. The different sample distances are the distance of the sample from the plasma source. The variation of the surface properties and adhesion characteristics of the films were investigated for different aging times after plasma exposure (up to 21 days) using contact angle measurement, atomic force microscopy, weight loss measurements and shear test. Results show that the treatment increases the polar component () and these changes improve adhesive properties of the material. After the twenty-first day, the ageing process causes a decrease of wettability and adhesive properties of the LDPE films (up to 60%).Fombuena Borrás, V.; García Sanoguera, D.; Sánchez Nacher, L.; Balart Gimeno, RA.; Boronat Vitoria, T. (2014). Optimization of atmospheric plasma treatment of LDPE films: Influence on adhesive properties and ageing behavior. Journal of Adhesion Science and Technology. 28(1):97-113. doi:10.1080/01694243.2013.847045S97113281Achilias, D. S., Roupakias, C., Megalokonomos, P., Lappas, A. A., & Antonakou, Ε. V. (2007). Chemical recycling of plastic wastes made from polyethylene (LDPE and HDPE) and polypropylene (PP). Journal of Hazardous Materials, 149(3), 536-542. doi:10.1016/j.jhazmat.2007.06.076Friedman, M., & Walsh, G. (2002). High performance films: Review of new materials and trends. Polymer Engineering & Science, 42(8), 1756-1788. doi:10.1002/pen.11069Wiles, D. M., & Scott, G. (2006). Polyolefins with controlled environmental degradability. Polymer Degradation and Stability, 91(7), 1581-1592. doi:10.1016/j.polymdegradstab.2005.09.010Gao, J., Lei, J., Li, Q., & Ye, S. (2004). Functionalized low-density polyethylene via a novel photografting method and its adhesion properties. Journal of Adhesion Science and Technology, 18(2), 195-203. doi:10.1163/156856104772759403Shenton, M. J., Lovell-Hoare, M. C., & Stevens, G. C. (2001). Adhesion enhancement of polymer surfaces by atmospheric plasma treatment. Journal of Physics D: Applied Physics, 34(18), 2754-2760. doi:10.1088/0022-3727/34/18/307Belgacem, M. N., Salon-Brochier, M. C., Krouit, M., & Bras, J. (2011). Recent Advances in Surface Chemical Modification of Cellulose Fibres. Journal of Adhesion Science and Technology, 25(6-7), 661-684. doi:10.1163/016942410x525867Friedrich, J., Unger, W., & Lippitz, A. (1995). Plasma modification of polymer surfaces. Macromolecular Symposia, 100(1), 111-115. doi:10.1002/masy.19951000118Ladizesky, N. H., & Ward, I. M. (1989). The adhesion behaviour of high modulus polyethylene fibres following plasma and chemical treatment. Journal of Materials Science, 24(10), 3763-3773. doi:10.1007/bf02385768Nardin, M., & Ward, I. M. (1987). Influence of surface treatment on adhesion of polyethylene fibres. Materials Science and Technology, 3(10), 814-826. doi:10.1179/mst.1987.3.10.814Villagra Di Carlo, B., Gottifredi, J. C., & Habert, A. C. (2010). Synthesis and characterization of composite membrane by deposition of acrylic acid plasma polymer onto pre-treated polyethersulfone support. Journal of Materials Science, 46(6), 1850-1856. doi:10.1007/s10853-010-5012-4Matsunaga, M., & Whitney, P. J. (2000). Surface changes brought about by corona discharge treatment of polyethylene film and the effect on subsequent microbial colonisation. Polymer Degradation and Stability, 70(3), 325-332. doi:10.1016/s0141-3910(00)00105-1Novák, I., Pollák, V., & Chodák, I. (2006). Study of Surface Properties of Polyolefins Modified by Corona Discharge Plasma. Plasma Processes and Polymers, 3(4-5), 355-364. doi:10.1002/ppap.200500163Arpagaus, C., Rossi, A., & Rudolf von Rohr, P. (2005). Short-time plasma surface modification of HDPE powder in a Plasma Downer Reactor – process, wettability improvement and ageing effects. Applied Surface Science, 252(5), 1581-1595. doi:10.1016/j.apsusc.2005.02.099Morra, M., Occhiello, E., Marola, R., Garbassi, F., Humphrey, P., & Johnson, D. (1990). On the aging of oxygen plasma-treated polydimethylsiloxane surfaces. Journal of Colloid and Interface Science, 137(1), 11-24. doi:10.1016/0021-9797(90)90038-pKim, K. S., Ryu, C. M., Park, C. S., Sur, G. S., & Park, C. E. (2003). Investigation of crystallinity effects on the surface of oxygen plasma treated low density polyethylene using X-ray photoelectron spectroscopy. Polymer, 44(20), 6287-6295. doi:10.1016/s0032-3861(03)00674-8Kim, S. H., Ha, H. J., Ko, Y. K., Yoon, S. J., Rhee, J. M., Kim, M. S., … Khang, G. (2007). Correlation of proliferation, morphology and biological responses of fibroblasts on LDPE with different surface wettability. Journal of Biomaterials Science, Polymer Edition, 18(5), 609-622. doi:10.1163/156856207780852514Borcia, G., Anderson, C. A., & Brown, N. M. D. (2004). The surface oxidation of selected polymers using an atmospheric pressure air dielectric barrier discharge. Part I. Applied Surface Science, 221(1-4), 203-214. doi:10.1016/s0169-4332(03)00879-1Pascual, M., Calvo, O., Sanchez-Nácher, L., Bonet, M. A., Garcia-Sanoguera, D., & Balart, R. (2009). Optimization of adhesive joints of low density polyethylene (LDPE) composite laminates with polyolefin foam using corona discharge plasma. Journal of Applied Polymer Science, 114(5), 2971-2977. doi:10.1002/app.30906Encinas, N., Díaz-Benito, B., Abenojar, J., & Martínez, M. A. (2010). Extreme durability of wettability changes on polyolefin surfaces by atmospheric pressure plasma torch. Surface and Coatings Technology, 205(2), 396-402. doi:10.1016/j.surfcoat.2010.06.069Takke, V., Behary, N., Perwuelz, A., & Campagne, C. (2009). Studies on the atmospheric air-plasma treatment of PET (polyethylene terephtalate) woven fabrics: Effect of process parameters and of aging. Journal of Applied Polymer Science, 114(1), 348-357. doi:10.1002/app.30618Awaja, F., Gilbert, M., Kelly, G., Fox, B., & Pigram, P. J. (2009). Adhesion of polymers. Progress in Polymer Science, 34(9), 948-968. doi:10.1016/j.progpolymsci.2009.04.007Garcia, D., Sanchez, L., Fenollar, O., Lopez, R., & Balart, R. (2008). Modification of polypropylene surface by CH4–O2 low-pressure plasma to improve wettability. Journal of Materials Science, 43(10), 3466-3473. doi:10.1007/s10853-007-2322-2Guimond, S., & Wertheimer, M. R. (2004). Surface degradation and hydrophobic recovery of polyolefins treated by air corona and nitrogen atmospheric pressure glow discharge. Journal of Applied Polymer Science, 94(3), 1291-1303. doi:10.1002/app.21134Pascual, M., Balart, R., Sánchez, L., Fenollar, O., & Calvo, O. (2008). Study of the aging process of corona discharge plasma effects on low density polyethylene film surface. Journal of Materials Science, 43(14), 4901-4909. doi:10.1007/s10853-008-2712-0Sanchis, R., Fenollar, O., García, D., Sánchez, L., & Balart, R. (2008). Improved adhesion of LDPE films to polyolefin foams for automotive industry using low-pressure plasma. International Journal of Adhesion and Adhesives, 28(8), 445-451. doi:10.1016/j.ijadhadh.2008.04.002Fresnais, J., Chapel, J. P., Benyahia, L., & Poncin-Epaillard, F. (2009). Plasma-Treated Superhydrophobic Polyethylene Surfaces: Fabrication, Wetting and Dewetting Properties. Journal of Adhesion Science and Technology, 23(3), 447-467. doi:10.1163/156856108x370127Abenojar, J., Colera, I., Martínez, M. A., & Velasco, F. (2010). Study by XPS of an Atmospheric Plasma-Torch Treated Glass: Influence on Adhesion. Journal of Adhesion Science and Technology, 24(11-12), 1841-1854. doi:10.1163/016942410x507614Lommatzsch, U., Pasedag, D., Baalmann, A., Ellinghorst, G., & Wagner, H.-E. (2007). Atmospheric Pressure Plasma Jet Treatment of Polyethylene Surfaces for Adhesion Improvement. Plasma Processes and Polymers, 4(S1), S1041-S1045. doi:10.1002/ppap.200732402Balu, B., Berry, A. D., Patel, K. T., Breedveld, V., & Hess, D. W. (2011). Directional Mobility and Adhesion of Water Drops on Patterned Superhydrophobic Surfaces. Journal of Adhesion Science and Technology, 25(6-7), 627-642. doi:10.1163/016942410x525849Bhattacharya, S., Singh, R. K., Mandal, S., Ghosh, A., Bok, S., Korampally, V., … Gangopadhyay, S. (2010). Plasma Modification of Polymer Surfaces and Their Utility in Building Biomedical Microdevices. Journal of Adhesion Science and Technology, 24(15-16), 2707-2739. doi:10.1163/016942410x511105Das, S., Neogi, S., Chainy, G. B. N., & Guha, S. K. (2011). A Novel Two-Step Procedure for Plasma Surface Modification of Low-Density Polyethylene for Improved Drug Adhesion in Intra Uterine Devices (IUDs). Journal of Adhesion Science and Technology, 25(1-3), 151-167. doi:10.1163/016942410x503285Schulz, U., Munzert, P., & Kaiser, N. (2010). Plasma Surface Modification of PMMA for Optical Applications. Journal of Adhesion Science and Technology, 24(7), 1283-1289. doi:10.1163/016942409x12561252292026Silverstein, M. S., Breuer, O., & Dodiuk, H. (1994). Surface modification of UHMWPE fibers. Journal of Applied Polymer Science, 52(12), 1785-1795. doi:10.1002/app.1994.070521213Inagaki, N., Narushim, K., Tuchida, N., & Miyazaki, K. (2004). Surface characterization of plasma-modified poly(ethylene terephthalate) film surfaces. Journal of Polymer Science Part B: Polymer Physics, 42(20), 3727-3740. doi:10.1002/polb.20234Nakamatsu, J., Delgado-Aparicio, L. F., Da Silva, R., & Soberon, F. (1999). Ageing of plasma-treated poly(tetrafluoroethylene) surfaces. Journal of Adhesion Science and Technology, 13(7), 753-761. doi:10.1163/156856199x00983Yun, Y. I., Kim, K. S., Uhm, S.-J., Khatua, B. B., Cho, K., Kim, J. K., & Park, C. E. (2004). Aging behavior of oxygen plasma-treated polypropylene with different crystallinities. Journal of Adhesion Science and Technology, 18(11), 1279-1291. doi:10.1163/1568561041588200Morent, R., De Geyter, N., Leys, C., Gengembre, L., & Payen, E. (2007). Study of the ageing behaviour of polymer films treated with a dielectric barrier discharge in air, helium and argon at medium pressure. Surface and Coatings Technology, 201(18), 7847-7854. doi:10.1016/j.surfcoat.2007.03.018Zhao, B., & Kwon, H. J. (2011). Adhesion of Polymers in Paper Products from the Macroscopic to Molecular Level — An Overview. Journal of Adhesion Science and Technology, 25(6-7), 557-579. doi:10.1163/016942410x52582