Huurdeman, H.

Kamps, J.

Koolen, M.

English

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Effective metadata for social book search from a user perspectiveHuurdeman, H.; Kamps, J.; Koolen, M.Publication date2014Document VersionFinal published versionPublished inCLEF 2014 : CLEF2014 Working NotesLink to publicationCitation for published version (APA):Huurdeman, H., Kamps, J., & Koolen, M. (2014). Effective metadata for social book searchfrom a user perspective. In L. Cappellato, N. Ferro, M. Halvey, & W. Kraaij (Eds.), CLEF 2014: CLEF2014 Working Notes: Working Notes for CLEF 2014 Conference : Sheffield, UK,September 15-18, 2014 (pp. 543-548). (CEUR Workshop Proceedings; Vol. 1180). CEUR-WS. http://ceur-ws.org/Vol-1180/CLEF2014wn-Inex-HuudermanEt2014.pdfGeneral rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:10 Mar 2023Effective Metadata for Social Book Search froma User PerspectiveHugo Huurdeman1,2, Jaap Kamps1,2,3, and Marijn Koolen11 Institute for Logic, Language and Computation, University of Amsterdam2 Archives and Information Studies, Faculty of Humanities, University of Amsterdam3 ISLA, Faculty of Science, University of AmsterdamAbstract. In this extended abstract we describe our participation in theINEX 2014 Interactive Social Book Search Track. In previous work, wehave looked at the impact of professional and user-generated metadatain the context of book search, and compared these different categoriesof metadata in terms of retrieval effectiveness. Here, we take a differentapproach and study the use of professional and user-generated metadataof books in an interactive setting, and the effectivity of this metadatafrom a user perspective.We compare the perceived usefulness of general descriptions, publicationmetadata, user reviews and tags in focused and open-ended search tasks,based on data gathered in the INEX Interactive Social Book SearchTrack. Furthermore, we take a tentative look at the actual use of differenttypes of metadata over time in the aggregated search tasks.Our preliminary findings in the surveyed tasks indicate that user reviewsare generally perceived to be more useful than other types of metadata,and they are frequently mentioned in users’ rationales for selecting books.Furthermore, we observe a varying usage frequency of traditional anduser-generated metadata across time in the aggregated search tasks, pro-viding initial indications that these types of metadata might be usefulat different stages of a search task.1 IntroductionIn 2014, the Interactive Social Book Search (iSBS) task has been introduced.The goal of this task is “to investigate how book searchers deal with professionaland user-generated content at different stages of the of the search process” 1.The iSBS track uses the Amazon/LibraryThing collection with book descrip-tions for 1.5 million books, as crawled by the University of Duisburg-Essen inearly 2009. This dataset contains both professional metadata and user-generateddescriptions from Amazon and LibraryThing.The task has introduced two interfaces for book search: a baseline interface,including search elements, results lists, and item details, and a ‘multistage’ in-terface, in which different features for different stages of a search have beenintroduced. Both interfaces feature a “book-bag”, to which users can add books1 https://inex.mmci.uni-saarland.de/tracks/books/#interact543when completing the assigned tasks2. Two kinds of tasks are employed: a goal-oriented task, and a non-goal oriented task.The goal-oriented task was the following: Imagine you are looking for someinteresting physics and mathematics books for a layperson. You have heard aboutthe Feynman books but you have never really read anything in this area. Youwould also like to find an “interesting facts” sort of book on mathematics.The non-goal oriented task, without a predefined information need, was thefollowing: Imagine you are waiting to meet a friend in a coffee shop or pub or theairport or your office. While waiting, you come across this website and exploreit looking for any book that you find interesting, or engaging or relevant...The participants in the iSBS experiment carried out one goal, and one non-goal oriented task, and were randomly assigned the baseline or multistage inter-face. A total of 41 users participated in the experiment, at Aalborg UniversityCopenhagen, Edge Hill University, Humboldt University, and the University ofAmsterdam. Of these participants, 19 used the baseline interface, and 22 usersutilized the multistage interface. The resulting data, in the form of usage logsand questionnaires gathered in the experiment, was shared among the partipat-ing teams.In this extended abstract, we take an initial look at the results, and specif-ically focus on the usefulness and effectivity of professional and user-generatedmetadata for books in an interactive setting, and compare the perceived useful-ness of general descriptions, publication metadata, user reviews and tags acrosstasks.2 Previous workPrevious work related to the iSBS track has been carried out in the INEX In-teractive Retrieval Experiments (2004-2010) [7] and the Cultural Heritage inCLEF (CHiC) Interactive Track 2013 [8]. In these tracks, a standard procedurefor collecting data was being used by participating research groups, includingcommon topics and tasks, standardized search systems, document corpora andprocedures. The system used for the iSBS track is a modified version of the oneused for CHiC and is based on the Interactive IR evaluation platform devel-oped by Hall and Toms [1], where different search engines and interfaces can beplugged into fully developed IIR framework that runs the entire user study [2].The Interactive SBS Track complements the system-oriented Social BookSearch Track [6]. Both tracks use the Amazon/LibraryThing collection and in-vestigate the value of professional metadata and user-generated content. Thesystem-oriented evaluation of the SBS Track has shown that retrieval effective-ness increases when systems include user-generated content for a broad range oftasks [4, 5]. These findings prompted the desire to study how book searchers useprofessional metadata and user-generated content.2 More information about the experiment and different interfaces is available in [3]5443 ResultsThe results described in this extended abstract are preliminary, and based on arelatively small dataset. Nevertheless, they can provide basic insights into booksearch from a user perspective.We focus on four different types of displayed metadata in the baseline andmultistage interface:– general book descriptions which contain publisher-provided product descrip-tions and editorial reviews (available for 78% of all books).– publication metadata, including information on publisher, price, number ofpages and binding (available for all books)– reviews metadata, containing up to 10 Amazon user reviews and includeratings (available for 45% of all books)– user-provided tags from LibraryThing (LT), the displayed size based on howmany LT users assigned each tag to the book (available for 83% of all books).usefulness First of all, we look at the perceived usefulness of the general descrip-tions, publication metadata, user reviews and user tags.Table 1: average perceived usefulness of metadata featuresdescription publication reviews tagsbaseline 4.0 3.0 3.8 2.8multistage 4.3 2.6 3.9 2.9total 4.2 2.8 3.9 2.7In the post-questionnaire following the focused and open task of the exper-iment, users had to indicate how useful the metadata items about a book wereon a rating scale of 1 (not at all useful) to 5 (extremely useful). Table 1 con-tains the results, differentiated by interface. It shows that for both interfaces,the general descriptions were deemed most useful, with an average rating of 4.2out of 5. This is followed by the reviews, that were almost as useful, with anaverage rating of 3.9. Finally, both formal publication metadata, and tags werenot considered highly useful by the participants.Table 2: percentage of users not using metadata elementsdescription publication reviews tagsbaseline 10.5% 39.5% 29.0% 44.7%multistage 4.6% 34.1% 20.5% 18.2%total 7.5% 36.8% 24.7% 31.5%Secondly, the usage of each feature is of importance, since in the same surveyquestion participants could also indicate that they did not use an element. Table2 indicates that only a small minority (7.5%) of participants did not use the545general book descriptions. Again, it is followed by the reviews, that are not usedby 25% of all participants. Finally, the tags and publication metadata are unusedby more than 30% of all users.So despite the fact that user reviews and ratings are only available for 45%of all books, they are actually considered almost as useful as general descriptivemetadata by the participants in the experiment, and used more often than pub-lication metadata and user tags. To contextualize this finding, we now look atthe rationales of users for adding books to their bookbag selections.contextualizing perceived usefulness In the post-task questionnaire, users couldreview their bookbag immediately after finishing their task, and were askedto describe why they selected these books. Users describe various reasons foradding books to their bookbags. A commonly mentioned selection criterion isrelated to the book’s title, subject and description. One user, for example, pointsout that “the first book, due to the title, seemed like a simpler read then theFeynman volumes”. Other users, surprisingly, mention that they chose a resultbecause of it’s high position in the results list (being “closer to the top of theresults”). Another reason for choosing particular books, in line with the resultsdiscussed in the previous section, is based on their ratings and reviews. Forthe open task, 7 out of 41 participants mention ratings and reviews as beingimportant in their final book selections, while for the focused task another 7participants participants explicitly mention ratings and reviews. For example,one user reports that based on the number of reviews, (s)he decided that abook is engaging: “I tried to look at that in the comments, if books had nocomments I didn’t select them”. Reviews are also frequently used to chooseamong alternatives: “the first book has also a very good review”. Hence, the user-generated ratings and reviews seem to play an important role in the selectionprocess in the context of the employed tasks.usage statistics of metadata elements over time Based on the available usagelogs, we can take a tentative look at the usage of the metadata available in themultistage experimental user interface. Here, we do not yet have statistics onthe number of times the general book descriptions were seen, since they appearby default, but we can get insights on the usage of the publication, review andtag metadata.Table 3: use of metadata features in the multistage interface (n=22)begin middle end totalpublication 5 17 10 32reviews 8 20 24 52tags 6 14 20 40Table 3 shows the number of times publication metadata, reviews and tagswere explicitly selected in the focused and open-ended search task. While thesenumbers, upon a total number of 22 users across two tasks are quite low, they546do indicate a higher utilization of user-generated reviews in the combined tasks,used in total 52 times by 22 users.Furthermore, the use of these metadata features over time can be surveyed.Therefore we divided the total task time into three segments, begin, middle andend, each taking up one third of the task process. From this data, we see thatthe number of times these data elements are used varies over time: the publi-cation metadata is used more in the middle, and the reviews and tags are usedmore frequently in the end. Due to the low number of data points, we cannotderive strong conclusions, but the use of metadata features over time could beinteresting to assess in future studies.4 ConclusionThe preliminary results documented in this extended abstract provide indica-tions that besides professional metadata, also user-generated metadata mightbe useful for book search, from the perspective of users. In terms of perceivedusefulness, and usage, especially user reviews seem to aid users in determiningtheir book selections for different types of tasks, even though they are not avail-able for all books. While the findings here are based on the first iteration of theInteractive Social Book Search task, they provide handles for future research,and we intend to extend these findings in future work in the context of SocialBook Search.Acknowledgments This research was supported by the Netherlands Organiza-tion for Scientific Research (NWO projects # 612.066.513, 639.072.601, and640.005.001) and by the European Community’s Seventh Framework Program(FP7 2007/2013, Grant Agreement 270404).Bibliography1. M. Hall and E. Toms. Building a common framework for iir evaluation. InP. Forner, H. Müller, R. Paredes, P. Rosso, and B. Stein, editors, InformationAccess Evaluation. Multilinguality, Multimodality, and Visualization, volume8138 of Lecture Notes in Computer Science, pages 17–28. Springer BerlinHeidelberg, 2013. ISBN 978-3-642-40801-4. doi: 10.1007/978-3-642-40802-13.2. M. Hall, S. Katsaris, and E. Toms. A pluggable interactive ir evaluationwork-bench. In European Workshop on Human-Computer Interaction andInformation Retrieval, pages 35–38, 2013.3. M. Hall, H. Huurdeman, M. Koolen, M. Skov, and D. Walsh. Overview of theINEX 2014 interactive social book search track. In L. Cappellato, N. Ferro,M. Halvey, and W. Kraaij, editors, CLEF 2014 Labs and Workshops, NotebookPapers, CEUR Workshop Proceedings (CEUR-WS.org), 2014.5474. M. Koolen. ”user reviews in the search index? that’ll never work!”. InM. de Rijke, T. Kenter, A. P. de Vries, C. Zhai, F. de Jong, K. Radinsky,and K. Hofmann, editors, ECIR, volume 8416 of Lecture Notes in ComputerScience, pages 323–334. Springer, 2014. ISBN 978-3-319-06027-9.5. M. Koolen, J. Kamps, and G. Kazai. Social Book Search: The Impact ofProfessional and User-Generated Content on Book Suggestions. In CIKM2012. ACM, 2012.6. M. Koolen, G. Kazai, M. Preminger, and A. Doucet. Overview of the INEX2013 social book search track. In CLEF 2013 Evaluation Labs and Workshop,Online Working Notes, 2013.7. R. Nordlie and N. Pharo. Seven years of inex interactive retrieval experiments- lessons and challenges. In T. Catarci, P. Forner, D. Hiemstra, A. Peñas,and G. Santucci, editors, CLEF, volume 7488 of Lecture Notes in ComputerScience, pages 13–23. Springer, 2012. ISBN 978-3-642-33246-3.8. V. Petras, T. Bogers, E. Toms, M. Hall, J. Savoy, P. Malak, A. Pawowski,N. Ferro, and I. Masiero. Cultural heritage in clef (chic) 2013. In P. Forner,H. Müller, R. Paredes, P. Rosso, and B. Stein, editors, Information AccessEvaluation. Multilinguality, Multimodality, and Visualization, volume 8138 ofLecture Notes in Computer Science, pages 192–211. Springer Berlin Heidel-berg, 2013. ISBN 978-3-642-40801-4. doi: 10.1007/978-3-642-40802-1 23.548

Effective metadata for social book search from a user perspective

https://pure.uva.nl/ws/files/55510997/CLEF2014wn_Inex_HuudermanEt2014.pdf

Effective metadata for social book search from a user perspective

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications