International audienceIn this demonstration, we present Exquisitor, a media explorer capable of learning user preferences in real-time during interactions with the 99.2 million images of YFCC100M. Exquisitor owes its efficiency to innovations in data representation, compression, and indexing. Exquisitor can complete each interaction round, including learning preferences and presenting the most relevant results, in less than 30 ms using only a single CPU core and modest RAM. In short, Exquisitor can bring large-scale interactive learning to standard desktops and laptops, and even high-end mobile devices

Amsaleg, Laurent

Guðmundsson, Gylfi Þór

Jónsson, Björn Thór

Khan, Omar Shahbaz

Ragnarsdóttir, Hanna

Rudinac, Stevan

Worring, Marcel

Zahálka, Jan

Þorleiksdóttir, Þórhildur

English

In this demonstration, we present Exquisitor, a media explorer capable of learning user preferences in real-time during interactions with the 99.2 million images of YFCC100M. Exquisitor owes its efficiency to innovations in data representation, compression, and indexing. Exquisitor can complete each interaction round, including learning preferences and presenting the most relevant results, in less than 30 ms using only a single CPU core and modest RAM. In short, Exquisitor can bring large-scale interactive learning to standard desktops and laptops, and even high-end mobile devices

Ragnarsdóttir, H.

Þorleiksdottir, Þ.

Khan, O.S.

Jónsson, B.Þ.

Guðmundsson, G.Þ.

Zahálka, J.

Rudinac, S.

Amsaleg, L.

Worring, M.

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Exquisitor: Breaking the Interaction Barrier for Exploration of 100 Million ImagesRagnarsdóttir, H.; Þorleiksdottir, Þ.; Khan, O.S.; Jónsson, B.Þ.; Guðmundsson, G.Þ.;Zahálka, J.; Rudinac, S.; Amsaleg, L.; Worring, M.DOI10.1145/3343031.3350580Publication date2019Document VersionFinal published versionPublished inMM'19LicenseArticle 25fa Dutch Copyright ActLink to publicationCitation for published version (APA):Ragnarsdóttir, H., Þorleiksdottir, Þ., Khan, O. S., Jónsson, B. Þ., Guðmundsson, G. Þ.,Zahálka, J., Rudinac, S., Amsaleg, L., & Worring, M. (2019). Exquisitor: Breaking theInteraction Barrier for Exploration of 100 Million Images. In MM'19: proceedings of the 27thACM Conference on Multimedia : October 21-25, 2019, Nice, France (pp. 1029-1031).Association for Computing Machinery. https://doi.org/10.1145/3343031.3350580General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:09 Mar 2023Exquisitor: Breaking the Interaction Barrierfor Exploration of 100 Million ImagesHanna RagnarsdóttirReykjavik UniversityReykjavik, Icelandhannar15@ru.isÞórhildur ÞorleiksdóttirReykjavik UniversityReykjavik, Icelandthorhildurt15@ru.isOmar Shahbaz KhanIT University of CopenhagenCopenhagen, Denmarkomsh@itu.dkBjörn Þór JónssonIT University of CopenhagenCopenhagen, Denmarkbjorn@itu.dkGylfi Þór GuðmundssonReykjavik UniversityReykjavik, Icelandgylfig@ru.isJan Zahálkabohem.aiPrague, Czech Republicjan.zahalka@bohem.aiStevan RudinacUniversity of AmsterdamAmsterdam, Netherlandss.rudinac@uva.nlLaurent AmsalegCNRS–IRISARennes, Francelaurent.amsaleg@irisa.frMarcel WorringUniversity of AmsterdamAmsterdam, Netherlandsm.worring@uva.nlABSTRACTIn this demonstration, we present Exquisitor, a media explorercapable of learning user preferences in real-time during interactionswith the 99.2 million images of YFCC100M. Exquisitor owes itsefficiency to innovations in data representation, compression, andindexing. Exquisitor can complete each interaction round, includinglearning preferences and presenting the most relevant results, inless than 30 ms using only a single CPU core and modest RAM.In short, Exquisitor can bring large-scale interactive learning tostandard desktops and laptops, and even high-end mobile devices.CCS CONCEPTS• Information systems → Multimedia and multimodal re-trieval; Multimedia databases;KEYWORDSInteractive multimodal learning; Scalability; 100 million images.ACM Reference Format:Hanna Ragnarsdóttir, Þórhildur Þorleiksdóttir, Omar Shahbaz Khan, BjörnÞór Jónsson, Gylfi Þór Guðmundsson, Jan Zahálka, Stevan Rudinac, LaurentAmsaleg, and Marcel Worring. 2019. Exquisitor: Breaking the InteractionBarrier for Exploration of 100 Million Images. In Proceedings of the 27thACM International Conference on Multimedia (MM ’19), October 21–25, 2019,Nice, France. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3343031.33505801 INTRODUCTIONMultimedia collections have become a cornerstone data resource ina variety of scientific and industrial fields. One of the most difficultPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).MM ’19, October 21–25, 2019, Nice, France© 2019 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-6889-6/19/10.https://doi.org/10.1145/3343031.3350580Figure 1: The Exquisitor demonstration interface. Exquisi-tor interactively learns new analytic categories over the fullYFCC100M image collection, with average latency of lessthan 30 ms per interaction round, using hardware compa-rable with standard desktops and modern mobile devices.challenges for interactive exploration of such collections—not onlyfor data scientists working with these collections, but also for themultimedia community as a whole—is their scale. How can wefacilitate efficient access to multimedia collections comprising tensor hundreds of millions of images, let alone billions?Interactive learning, which was embraced by the multimediacommunity in the early days of content-based image and videoretrieval [4, 12], has recently experienced revival as an umbrellamethod capable of satisfying a variety of multimedia informationneeds, ranging from exploratory browsing to seeking a particularknown item [17]. Through feedback from the user, interactive learn-ing can adapt to the intent and knowledge of the user, and thuscollaboratewith the user towards, e.g., learning new or unknown an-alytic categories on the fly. Previous contributions, however, largelyoperated at a relatively small scale [1, 2, 8, 10, 13, 14].Demonstration I  MM ’19, October 21–25, 2019, Nice, France1029Consider, as an example, the YFCC100M collection [16], com-prising 99.2M images and 0.8M videos. It has existed for some timenow, but very few have a good idea about what its actual contentsare. This is no surprise, as it is difficult to tackle this collection withexisting techniques: the simple metadata-based filtering approachis impeded by the sparse and noisy nature of the metadata and, atthis scale, similarity search is akin to shooting in the dark. Semanticconcept detectors can be used to generate additional content-basedmetadata for both approaches, but that does not alleviate theirproblems. And thus, the YFCC100M collection remains a mystery.We have recently developed Exquisitor, a highly scalable inter-active multimodal learning approach [6]. A key feature that setsExquisitor apart from related approaches is its scalability: Exquisitorcan retrieve suggestions from 100 million images with sub-secondlatency, using extremely modest computing resources, thus break-ing the interaction barrier for large-scale interactive learning. Inthis demonstration, we propose to allowACMMultimedia attendeesto interactively explore the YFCC100M collection with Exquisitor.2 EXQUISITOR INTERFACEThe current Exquisitor user interface, shown in Figure 1, is browser-based and implemented using the React JavaScript library. It is afairly traditional interactive learning interface, in that users areasked to label positive and negative examples, which are then usedto learn their preferences and determine the new round of sug-gestions. Due to the extreme efficiency of the interactive learningprocess, however, there are some notable differences from tradi-tional interactive learning interfaces:• The learning process runs unobtrusively in the background,continuously providing new on-demand relevant examplesas the user progresses with her exploration, instead of re-quiring explicit management.• Individual images are replaced, rather than the entire screen,for a smooth transition from one interaction round to theother. Users are allowed to indicate that images are neitherpositive nor negative to get a new suggestion, and imagesthat have been visible for some time, but not tagged as posi-tive or negative, can also be replaced with new suggestions.• Users can revisit positive and negative examples, removingor even reversing the feedback label, as their understandingof the collection contents and its relevance evolves.Overall, the user interface is intended to provide a smooth learningexperience. We have already used Exquisitor in the Lifelog SearchChallenge (LSC) 2019 [7], and a detailed evaluation of the userexperience is part of our future work.3 THE LEARNING PROCESSExquisitor’s back-end produces relevant results to show to the userin less than 30 ms per interaction round, including learning theuser preference and scoring the collection, using one 2.4 GHz CPUcore and less than 6 GB of RAM. The back-end system is composedof two web services, as shown in Figure 2. The ImageAPI serviceserves thumbnails for the YFCC100M image collection, as requiredby the user interface. The LearningAPI servicewraps the underlyingmultimodal learning engine, decribed in [6]. In the remainder ofthis section, we briefly outline the multimodal learning process.Figure 2: Exquisitor demonstration system overview.To enable interactive multimodal learning, visual and text fea-tures were extracted from the images of the YFCC100M collection.For visual features, the 1000 ImageNet semantic concepts wereextracted using a GoogLeNet architecture [15]. For the text modal-ity, 100 LDA topics were extracted from the image title, tags anddescription using the gensim framework [11].Uncompressed features for the 99.2M images require nearly880GB of memory. Exquisitor uses the recently proposed Ratio-64 representation, which preserves only the top visual conceptsand text topics for each image [18]. The compressed feature collec-tions require less than 6GB of storage, thus fitting into the memoryof a standard consumer PC, as well as some high-endmobile devices.This effective compression method has been shown to preserve thesemantic descriptiveness of the visual and text features [18].The interactive learning process is facilitated using a linear SVMmodel, proven to provide a good balance between efficiency andaccuracy when classifying large datasets based on few trainingexamples [5, 9]. Based on the relevance indication provided bythe user, a classifier is trained separately for the text and visualmodalities and the images furthest from the hyperplane are selected.The final list of results is created using rank aggregation.The compressed feature data is indexed using a variant of theextended Cluster Pruning (eCP) high-dimensional indexing algo-rithm [3]. By directing Exquisitor’s attention to the clusters withrepresentatives most relevant to the learned linear SVM model,the work of scoring candidates is reduced by nearly two orders ofmagnitude, with an actual increase in quality [6]. The combinationof all these state-of-the-art methods enables an interaction roundof less than 30 ms on average using only limited computationalresources: one 2.4 GHz CPU core and less than 6 GB of RAM.4 DEMONSTRATIONThemain emphasis of the demonstration will be to allow conferenceparticipants to explore the 99.2 million images of the YFCC100Mcollection using Exquisitor. The authorswill also prepare some inter-esting exploration scenarios that highlight aspects of the YFCC100Mcollection. During the demonstration we hope to engage conferenceparticipants in a discussion that can inspire the multimedia commu-nity to work on scalable multimedia techniques and applications.Demonstration I  MM ’19, October 21–25, 2019, Nice, France1030REFERENCES[1] William H. Beluch, Tim Genewein, Andreas Nürnberger, and Jan M. Köhler. 2018.The Power of Ensembles for Active Learning in Image Classification. In Proc.IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).IEEE Computer Society, Salt Lake City, UT, USA, 9368–9377.[2] Kashyap Chitta, Jose M. Alvarez, and Adam Lesnikowski. 2018. Large-ScaleVisual Active Learning with Deep Probabilistic Ensembles. arXiv:1811.03575.(2018), 10 pages.[3] Gylfi Þór Guðmundsson, Björn Þór Jónsson, and Laurent Amsaleg. 2010. ALarge-scale Performance Study of Cluster-based High-dimensional Indexing. InProc. International Workshop on Very-large-scale Multimedia Corpus, Mining andRetrieval (VLS-MCMR). ACM, Firenze, Italy, 31–36.[4] Thomas S. Huang, Charlie K. Dagli, Shyamsundar Rajaram, Edward Y. Chang,Michael I. Mandel, Graham E. Poliner, and Daniel P. W. Ellis. 2008. ActiveLearning for Interactive Multimedia Retrieval. Proc. IEEE 96, 4 (2008), 648–667.[5] Prateek Jain, Sudheendra Vijayanarasimhan, and Kristen Grauman. 2010. HashingHyperplane Queries to Near Points with Applications to Large-Scale ActiveLearning. In Proc. Conference on Neural Information Processing Systems (NIPS).Curran Associates, Inc., Vancouver, BC, Canada, 928–936.[6] Björn Þór Jónsson, Omar Shahbaz Khan, Hanna Ragnarsdóttir, Þórhildur Þor-leiksdóttir, Jan Zahálka, Stevan Rudinac, Gylfi Þór Guðmundsson, Laurent Am-saleg, and Marcel Worring. 2019. Exquisitor: Interactive Learning at Large.arXiv:1904.08689. (2019), 10 pages.[7] Omar Shahbaz Khan, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, and MarcelWorring. 2019. Exquisitor at the Lifelog Search Challenge 2019. In Proceedings ofthe ACM Workshop on Lifelog Search Challenge, LSC@ICMR 2019. ACM, Ottawa,ON, Canada, 7–11.[8] Akshay Mehra, Jihun Hamm, and Mikhail Belkin. 2018. Fast Interactive ImageRetrieval using Large-Scale Unlabeled Data. arXiv:1802.04204. (2018), 15 pages.[9] Ionuţ Mironică, Bogdan Ionescu, Jasper Uijlings, and Nicu Sebe. 2016. Fisher Ker-nel Temporal Variation-based Relevance Feedback for Video Retrieval. ComputerVision and Image Understanding 143 (2016), 38 – 51.[10] Karl Ni, Roger A. Pearce, Kofi Boakye, Brian Van Essen, Damian Borth, BarryChen, and Eric X. Wang. 2015. Large-Scale Deep Learning on the YFCC100MDataset. arXiv:1502.03409. (2015), 5 pages.[11] Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modellingwith Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challengesfor NLP Frameworks. ELRA, Valletta, Malta, 45–50.[12] Yong Rui, Thomas S. Huang, and Sharad Mehrotra. 1997. Content-Based ImageRetrieval with Relevance Feedback in MARS. In Proc. International Conferenceon Image Processing (ICIP). IEEE Computer Society, Santa Barbara, CA, USA,815–818.[13] Nicolae Suditu and François Fleuret. 2012. Iterative relevance feedback withadaptive exploration/exploitation trade-off. In Proc. ACM International Conferenceon Information and Knowledge Management (CIKM). ACM, Maui, HI, USA, 1323–1331.[14] Nicolae Suditu and François Fleuret. 2016. Adaptive Relevance Feedback forLarge-Scale Image Retrieval. Multimedia Tools Appl. 75, 12 (2016), 6777–6807.[15] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V.Vanhoucke, and A. Rabinovich. 2015. Going Deeper with Convolutions. In Proc.IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).IEEE Computer Society, Boston, MA, USA, 1–9.[16] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni,Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCC100M: The New Datain Multimedia Research. Commun. ACM 59, 2 (2016), 64–73.[17] Jan Zahálka and Marcel Worring. 2014. Towards Interactive, Intelligent, andIntegrated Multimedia Analytics. In Proc. IEEE Conference on Visual AnalyticsScience and Technology (VAST). IEEE Computer Society, Paris, France, 3–12.[18] Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C. Koelma, and MarcelWorring. 2018. Blackthorn: Large-Scale Interactive Multimodal Learning. IEEETransactions on Multimedia 20, 3 (2018), 687–698.Demonstration I  MM ’19, October 21–25, 2019, Nice, France1031

Exquisitor: Breaking the Interaction Barrier for Exploration of 100 Million Images

UvA-DARE

Search4Dev

Thorleiksdóttir, Thórhildur

Gudmundsson, Gylfi Thór

HAL-CentraleSupelec

NARCIS 

Jónsson, Björn Þór

Archive Ouverte en Sciences de l'Information et de la Communication

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Exquisitor: Breaking the Interaction Barrier for Exploration of 100 Million ImagesRagnarsdóttir, H.; Þorleiksdottir, Þ.; Khan, O.S.; Jónsson, B.Þ.; Guðmundsson, G.Þ.;Zahálka, J.; Rudinac, S.; Amsaleg, L.; Worring, M.DOI10.1145/3343031.3350580Publication date2019Document VersionFinal published versionPublished inMM'19LicenseArticle 25fa Dutch Copyright ActLink to publicationCitation for published version (APA):Ragnarsdóttir, H., Þorleiksdottir, Þ., Khan, O. S., Jónsson, B. Þ., Guðmundsson, G. Þ.,Zahálka, J., Rudinac, S., Amsaleg, L., & Worring, M. (2019). Exquisitor: Breaking theInteraction Barrier for Exploration of 100 Million Images. In MM'19: proceedings of the 27thACM Conference on Multimedia : October 21-25, 2019, Nice, France (pp. 1029-1031).Association for Computing Machinery. https://doi.org/10.1145/3343031.3350580General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:26 Jul 2022Exquisitor: Breaking the Interaction Barrierfor Exploration of 100 Million ImagesHanna RagnarsdóttirReykjavik UniversityReykjavik, Icelandhannar15@ru.isÞórhildur ÞorleiksdóttirReykjavik UniversityReykjavik, Icelandthorhildurt15@ru.isOmar Shahbaz KhanIT University of CopenhagenCopenhagen, Denmarkomsh@itu.dkBjörn Þór JónssonIT University of CopenhagenCopenhagen, Denmarkbjorn@itu.dkGylfi Þór GuðmundssonReykjavik UniversityReykjavik, Icelandgylfig@ru.isJan Zahálkabohem.aiPrague, Czech Republicjan.zahalka@bohem.aiStevan RudinacUniversity of AmsterdamAmsterdam, Netherlandss.rudinac@uva.nlLaurent AmsalegCNRS–IRISARennes, Francelaurent.amsaleg@irisa.frMarcel WorringUniversity of AmsterdamAmsterdam, Netherlandsm.worring@uva.nlABSTRACTIn this demonstration, we present Exquisitor, a media explorercapable of learning user preferences in real-time during interactionswith the 99.2 million images of YFCC100M. Exquisitor owes itsefficiency to innovations in data representation, compression, andindexing. Exquisitor can complete each interaction round, includinglearning preferences and presenting the most relevant results, inless than 30 ms using only a single CPU core and modest RAM.In short, Exquisitor can bring large-scale interactive learning tostandard desktops and laptops, and even high-end mobile devices.CCS CONCEPTS• Information systems → Multimedia and multimodal re-trieval; Multimedia databases;KEYWORDSInteractive multimodal learning; Scalability; 100 million images.ACM Reference Format:Hanna Ragnarsdóttir, Þórhildur Þorleiksdóttir, Omar Shahbaz Khan, BjörnÞór Jónsson, Gylfi Þór Guðmundsson, Jan Zahálka, Stevan Rudinac, LaurentAmsaleg, and Marcel Worring. 2019. Exquisitor: Breaking the InteractionBarrier for Exploration of 100 Million Images. In Proceedings of the 27thACM International Conference on Multimedia (MM ’19), October 21–25, 2019,Nice, France. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3343031.33505801 INTRODUCTIONMultimedia collections have become a cornerstone data resource ina variety of scientific and industrial fields. One of the most difficultPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).MM ’19, October 21–25, 2019, Nice, France© 2019 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-6889-6/19/10.https://doi.org/10.1145/3343031.3350580Figure 1: The Exquisitor demonstration interface. Exquisi-tor interactively learns new analytic categories over the fullYFCC100M image collection, with average latency of lessthan 30 ms per interaction round, using hardware compa-rable with standard desktops and modern mobile devices.challenges for interactive exploration of such collections—not onlyfor data scientists working with these collections, but also for themultimedia community as a whole—is their scale. How can wefacilitate efficient access to multimedia collections comprising tensor hundreds of millions of images, let alone billions?Interactive learning, which was embraced by the multimediacommunity in the early days of content-based image and videoretrieval [4, 12], has recently experienced revival as an umbrellamethod capable of satisfying a variety of multimedia informationneeds, ranging from exploratory browsing to seeking a particularknown item [17]. Through feedback from the user, interactive learn-ing can adapt to the intent and knowledge of the user, and thuscollaboratewith the user towards, e.g., learning new or unknown an-alytic categories on the fly. Previous contributions, however, largelyoperated at a relatively small scale [1, 2, 8, 10, 13, 14].Demonstration I  MM ’19, October 21–25, 2019, Nice, France1029Consider, as an example, the YFCC100M collection [16], com-prising 99.2M images and 0.8M videos. It has existed for some timenow, but very few have a good idea about what its actual contentsare. This is no surprise, as it is difficult to tackle this collection withexisting techniques: the simple metadata-based filtering approachis impeded by the sparse and noisy nature of the metadata and, atthis scale, similarity search is akin to shooting in the dark. Semanticconcept detectors can be used to generate additional content-basedmetadata for both approaches, but that does not alleviate theirproblems. And thus, the YFCC100M collection remains a mystery.We have recently developed Exquisitor, a highly scalable inter-active multimodal learning approach [6]. A key feature that setsExquisitor apart from related approaches is its scalability: Exquisitorcan retrieve suggestions from 100 million images with sub-secondlatency, using extremely modest computing resources, thus break-ing the interaction barrier for large-scale interactive learning. Inthis demonstration, we propose to allowACMMultimedia attendeesto interactively explore the YFCC100M collection with Exquisitor.2 EXQUISITOR INTERFACEThe current Exquisitor user interface, shown in Figure 1, is browser-based and implemented using the React JavaScript library. It is afairly traditional interactive learning interface, in that users areasked to label positive and negative examples, which are then usedto learn their preferences and determine the new round of sug-gestions. Due to the extreme efficiency of the interactive learningprocess, however, there are some notable differences from tradi-tional interactive learning interfaces:• The learning process runs unobtrusively in the background,continuously providing new on-demand relevant examplesas the user progresses with her exploration, instead of re-quiring explicit management.• Individual images are replaced, rather than the entire screen,for a smooth transition from one interaction round to theother. Users are allowed to indicate that images are neitherpositive nor negative to get a new suggestion, and imagesthat have been visible for some time, but not tagged as posi-tive or negative, can also be replaced with new suggestions.• Users can revisit positive and negative examples, removingor even reversing the feedback label, as their understandingof the collection contents and its relevance evolves.Overall, the user interface is intended to provide a smooth learningexperience. We have already used Exquisitor in the Lifelog SearchChallenge (LSC) 2019 [7], and a detailed evaluation of the userexperience is part of our future work.3 THE LEARNING PROCESSExquisitor’s back-end produces relevant results to show to the userin less than 30 ms per interaction round, including learning theuser preference and scoring the collection, using one 2.4 GHz CPUcore and less than 6 GB of RAM. The back-end system is composedof two web services, as shown in Figure 2. The ImageAPI serviceserves thumbnails for the YFCC100M image collection, as requiredby the user interface. The LearningAPI servicewraps the underlyingmultimodal learning engine, decribed in [6]. In the remainder ofthis section, we briefly outline the multimodal learning process.Figure 2: Exquisitor demonstration system overview.To enable interactive multimodal learning, visual and text fea-tures were extracted from the images of the YFCC100M collection.For visual features, the 1000 ImageNet semantic concepts wereextracted using a GoogLeNet architecture [15]. For the text modal-ity, 100 LDA topics were extracted from the image title, tags anddescription using the gensim framework [11].Uncompressed features for the 99.2M images require nearly880GB of memory. Exquisitor uses the recently proposed Ratio-64 representation, which preserves only the top visual conceptsand text topics for each image [18]. The compressed feature collec-tions require less than 6GB of storage, thus fitting into the memoryof a standard consumer PC, as well as some high-endmobile devices.This effective compression method has been shown to preserve thesemantic descriptiveness of the visual and text features [18].The interactive learning process is facilitated using a linear SVMmodel, proven to provide a good balance between efficiency andaccuracy when classifying large datasets based on few trainingexamples [5, 9]. Based on the relevance indication provided bythe user, a classifier is trained separately for the text and visualmodalities and the images furthest from the hyperplane are selected.The final list of results is created using rank aggregation.The compressed feature data is indexed using a variant of theextended Cluster Pruning (eCP) high-dimensional indexing algo-rithm [3]. By directing Exquisitor’s attention to the clusters withrepresentatives most relevant to the learned linear SVM model,the work of scoring candidates is reduced by nearly two orders ofmagnitude, with an actual increase in quality [6]. The combinationof all these state-of-the-art methods enables an interaction roundof less than 30 ms on average using only limited computationalresources: one 2.4 GHz CPU core and less than 6 GB of RAM.4 DEMONSTRATIONThemain emphasis of the demonstration will be to allow conferenceparticipants to explore the 99.2 million images of the YFCC100Mcollection using Exquisitor. The authorswill also prepare some inter-esting exploration scenarios that highlight aspects of the YFCC100Mcollection. During the demonstration we hope to engage conferenceparticipants in a discussion that can inspire the multimedia commu-nity to work on scalable multimedia techniques and applications.Demonstration I  MM ’19, October 21–25, 2019, Nice, France1030REFERENCES[1] William H. Beluch, Tim Genewein, Andreas Nürnberger, and Jan M. Köhler. 2018.The Power of Ensembles for Active Learning in Image Classification. In Proc.IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).IEEE Computer Society, Salt Lake City, UT, USA, 9368–9377.[2] Kashyap Chitta, Jose M. Alvarez, and Adam Lesnikowski. 2018. Large-ScaleVisual Active Learning with Deep Probabilistic Ensembles. arXiv:1811.03575.(2018), 10 pages.[3] Gylfi Þór Guðmundsson, Björn Þór Jónsson, and Laurent Amsaleg. 2010. ALarge-scale Performance Study of Cluster-based High-dimensional Indexing. InProc. International Workshop on Very-large-scale Multimedia Corpus, Mining andRetrieval (VLS-MCMR). ACM, Firenze, Italy, 31–36.[4] Thomas S. Huang, Charlie K. Dagli, Shyamsundar Rajaram, Edward Y. Chang,Michael I. Mandel, Graham E. Poliner, and Daniel P. W. Ellis. 2008. ActiveLearning for Interactive Multimedia Retrieval. Proc. IEEE 96, 4 (2008), 648–667.[5] Prateek Jain, Sudheendra Vijayanarasimhan, and Kristen Grauman. 2010. HashingHyperplane Queries to Near Points with Applications to Large-Scale ActiveLearning. In Proc. Conference on Neural Information Processing Systems (NIPS).Curran Associates, Inc., Vancouver, BC, Canada, 928–936.[6] Björn Þór Jónsson, Omar Shahbaz Khan, Hanna Ragnarsdóttir, Þórhildur Þor-leiksdóttir, Jan Zahálka, Stevan Rudinac, Gylfi Þór Guðmundsson, Laurent Am-saleg, and Marcel Worring. 2019. Exquisitor: Interactive Learning at Large.arXiv:1904.08689. (2019), 10 pages.[7] Omar Shahbaz Khan, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, and MarcelWorring. 2019. Exquisitor at the Lifelog Search Challenge 2019. In Proceedings ofthe ACM Workshop on Lifelog Search Challenge, LSC@ICMR 2019. ACM, Ottawa,ON, Canada, 7–11.[8] Akshay Mehra, Jihun Hamm, and Mikhail Belkin. 2018. Fast Interactive ImageRetrieval using Large-Scale Unlabeled Data. arXiv:1802.04204. (2018), 15 pages.[9] Ionuţ Mironică, Bogdan Ionescu, Jasper Uijlings, and Nicu Sebe. 2016. Fisher Ker-nel Temporal Variation-based Relevance Feedback for Video Retrieval. ComputerVision and Image Understanding 143 (2016), 38 – 51.[10] Karl Ni, Roger A. Pearce, Kofi Boakye, Brian Van Essen, Damian Borth, BarryChen, and Eric X. Wang. 2015. Large-Scale Deep Learning on the YFCC100MDataset. arXiv:1502.03409. (2015), 5 pages.[11] Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modellingwith Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challengesfor NLP Frameworks. ELRA, Valletta, Malta, 45–50.[12] Yong Rui, Thomas S. Huang, and Sharad Mehrotra. 1997. Content-Based ImageRetrieval with Relevance Feedback in MARS. In Proc. International Conferenceon Image Processing (ICIP). IEEE Computer Society, Santa Barbara, CA, USA,815–818.[13] Nicolae Suditu and François Fleuret. 2012. Iterative relevance feedback withadaptive exploration/exploitation trade-off. In Proc. ACM International Conferenceon Information and Knowledge Management (CIKM). ACM, Maui, HI, USA, 1323–1331.[14] Nicolae Suditu and François Fleuret. 2016. Adaptive Relevance Feedback forLarge-Scale Image Retrieval. Multimedia Tools Appl. 75, 12 (2016), 6777–6807.[15] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V.Vanhoucke, and A. Rabinovich. 2015. Going Deeper with Convolutions. In Proc.IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).IEEE Computer Society, Boston, MA, USA, 1–9.[16] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni,Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCC100M: The New Datain Multimedia Research. Commun. ACM 59, 2 (2016), 64–73.[17] Jan Zahálka and Marcel Worring. 2014. Towards Interactive, Intelligent, andIntegrated Multimedia Analytics. In Proc. IEEE Conference on Visual AnalyticsScience and Technology (VAST). IEEE Computer Society, Paris, France, 3–12.[18] Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C. Koelma, and MarcelWorring. 2018. Blackthorn: Large-Scale Interactive Multimodal Learning. IEEETransactions on Multimedia 20, 3 (2018), 687–698.Demonstration I  MM ’19, October 21–25, 2019, Nice, France1031

HAL-Rennes 1

The IT University of Copenhagen's Repository

Exquisitor: Breaking the Interaction Barrierfor Exploration of 100 Million ImagesHanna RagnarsdóttirReykjavik UniversityReykjavik, Icelandhannar15@ru.isÞórhildur ÞorleiksdóttirReykjavik UniversityReykjavik, Icelandthorhildurt15@ru.isOmar Shahbaz KhanIT University of CopenhagenCopenhagen, Denmarkomsh@itu.dkBjörn Þór JónssonIT University of CopenhagenCopenhagen, Denmarkbjorn@itu.dkGylfi Þór GuðmundssonReykjavik UniversityReykjavik, Icelandgylfig@ru.isJan Zahálkabohem.aiPrague, Czech Republicjan.zahalka@bohem.aiStevan RudinacUniversity of AmsterdamAmsterdam, Netherlandss.rudinac@uva.nlLaurent AmsalegCNRS–IRISARennes, Francelaurent.amsaleg@irisa.frMarcel WorringUniversity of AmsterdamAmsterdam, Netherlandsm.worring@uva.nlABSTRACTIn this demonstration, we present Exquisitor, a media explorercapable of learning user preferences in real-time during interactionswith the 99.2 million images of YFCC100M. Exquisitor owes itsefficiency to innovations in data representation, compression, andindexing. Exquisitor can complete each interaction round, includinglearning preferences and presenting the most relevant results, inless than 30 ms using only a single CPU core and modest RAM.In short, Exquisitor can bring large-scale interactive learning tostandard desktops and laptops, and even high-end mobile devices.CCS CONCEPTS• Information systems → Multimedia and multimodal re-trieval;Multimedia databases;KEYWORDSInteractive multimodal learning; Scalability; 100 million images.ACM Reference Format:Hanna Ragnarsdóttir, Þórhildur Þorleiksdóttir, Omar Shahbaz Khan, BjörnÞór Jónsson, Gylfi Þór Guðmundsson, Jan Zahálka, Stevan Rudinac, LaurentAmsaleg, and Marcel Worring. 2019. Exquisitor: Breaking the InteractionBarrier for Exploration of 100 Million Images. In Proceedings of the 27thACM International Conference on Multimedia (MM ’19), October 21–25, 2019,Nice, France. ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3343031.33505801 INTRODUCTIONMultimedia collections have become a cornerstone data resource ina variety of scientific and industrial fields. One of the most difficultPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).MM ’19, October 21–25, 2019, Nice, France© 2019 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-6889-6/19/10.https://doi.org/10.1145/3343031.3350580Figure 1: The Exquisitor demonstration interface. Exquisi-tor interactively learns new analytic categories over the fullYFCC100M image collection, with average latency of lessthan 30 ms per interaction round, using hardware compa-rable with standard desktops and modern mobile devices.challenges for interactive exploration of such collections—not onlyfor data scientists working with these collections, but also for themultimedia community as a whole—is their scale. How can wefacilitate efficient access to multimedia collections comprising tensor hundreds of millions of images, let alone billions?Interactive learning, which was embraced by the multimediacommunity in the early days of content-based image and videoretrieval [4, 12], has recently experienced revival as an umbrellamethod capable of satisfying a variety of multimedia informationneeds, ranging from exploratory browsing to seeking a particularknown item [17]. Through feedback from the user, interactive learn-ing can adapt to the intent and knowledge of the user, and thuscollaboratewith the user towards, e.g., learning new or unknown an-alytic categories on the fly. Previous contributions, however, largelyoperated at a relatively small scale [1, 2, 8, 10, 13, 14].Consider, as an example, the YFCC100M collection [16], com-prising 99.2M images and 0.8M videos. It has existed for some timenow, but very few have a good idea about what its actual contentsare. This is no surprise, as it is difficult to tackle this collection withexisting techniques: the simple metadata-based filtering approachis impeded by the sparse and noisy nature of the metadata and, atthis scale, similarity search is akin to shooting in the dark. Semanticconcept detectors can be used to generate additional content-basedmetadata for both approaches, but that does not alleviate theirproblems. And thus, the YFCC100M collection remains a mystery.We have recently developed Exquisitor, a highly scalable inter-active multimodal learning approach [6]. A key feature that setsExquisitor apart from related approaches is its scalability: Exquisitorcan retrieve suggestions from 100 million images with sub-secondlatency, using extremely modest computing resources, thus break-ing the interaction barrier for large-scale interactive learning. Inthis demonstration, we propose to allowACMMultimedia attendeesto interactively explore the YFCC100M collection with Exquisitor.2 EXQUISITOR INTERFACEThe current Exquisitor user interface, shown in Figure 1, is browser-based and implemented using the React JavaScript library. It is afairly traditional interactive learning interface, in that users areasked to label positive and negative examples, which are then usedto learn their preferences and determine the new round of sug-gestions. Due to the extreme efficiency of the interactive learningprocess, however, there are some notable differences from tradi-tional interactive learning interfaces:• The learning process runs unobtrusively in the background,continuously providing new on-demand relevant examplesas the user progresses with her exploration, instead of re-quiring explicit management.• Individual images are replaced, rather than the entire screen,for a smooth transition from one interaction round to theother. Users are allowed to indicate that images are neitherpositive nor negative to get a new suggestion, and imagesthat have been visible for some time, but not tagged as posi-tive or negative, can also be replaced with new suggestions.• Users can revisit positive and negative examples, removingor even reversing the feedback label, as their understandingof the collection contents and its relevance evolves.Overall, the user interface is intended to provide a smooth learningexperience. We have already used Exquisitor in the Lifelog SearchChallenge (LSC) 2019 [7], and a detailed evaluation of the userexperience is part of our future work.3 THE LEARNING PROCESSExquisitor’s back-end produces relevant results to show to the userin less than 30 ms per interaction round, including learning theuser preference and scoring the collection, using one 2.4 GHz CPUcore and less than 6 GB of RAM. The back-end system is composedof two web services, as shown in Figure 2. The ImageAPI serviceserves thumbnails for the YFCC100M image collection, as requiredby the user interface. The LearningAPI servicewraps the underlyingmultimodal learning engine, decribed in [6]. In the remainder ofthis section, we briefly outline the multimodal learning process.Figure 2: Exquisitor demonstration system overview.To enable interactive multimodal learning, visual and text fea-tures were extracted from the images of the YFCC100M collection.For visual features, the 1000 ImageNet semantic concepts wereextracted using a GoogLeNet architecture [15]. For the text modal-ity, 100 LDA topics were extracted from the image title, tags anddescription using the gensim framework [11].Uncompressed features for the 99.2M images require nearly880GB of memory. Exquisitor uses the recently proposed Ratio-64 representation, which preserves only the top visual conceptsand text topics for each image [18]. The compressed feature collec-tions require less than 6GB of storage, thus fitting into the memoryof a standard consumer PC, as well as some high-endmobile devices.This effective compression method has been shown to preserve thesemantic descriptiveness of the visual and text features [18].The interactive learning process is facilitated using a linear SVMmodel, proven to provide a good balance between efficiency andaccuracy when classifying large datasets based on few trainingexamples [5, 9]. Based on the relevance indication provided bythe user, a classifier is trained separately for the text and visualmodalities and the images furthest from the hyperplane are selected.The final list of results is created using rank aggregation.The compressed feature data is indexed using a variant of theextended Cluster Pruning (eCP) high-dimensional indexing algo-rithm [3]. By directing Exquisitor’s attention to the clusters withrepresentatives most relevant to the learned linear SVM model,the work of scoring candidates is reduced by nearly two orders ofmagnitude, with an actual increase in quality [6]. The combinationof all these state-of-the-art methods enables an interaction roundof less than 30 ms on average using only limited computationalresources: one 2.4 GHz CPU core and less than 6 GB of RAM.4 DEMONSTRATIONThemain emphasis of the demonstration will be to allow conferenceparticipants to explore the 99.2 million images of the YFCC100Mcollection using Exquisitor. The authorswill also prepare some inter-esting exploration scenarios that highlight aspects of the YFCC100Mcollection. During the demonstration we hope to engage conferenceparticipants in a discussion that can inspire the multimedia commu-nity to work on scalable multimedia techniques and applications.REFERENCES[1] William H. Beluch, Tim Genewein, Andreas Nürnberger, and Jan M. Köhler. 2018.The Power of Ensembles for Active Learning in Image Classification. In Proc.IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).IEEE Computer Society, Salt Lake City, UT, USA, 9368–9377.[2] Kashyap Chitta, Jose M. Alvarez, and Adam Lesnikowski. 2018. Large-ScaleVisual Active Learning with Deep Probabilistic Ensembles. arXiv:1811.03575.(2018), 10 pages.[3] Gylfi Þór Guðmundsson, Björn Þór Jónsson, and Laurent Amsaleg. 2010. ALarge-scale Performance Study of Cluster-based High-dimensional Indexing. InProc. International Workshop on Very-large-scale Multimedia Corpus, Mining andRetrieval (VLS-MCMR). ACM, Firenze, Italy, 31–36.[4] Thomas S. Huang, Charlie K. Dagli, Shyamsundar Rajaram, Edward Y. Chang,Michael I. Mandel, Graham E. Poliner, and Daniel P. W. Ellis. 2008. ActiveLearning for Interactive Multimedia Retrieval. Proc. IEEE 96, 4 (2008), 648–667.[5] Prateek Jain, Sudheendra Vijayanarasimhan, and Kristen Grauman. 2010. HashingHyperplane Queries to Near Points with Applications to Large-Scale ActiveLearning. In Proc. Conference on Neural Information Processing Systems (NIPS).Curran Associates, Inc., Vancouver, BC, Canada, 928–936.[6] Björn Þór Jónsson, Omar Shahbaz Khan, Hanna Ragnarsdóttir, Þórhildur Þor-leiksdóttir, Jan Zahálka, Stevan Rudinac, Gylfi Þór Guðmundsson, Laurent Am-saleg, and Marcel Worring. 2019. Exquisitor: Interactive Learning at Large.arXiv:1904.08689. (2019), 10 pages.[7] Omar Shahbaz Khan, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, and MarcelWorring. 2019. Exquisitor at the Lifelog Search Challenge 2019. In Proceedings ofthe ACM Workshop on Lifelog Search Challenge, LSC@ICMR 2019. ACM, Ottawa,ON, Canada, 7–11.[8] Akshay Mehra, Jihun Hamm, and Mikhail Belkin. 2018. Fast Interactive ImageRetrieval using Large-Scale Unlabeled Data. arXiv:1802.04204. (2018), 15 pages.[9] Ionuţ Mironică, Bogdan Ionescu, Jasper Uijlings, and Nicu Sebe. 2016. Fisher Ker-nel Temporal Variation-based Relevance Feedback for Video Retrieval. ComputerVision and Image Understanding 143 (2016), 38 – 51.[10] Karl Ni, Roger A. Pearce, Kofi Boakye, Brian Van Essen, Damian Borth, BarryChen, and Eric X. Wang. 2015. Large-Scale Deep Learning on the YFCC100MDataset. arXiv:1502.03409. (2015), 5 pages.[11] Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modellingwith Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challengesfor NLP Frameworks. ELRA, Valletta, Malta, 45–50.[12] Yong Rui, Thomas S. Huang, and Sharad Mehrotra. 1997. Content-Based ImageRetrieval with Relevance Feedback in MARS. In Proc. International Conferenceon Image Processing (ICIP). IEEE Computer Society, Santa Barbara, CA, USA,815–818.[13] Nicolae Suditu and François Fleuret. 2012. Iterative relevance feedback withadaptive exploration/exploitation trade-off. In Proc. ACM International Conferenceon Information and Knowledge Management (CIKM). ACM, Maui, HI, USA, 1323–1331.[14] Nicolae Suditu and François Fleuret. 2016. Adaptive Relevance Feedback forLarge-Scale Image Retrieval. Multimedia Tools Appl. 75, 12 (2016), 6777–6807.[15] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V.Vanhoucke, and A. Rabinovich. 2015. Going Deeper with Convolutions. In Proc.IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).IEEE Computer Society, Boston, MA, USA, 1–9.[16] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni,Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCC100M: The New Datain Multimedia Research. Commun. ACM 59, 2 (2016), 64–73.[17] Jan Zahálka and Marcel Worring. 2014. Towards Interactive, Intelligent, andIntegrated Multimedia Analytics. In Proc. IEEE Conference on Visual AnalyticsScience and Technology (VAST). IEEE Computer Society, Paris, France, 3–12.[18] Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C. Koelma, and MarcelWorring. 2018. Blackthorn: Large-Scale Interactive Multimodal Learning. IEEETransactions on Multimedia 20, 3 (2018), 687–698.

Beluch William H.

Jain Prateek

Khan Omar Shahbaz

Radim

Szegedy C.

Crossref

Exquisitor

https://pure.uva.nl/ws/files/49457630/3343031.3350580.pdf

Exquisitor: Breaking the Interaction Barrier for Exploration of 100 Million Images

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications

UvA-DARE

Search4Dev

HAL-CentraleSupelec

NARCIS

Archive Ouverte en Sciences de l'Information et de la Communication

International Migration, Integration and Social Cohesion online publications

HAL-Rennes 1

The IT University of Copenhagen's Repository

Crossref