The 2nd workshop on the evaluation of collaborative information retrieval and seeking (ECol) was held in conjunction with the ACM SIGIR Conference on Human Information Interaction &amp; Retrieval (CHIIR) in Oslo, Norway. The workshop focused on discussing the challenges and difficulties of researching and studying collaborative information retrieval and seeking (CIS/CIR). After an introductory and scene setting overview of developments in CIR/CIS, participants were challenged with devising a range of possible CIR/CIS tasks that could be used for evaluation purposes. Through the brainstorming and discussions, valuable insights regarding the evaluation of CIR/CIS tasks become apparent ? for particular tasks efficiency and/or effectiveness is most important, however for the majority of tasks the success and quality of outcomes along with knowledge sharing and sense-making were most important ? of which these latter attributes are much more difficult to measure and evaluate. Thus the major challenge for CIR/CIS research is to develop methods, measures and methodologies to evaluate these high order attributes

Azzopardi, Leif

Pickens, Jeremy

Shah, Chirag

Soulier, Laure

Tamine, Lynda

English

Enlighten: Publications

  
 
 
 
 
Azzopardi, L., Pickens, J., Shah, C., Soulier, L. and Tamine, L. (2017) Report on 
the Second International Workshop on the Evaluation on Collaborative Information 
Seeking and Retrieval (ECol'2017 @ CHIIR). ACM SIGIR Forum, 51(3), pp. 122-
127. 
 
   
There may be differences between this version and the published version. You are 
advised to consult the publisher’s version if you wish to cite from it. 
 
© ACM 2017. This is the author's version of the work. It is posted here for your 
personal use. Not for redistribution. The definitive Version of Record was 
published in ACM SIGIR Forum, 51(3), pp. 122-127, 
https://doi.org/10.1145/3190580.3190599.   
 
 
http://eprints.gla.ac.uk/158108/ 
     
 
 
 
 
 
 
Deposited on: 15 March 2018 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Enlighten – Research publications by members of the University of Glasgow 
http://eprints.gla.ac.uk 
Second International Workshop On the Evaluation of
Collaborative Information Seeking and Retrieval (Ecol’17)
Leif Azzopardi
University of Strathclyde
Glasgow, UK
Leif.Azzopardi@strath.ac.uk
Jeremy Pickens
Catalyst Repository Systems
Palo Alto, CA, USA
jpickens@catalystsecure.com
Chirag Shah
Rutgers University
New Brunswick, NJ, USA
chirags@rutgers.edu
Laure Soulier
Sorbonne Universités, UPMC
Univ Paris 06, LIP6 UMR 7606
75005 Paris, France
laure.soulier@lip6.fr
Lynda Tamine
Université de Toulouse UPS IRIT
Toulouse, France
tamine@irit.fr
ABSTRACT
The workshop on the evaluation of collaborative informa-
tion retrieval and seeking (ECol) is held in conjunction with
the ACM SIGIR Conference on Human Information Inter-
action & Retrieval (CHIIR) in Oslo, Norway. To make the
workshop active and the participant pro-active, we released
datasets and tools so as to help researchers contributing to
the formalization of evaluation frameworks for challenging
collaborative tasks. The workshop is split into two parts.
First, a presentation session. Then, the afternoon is devoted
to group discussion addressing challenges of evaluating and
designing models for social and collaborative search.
Keywords
Collaborative information retrieval, social information re-
trieval, task, evaluation
1. INTRODUCTION
The paradigm of Collaborative Information Seeking (CIS)
and Retrieval (CIR) refers to methodologies and technolo-
gies that support collective-knowledge sharing within a work
team in order to solve a shared complex problem [5]. Accord-
ingly, the main underlying challenge is to satisfy the mutual
beneficial goals of both individual users and the collabora-
tive group while maintaining a reasonable level of cognitive
e↵ort underlying users’ interactions [11]. Indeed, collabo-
rative search is also known as a social process [4] in which
users leverage from other users’ interactions and social sig-
nals (e.g., bookmarks and annotations). In the recent years,
several studies [7, 16, 8] have been carried out to understand
the behavioral process of users in terms of question asking
on search engines vs. social networks. In this context, so-
cial platforms (e.g., social networks [9, 2] and community
question-answering [6, 3]) have been acknowledged as a place
where users collaborate to solve an information need. This
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
CHIIR ’17, March 7–11, 2017, Oslo, Norway.
c  2017 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-4677-1/17/03.
DOI: http://dx.doi.org/10.1145/3020165.3022164
provides numerous opportunities for new and novel research
within the field of CIR/CIS where the gap between social
and collaborative search can be brought together.
However, the evaluation for CIS/CIR and social & collab-
orative IR models are still challenging, as there are a vari-
ety of confounding factors such as the multi-user and multi-
level contexts, the exploratory aspect of the search through
multi-session search activities, the multiplicity of relevance
factors, the individual vs. collective value of relevance, the
search interfaces supporting the collaborative interactions as
well as the social interactions themselves. While, substan-
tial research advances in the evaluation of non-collaborative
information retrieval and seeking tasks have been achieved
through international evaluation campaigns such as TREC,
CLEF and NTIR, to date and to our knowledge, no stan-
dardization e↵ort has been achieved for the evaluation of
CIS/CIR and social-CIR. During the first ECol Workshop
there was strong consensus that there is an important need
to investigate the evaluation challenge in CIS/CIR/social-
CIR with the aim of creating common evaluation frameworks
that would foster the research area.
A first edition of this workshop has been organized at
CIKM 2015 in which we discussed about the issue of evalua-
tion in CIS/CIR [14]. The second edition of the ECol Work-
shop1 [14] focuses the discussion and provides concrete con-
tributions in developing such an evaluation framework for
the unique challenges in CIS/CIR and social-collaborative
IR. To make the workshop active and the participant pro-
active, we release datasets and tools so as to help researchers
contributing to the formalization of evaluation frameworks
for challenging collaborative tasks (that we will identify).
Finally, we hope that this workshop will be beneficial for
the community in both short- and long-term. First, this
would allow to understand relevance factors or design stan-
dard evaluation frameworks. Second, provided resources
(datasets, tool, and tasks) would remain available for the
whole community, facilitating in the future comparable and
reproducible experiments.
2. SCOPE AND NOVELTY OF THE
WORKSHOP
These last years and particularly since 2005, CIS and CIR
have became emerging topics that have been addressed in
1https://www.irit.fr/ECol2015/
several IR and IS conferences including CIKM and SIGIR
conferences [1, 14]. While the potential of collaboration has
been highlighted with respect to individual settings, other
challenges remain and need to be thoroughly explored. De-
spite most of experimental evaluations have been done with
the objective of highlighting the synergic e↵ect of the pro-
posed contributions, there is an important need for the fu-
ture to discuss about what should be evaluated in terms of
collaboration aspects (e.g. cognitive e↵ort, mutual benefi-
cial goal satisfaction, collective relevance...). Moreover, it
does not exist standard framework as proposed in ad-hoc
information retrieval through the evaluation campaign, as
those proposed by TREC, INEX, CLEF, etc.
While a follow on from our previous workshop, this
workshop has two distinguishing and novel elements: (i)
it has a specific focus on social IR and collaborative IR
evaluation, bridging the gap within this space, and (ii) it
provides datasets, tools and new tasks for participants and
others to undertake evaluations and explore this space. We
believe that formalizing evaluation frameworks for a such
domain would lead researchers to investigate this research
area and propose new social-collaborative models.
Participants will have di↵erent ways to participate to the
workshop:
• Use the provided datasets/tools to propose an evalu-
ation framework for the identified tasks.
• Use their own datasets/tools to propose an evaluation
framework for the identified tasks.
• Use the provided datasets/tools to propose an eval-
uation framework for a task they have identified.
• Propose an evaluation framework without support of
datasets/tools for a task we/they have identified.
• Designing possible tasks (with/without proposing
models and evaluation frameworks) on provided or
their own datasets.
This opens future research directions and would lead to the
design of evaluation frameworks. This would also enable
comparable contributions in terms of evaluation framework
and constructive discussions throughout the workshop.
3. RELEASED DATASETS AND TOOLS
To make the workshop interactive and also in a long-term
objective in the field, we share resources and tools:
• Datasets:
– Social-based datasets: tweet collections about
two crises, respectively the Ebola virus epidemic
and the hurricane Sandy2 [13, 15].
– Collaborative search logs3 [12].
• Tools:
– Coagmento collaborative interface [10].
– Open-source version of Coagmento for doing user-
studies on github4.
2https://figshare.com/collections/expac/3283118
3http://infoseeking.org/data.php#cis2010
4https://github.com/InfoSeeking/CoagmentoCollaboratory
4. EXAMPLES OF TOPICS
Taking into consideration the release of datasets and tools,
we will propose di↵erent topics that would guide participants
in their contributions. For each topic, participants would be
invited either to design the task or formalize the evaluation
framework (including metrics, datasets, ground truth, base-
lines, etc.). Below we list examples of topics:
• Recommending social collaborators (experts, answer-
ers, sympathizers).
• Collaborative ranking on social platforms (criteria:
federated, novelty, diversity, interactive, time consid-
eration).
• Identifying impacting factors on search e↵ectiveness
(failure, success, struggling).
• Preserving data privacy through multi-level CIR.
• Exploratory search (knowledge acquisition, multi-
faceted search).
• Collaborative intent understanding.
5. INTENDED PROGRAM COMMITTEE
• Thilo Boehm, University Hagen - Germany
• Luanne Freund, University of British Columbia - USA
• Martin Halvey, University of Strathclyde - UK
• Preben Hansen, Stockholm University - Sweden
• Ryan Kelly, University of Bath - UK
• Simon Knight, University of Technology Sydney - Aus-
tralia
• Stephann Makri, University of London - UK
• Jian-Yun Nie, University de Montreal - Canada
• Takehiro Yamamoto, Department of Social Informat-
ics, Graduate School of Informatics, Kyoto University
- Japan
6. REFERENCES
[1] CIR ’11: Proceedings of the 3rd International
Workshop on Collaborative Information Retrieval,
New York, NY, USA, 2011. ACM. 605110.
[2] D. Castilho, P. Melo, D. Quercia, and F. Benevenuto.
Working with friends: Unveiling working a nity
features from facebook data. In ICWSM, 2014.
[3] S. Chang and A. Pal. Routing questions for
collaborative answering in community question
answering. In ASONAM, pages 494–501. ACM, 2013.
[4] B. M. Evans and E. H. Chi. An elaborated model of
social search. IP&M, 46(6):656–678, 2010.
[5] J. Foster. Collaborative information seeking and
retrieval. Annual Review of Information Science and
Technology, 40(1), 2006.
[6] B. Li and I. King. Routing questions to appropriate
answerers in community question answering services.
In CIKM. ACM, 2010.
[7] M. R. Morris, J. Teevan, and K. Panovich. What do
people ask their social networks, and why?: A survey
study of status messages. In CHI, pages 1739–1748,
2010.
[8] A. Oeldorf-Hirsh, M. R. Morris, J. Teevan, and
G. Darren. To search or to ask: The routing of
information needs between traditional search engines
and social networks. In ICWSM, 2014.
[9] S. A. Paul, L. Hong, and H. Chi. Is twitter a good
place for asking questions? a characterization study.
In ICWSM, 2011.
[10] C. Shah. Coagmento- a collaborative information
seeking, synthesis and sense-making framework. In
Integrated demo at CSCW 2010, 2010.
[11] C. Shah and R. Gonza´lez-Iba´n˜ez. Evaluating the
synergic e↵ect of collaboration in information seeking.
In SIGIR, pages 913–922, 2011.
[12] C. Shah and R. Gonzalez-Ibanez. Collaborative
information seeking lab experiments dataset.
http://infoseeking.org/data.php#cis2010, 2017.
[13] L. Soulier, L. Tamine, and G. Nguyen. Answering
twitter questions: a model for recommending
answerers through social collaboration. In CIKM,
pages 267–276, 2016.
[14] L. Soulier, L. Tamine, T. Sakai, L. Azzopardi, and
J. Pickens. Report on the first international workshop
on the evaluation on collaborative information seeking
and retrieval (ecol’2015). SIGIR Forum, 50(1):42–48,
2016.
[15] L. Tamine, L. Soulier, L. Ben Jabeur, F. Amblard,
C. Hanachi, G. Hubert, and C. Roth. Social
media-based collaborative information access:
Analysis of online crisis-related twitter conversations.
In HT, pages 159–168. ACM, 2016.
[16] J. Teevan, D. Ramage, and M.-R. Morris. Twitter
search: A comparison of microblog search and web
search. In WSDM, pages 35–44, 2011.


Report on the Second International Workshop on the Evaluation on Collaborative Information Seeking and Retrieval (ECol'2017 @ CHIIR)

Enlighten

      Azzopardi, L., Pickens, J., Shah, C., Soulier, L. and Tamine, L. (2017) Report on the Second International Workshop on the Evaluation on Collaborative Information Seeking and Retrieval (ECol'2017 @ CHIIR). ACM SIGIR Forum, 51(3), pp. 122-127.     There may be differences between this version and the published version. You are advised to consult the publisher’s version if you wish to cite from it.  © ACM 2017. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ACM SIGIR Forum, 51(3), pp. 122-127, https://doi.org/10.1145/3190580.3190599.     http://eprints.gla.ac.uk/158108/            Deposited on: 15 March 2018                       Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Second International Workshop On the Evaluation ofCollaborative Information Seeking and Retrieval (Ecol’17)Leif AzzopardiUniversity of StrathclydeGlasgow, UKLeif.Azzopardi@strath.ac.ukJeremy PickensCatalyst Repository SystemsPalo Alto, CA, USAjpickens@catalystsecure.comChirag ShahRutgers UniversityNew Brunswick, NJ, USAchirags@rutgers.eduLaure SoulierSorbonne Universités, UPMCUniv Paris 06, LIP6 UMR 760675005 Paris, Francelaure.soulier@lip6.frLynda TamineUniversité de Toulouse UPS IRITToulouse, Francetamine@irit.frABSTRACTThe workshop on the evaluation of collaborative informa-tion retrieval and seeking (ECol) is held in conjunction withthe ACM SIGIR Conference on Human Information Inter-action & Retrieval (CHIIR) in Oslo, Norway. To make theworkshop active and the participant pro-active, we releaseddatasets and tools so as to help researchers contributing tothe formalization of evaluation frameworks for challengingcollaborative tasks. The workshop is split into two parts.First, a presentation session. Then, the afternoon is devotedto group discussion addressing challenges of evaluating anddesigning models for social and collaborative search.KeywordsCollaborative information retrieval, social information re-trieval, task, evaluation1. INTRODUCTIONThe paradigm of Collaborative Information Seeking (CIS)and Retrieval (CIR) refers to methodologies and technolo-gies that support collective-knowledge sharing within a workteam in order to solve a shared complex problem [5]. Accord-ingly, the main underlying challenge is to satisfy the mutualbeneficial goals of both individual users and the collabora-tive group while maintaining a reasonable level of cognitivee↵ort underlying users’ interactions [11]. Indeed, collabo-rative search is also known as a social process [4] in whichusers leverage from other users’ interactions and social sig-nals (e.g., bookmarks and annotations). In the recent years,several studies [7, 16, 8] have been carried out to understandthe behavioral process of users in terms of question askingon search engines vs. social networks. In this context, so-cial platforms (e.g., social networks [9, 2] and communityquestion-answering [6, 3]) have been acknowledged as a placewhere users collaborate to solve an information need. ThisPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).CHIIR ’17, March 7–11, 2017, Oslo, Norway.c  2017 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-4677-1/17/03.DOI: http://dx.doi.org/10.1145/3020165.3022164provides numerous opportunities for new and novel researchwithin the field of CIR/CIS where the gap between socialand collaborative search can be brought together.However, the evaluation for CIS/CIR and social & collab-orative IR models are still challenging, as there are a vari-ety of confounding factors such as the multi-user and multi-level contexts, the exploratory aspect of the search throughmulti-session search activities, the multiplicity of relevancefactors, the individual vs. collective value of relevance, thesearch interfaces supporting the collaborative interactions aswell as the social interactions themselves. While, substan-tial research advances in the evaluation of non-collaborativeinformation retrieval and seeking tasks have been achievedthrough international evaluation campaigns such as TREC,CLEF and NTIR, to date and to our knowledge, no stan-dardization e↵ort has been achieved for the evaluation ofCIS/CIR and social-CIR. During the first ECol Workshopthere was strong consensus that there is an important needto investigate the evaluation challenge in CIS/CIR/social-CIR with the aim of creating common evaluation frameworksthat would foster the research area.A first edition of this workshop has been organized atCIKM 2015 in which we discussed about the issue of evalua-tion in CIS/CIR [14]. The second edition of the ECol Work-shop1 [14] focuses the discussion and provides concrete con-tributions in developing such an evaluation framework forthe unique challenges in CIS/CIR and social-collaborativeIR. To make the workshop active and the participant pro-active, we release datasets and tools so as to help researcherscontributing to the formalization of evaluation frameworksfor challenging collaborative tasks (that we will identify).Finally, we hope that this workshop will be beneficial forthe community in both short- and long-term. First, thiswould allow to understand relevance factors or design stan-dard evaluation frameworks. Second, provided resources(datasets, tool, and tasks) would remain available for thewhole community, facilitating in the future comparable andreproducible experiments.2. SCOPE AND NOVELTY OF THEWORKSHOPThese last years and particularly since 2005, CIS and CIRhave became emerging topics that have been addressed in1https://www.irit.fr/ECol2015/several IR and IS conferences including CIKM and SIGIRconferences [1, 14]. While the potential of collaboration hasbeen highlighted with respect to individual settings, otherchallenges remain and need to be thoroughly explored. De-spite most of experimental evaluations have been done withthe objective of highlighting the synergic e↵ect of the pro-posed contributions, there is an important need for the fu-ture to discuss about what should be evaluated in terms ofcollaboration aspects (e.g. cognitive e↵ort, mutual benefi-cial goal satisfaction, collective relevance...). Moreover, itdoes not exist standard framework as proposed in ad-hocinformation retrieval through the evaluation campaign, asthose proposed by TREC, INEX, CLEF, etc.While a follow on from our previous workshop, thisworkshop has two distinguishing and novel elements: (i)it has a specific focus on social IR and collaborative IRevaluation, bridging the gap within this space, and (ii) itprovides datasets, tools and new tasks for participants andothers to undertake evaluations and explore this space. Webelieve that formalizing evaluation frameworks for a suchdomain would lead researchers to investigate this researcharea and propose new social-collaborative models.Participants will have di↵erent ways to participate to theworkshop:• Use the provided datasets/tools to propose an evalu-ation framework for the identified tasks.• Use their own datasets/tools to propose an evaluationframework for the identified tasks.• Use the provided datasets/tools to propose an eval-uation framework for a task they have identified.• Propose an evaluation framework without support ofdatasets/tools for a task we/they have identified.• Designing possible tasks (with/without proposingmodels and evaluation frameworks) on provided ortheir own datasets.This opens future research directions and would lead to thedesign of evaluation frameworks. This would also enablecomparable contributions in terms of evaluation frameworkand constructive discussions throughout the workshop.3. RELEASED DATASETS AND TOOLSTo make the workshop interactive and also in a long-termobjective in the field, we share resources and tools:• Datasets:– Social-based datasets: tweet collections abouttwo crises, respectively the Ebola virus epidemicand the hurricane Sandy2 [13, 15].– Collaborative search logs3 [12].• Tools:– Coagmento collaborative interface [10].– Open-source version of Coagmento for doing user-studies on github4.2https://figshare.com/collections/expac/32831183http://infoseeking.org/data.php#cis20104https://github.com/InfoSeeking/CoagmentoCollaboratory4. EXAMPLES OF TOPICSTaking into consideration the release of datasets and tools,we will propose di↵erent topics that would guide participantsin their contributions. For each topic, participants would beinvited either to design the task or formalize the evaluationframework (including metrics, datasets, ground truth, base-lines, etc.). Below we list examples of topics:• Recommending social collaborators (experts, answer-ers, sympathizers).• Collaborative ranking on social platforms (criteria:federated, novelty, diversity, interactive, time consid-eration).• Identifying impacting factors on search e↵ectiveness(failure, success, struggling).• Preserving data privacy through multi-level CIR.• Exploratory search (knowledge acquisition, multi-faceted search).• Collaborative intent understanding.5. INTENDED PROGRAM COMMITTEE• Thilo Boehm, University Hagen - Germany• Luanne Freund, University of British Columbia - USA• Martin Halvey, University of Strathclyde - UK• Preben Hansen, Stockholm University - Sweden• Ryan Kelly, University of Bath - UK• Simon Knight, University of Technology Sydney - Aus-tralia• Stephann Makri, University of London - UK• Jian-Yun Nie, University de Montreal - Canada• Takehiro Yamamoto, Department of Social Informat-ics, Graduate School of Informatics, Kyoto University- Japan6. REFERENCES[1] CIR ’11: Proceedings of the 3rd InternationalWorkshop on Collaborative Information Retrieval,New York, NY, USA, 2011. ACM. 605110.[2] D. Castilho, P. Melo, D. Quercia, and F. Benevenuto.Working with friends: Unveiling working a nityfeatures from facebook data. In ICWSM, 2014.[3] S. Chang and A. Pal. Routing questions forcollaborative answering in community questionanswering. In ASONAM, pages 494–501. ACM, 2013.[4] B. M. Evans and E. H. Chi. An elaborated model ofsocial search. IP&M, 46(6):656–678, 2010.[5] J. Foster. Collaborative information seeking andretrieval. Annual Review of Information Science andTechnology, 40(1), 2006.[6] B. Li and I. King. Routing questions to appropriateanswerers in community question answering services.In CIKM. ACM, 2010.[7] M. R. Morris, J. Teevan, and K. Panovich. What dopeople ask their social networks, and why?: A surveystudy of status messages. In CHI, pages 1739–1748,2010.[8] A. Oeldorf-Hirsh, M. R. Morris, J. Teevan, andG. Darren. To search or to ask: The routing ofinformation needs between traditional search enginesand social networks. In ICWSM, 2014.[9] S. A. Paul, L. Hong, and H. Chi. Is twitter a goodplace for asking questions? a characterization study.In ICWSM, 2011.[10] C. Shah. Coagmento- a collaborative informationseeking, synthesis and sense-making framework. InIntegrated demo at CSCW 2010, 2010.[11] C. Shah and R. Gonza´lez-Iba´n˜ez. Evaluating thesynergic e↵ect of collaboration in information seeking.In SIGIR, pages 913–922, 2011.[12] C. Shah and R. Gonzalez-Ibanez. Collaborativeinformation seeking lab experiments dataset.http://infoseeking.org/data.php#cis2010, 2017.[13] L. Soulier, L. Tamine, and G. Nguyen. Answeringtwitter questions: a model for recommendinganswerers through social collaboration. In CIKM,pages 267–276, 2016.[14] L. Soulier, L. Tamine, T. Sakai, L. Azzopardi, andJ. Pickens. Report on the first international workshopon the evaluation on collaborative information seekingand retrieval (ecol’2015). SIGIR Forum, 50(1):42–48,2016.[15] L. Tamine, L. Soulier, L. Ben Jabeur, F. Amblard,C. Hanachi, G. Hubert, and C. Roth. Socialmedia-based collaborative information access:Analysis of online crisis-related twitter conversations.In HT, pages 159–168. ACM, 2016.[16] J. Teevan, D. Ramage, and M.-R. Morris. Twittersearch: A comparison of microblog search and websearch. In WSDM, pages 35–44, 2011.

http://eprints.gla.ac.uk/158108/1/158108.pdf

Report on the Second International Workshop on the Evaluation on Collaborative Information Seeking and Retrieval (ECol'2017 @ CHIIR)

Abstract

Similar works

Full text

Available Versions

Enlighten: Publications

Enlighten