Thesauri have been important information and knowledge organisation tools for more than three decades. The recent emergence and phenomenal growth of the World Wide Web has created new opportunities to introduce thesauri as information search and retrieval aids to end user communities. While the number of web-based and hypertextual thesauri continues to grow, few investigations have yet been carried out to evaluate how end-users, for whom all these efforts are ostensibly made, interact with and make use of thesauri for query building and expansion. The present paper reports a pilot study carried out to determine the extent to which a thesaurus-enhanced search interface to a web-based database aided end-users in their selection of search terms. The study also investigated the ways in which users interacted with the thesaurus structure, terms, and interface. Thesaurus-based searching and browsing behaviours adopted by users while interacting with the thesaurus-enhanced search interface were also examined

Chowdhury, G.

Revie, C.W.

Shiri, A.A.

English

University of Strathclyde Institutional Repository

Strathprints Institutional RepositoryShiri, A.A. and Revie, C.W. and Chowdhury, G. (2002) Assessing the impact of user interactionwith thesaural knowledge structures: a quantitative analysis framework. In: Proceedings of the 7thInternational Society for Knowledge Organization (ISKO) Conference, 2002-07-10 - 2002-07-13,Granada, Spain.Strathprints is designed to allow users to access the research output of the University of Strathclyde.Copyright c© and Moral Rights for the papers on this site are retained by the individual authorsand/or other copyright owners. You may not engage in further distribution of the material for anyprofitmaking activities or any commercial gain. You may freely distribute both the url (http://strathprints.strath.ac.uk/) and the content of this paper for research or study, educational, ornot-for-profit purposes without prior permission or charge.Any correspondence concerning this service should be sent to Strathprints administrator:mailto:strathprints@strath.ac.ukhttp://strathprints.strath.ac.uk/Challenges in knowledge representation and organization for the 21st century: integration of knowledge across boundaries: Proceedings of the 7th International Society for Knowledge Organization (ISKO) Conference, Granada, Spain, 10-13 July 2002.   Ali Asghar Shiri, Crawford Revie and Gobinda Chowdhury Department of Computer and Information Sciences, University of Strathclyde, Glasgow, UK G1 1XH   Assessing the Impact of User Interaction with Thesaural Knowledge Structures: a Quantitative Analysis Framework   Abstract: Thesauri have been important information and knowledge organisation tools for more than three decades.  The recent emergence and phenomenal growth of the World Wide Web has created new opportunities to introduce thesauri as information search and retrieval aids to end user communities. While the number of web-based and hypertextual thesauri continues to grow, few investigations have yet been carried out to evaluate how end-users, for whom all these efforts are ostensibly made, interact with and make use of thesauri for query building and expansion. The present paper reports a pilot study carried out to determine the extent to which a thesaurus-enhanced search interface to a web-based database aided end-users in their selection of search terms.  The study also investigated the ways in which users interacted with the thesaurus structure, terms, and interface. Thesaurus-based searching and browsing behaviours adopted by users while interacting with the thesaurus-enhanced search interface were also examined.   1. Introduction The last decade has witnessed the emergence of a broad range of applications for knowledge structures in general and thesauri in particular.   A number of researchers have predicted that thesauri will increasingly be used in retrieval rather than for indexing (Milstead, 1998; Aitchison et al., 1997) and that their application in information retrieval systems will become more diverse due to the growth of full-text databases accessed over the Internet (Williamson, 2000). Some researchers have emphasised the need for tailoring the structure and content of thesauri as tools for end-user searching (Bates, 1986; Strong and Drott, 1986; Anderson and Rowley, 1991; Lopez-Huertas, 1997) while others have suggested thesaurus-enhanced user interfaces to support query formulation and expansion (Pollitt et.al., 1994; Jones et.al., 1995; Beaulieu, 1997). The recent phenomenal growth of the World Wide Web has created new opportunities to introduce thesauri as information search and retrieval aids to end user communities. While the number of web-based and hypertextual thesauri continues to grow, few investigations have been carried out to evaluate the ways in which end-users interact with and make use of online thesauri for query building and expansion. The work reported here expands on a pilot study (Shiri and Revie, 2001) carried out to investigate user – thesaurus  interaction in the domains of biology and veterinary medicine.   2. Objectives The main objective of the present study is to evaluate end-users' interactions with an online thesaurus for selecting search terms for query formulation and expansion, specifically: • to study the patterns of user behaviour in thesaurus-based browsing and searching; • to provide a framework for assessing the impact of thesaurus interaction on term selection; • to explore ways of enhancing user interfaces through the use of knowledge structures embedded within thesauri.    3. Methodology The Information retrieval system The web-based interface to the CAB Abstracts database provided by Ovid Technologies was used in this study as it provided the system features required for this experiment. The system provides a thesaurus-based searching facility based on the CAB thesaurus in its advanced search mode which maps users' search terms to thesaurus descriptors. It also caters for browsing and selecting terms during the query construction process. The subjects The purpose of the study requires the participation of genuine users with real information needs. Faculty and researchers were selected from the departments of veterinary medicine and biology at Glasgow University as these two subject areas are well covered in the CAB Abstracts database.  In order to inform and validate the proposed methodology a small pilot study involving four subjects was carried out. Search requests Search requests were elicited based on information needs of the researchers prior to running the experiment. Each researcher was asked to provide three search topics of interest. This decision was made on the assumption that evaluation of search term selection can effectively be carried out only if users having genuine information requests take part in the study. Data gathering techniques Due to the complex nature of capturing data on all aspects of user-system interaction, and in particular search term selection and thesaurus interaction, this study employed a combination of data collection techniques to effectively capture qualitative as well as quantitative data. A pre-search questionnaire, screen-capturing software, the 'think-aloud' technique, and post-search interviews were all used to collect data at various stages of the users’ interaction with the system.   4. Results To identify thesaurus-based search behaviours and patterns, a number of process measures were used to quantitatively analyse all individual search process characteristics.  These measures are as follows: state, move (step), and search term. State: major stages or conditions a user goes through while conducting a search; Move (or step): characterises any action a user takes while interacting with the system; Search term: a general characterisation of all types of terms used during the entire search, provided either by the user or the system. The results obtained from the small pilot study are summarised in terms of these key process measures.  4.1 Search states and moves Five main states were defined as characterising a typical end-user search interaction through the thesaurus-enabled interface. Specific moves were identified to describe the steps and actions taken by users within these main states and these are listed in Table 1. The 12 searches conducted by the four researchers were analysed using this framework of states and moves to shed light on various aspects of browsing and searching behaviours, and the results related to moves within the search process are provided in Table 1.    Moves    S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 Total Term input (C) 2 1 1 1 3 2 5 5 2 4 4 5 35 Perform search (S) 2 1 2 1 3 2 6 6 2 5 6 6 42 Browse initial mapped terms (C) 2 1 2 2 3 2 4 4 2 4 7 5 38 Browse thesaurus (C) 2 1 2 0 2 1 2 1 3 2 5 5 26 Select terms (C) 2 1 2 2 4 2 4 6 3 4 7 5 42 Combine terms (S)  1 0 0 3 3 1 2 2 1 3 3 3 22 Continue search (S) 3 1 3 3 7 3 8 7 4 7 9 9 64 Search reformulation (C) 1 0 1 2 3 1 1 1 0 3 1 2 16 Browse titles (C) 1 0 1 1 0 1 1 2 0 1 3 4 15 View complete records (C) 0 0 1 2 1 1 1 5 1 1 2 2 17 Total moves per search 16 6 15 17 29 16 34 39 18 34 47 46 317  Table 1.  Moves of different types summarised for all searches   Moves were assigned to one of two main categories to aid the analysis of user interaction behaviour in more detail: a) “conceptual moves” (C) in which users perform some kind of conceptual analysis of terms or documents; b) moves associated with using system features (S) such as performing a search or combining  terms. Users made a total of 317 moves of which 189 (60%) were of a conceptual nature, the rest being associated with the use of system features. On average users made 26.4 moves per search. Conceptual moves can be broken down into three types i.e. moves related to entering search terms, moves dealing with thesaurus browsing and term selection or query reformulation, and moves connected with the retrieved results such as browsing titles and viewing complete records. Of the 189 conceptual moves 106 (56%) were associated with browsing the thesaurus, viewing mapped terms, and term selection. It is also evident that around one third of all moves were connected with browsing and choosing search terms from mapped and thesaurus terms. 4.2 Search term selection In total users browsed 710 descriptors, selecting 82 of these for use during the 12 searches.  Thus on average around 12% of descriptors browsed were actually selected though there was clearly significant variability, with specific term selection figures ranging from 5% to 29% amongst searches. Summarising the measures on a search-level basis indicated that on average around 3 terms were initially provided; 60 terms viewed; and ultimately around 7 terms selected for inclusion in the search.  This ‘average behaviour’ could suggest a three parameter thesaurus interaction model of the type: f (initial terms entered, terms viewed, terms ultimately selected)  where, assuming N is the number of initial terms, then the number of terms viewed and the number of terms selected would be roughly equal to 20*N and 2*N respectively.  However, these average figures obscure a wide variety of interaction behaviours with users viewing between 6 and 132 terms over the 12 different searches and ultimately selecting anything between 1 and 22 terms.  A better approach might be to compare search characteristics according to their membership of some categorical groupings, for example: few initial terms and a low level of interaction, or many initial terms and a high degree of interaction.  An attempt to formulate a more flexible model is shown in the Thesaurus Interaction Impact Matrix(TI2M) in Figure 1, which categorises searches based on the number of initial search terms used and the impact that thesaurus interaction had on the number of the terms selected. (The average number of terms viewed for each quadrant of the model is also shown).   Initial term entry  Few terms (2 or less) Many terms (more than 2)    Number of terms    remains similar Average number of  terms viewed = 29                                                           27                                         6                                     53                             Average number of                             terms viewed = 93   109                                                   73                          97                           Effect of thesaurus interaction on                                        term selection    Number of terms increased by a    factor of  1.5 or more Average number of  terms viewed = 20                                                               25                         7                                  27                                Average number of                             terms viewed = 95          74                                   80                                                132    S7 S1 S12 S2 S6 S8 S5 S10 S4 S11 S3 S9  number Search Terms viewed  Figure 1.  Thesaurus Interaction Impact Matrix  (TI2M) It can be seen that those searches where a user had a larger number of terms in mind as they came to carry out a search, resulted in on average three times as many terms being viewed. However, in the case of both the "many" and "few" initial terms categories the relative number of terms eventually selected was not related to the number of terms viewed. It is anticipated that the use of the Thesaurus Interaction Impact Matrix (TI2M) may aid the modelling of interaction, browse and navigation behaviours adopted by different users in their formulation and expansion of queries.  4.3 Users' general impressions of the thesaurus and its interface The qualitative judgements of users were also collected to evaluate the usefulness of both the thesaurus and the interface. All users stated that the thesaurus provided them with additional search terms to choose from. It was seen to be useful when the user entered a term without any confidence and the mapped term was the exact term the user was looking for. It was commented that the thesaurus would certainly provide additional search terms and options for narrowing down the search and it gives a different perspective of the same subject. The users were also asked if there were additional search terms provided by the thesaurus, which they were not aware of at the beginning of search. All users stated that there were new search terms that they would not think of at the outset. They also pointed out that the thesaurus was helpful in dealing with variant spelling and word forms.   Users were also asked to comment on the user interface to the thesaurus and its search and browse functions. Check boxes and hypertext features associated with thesaurus descriptors together with the combine search feature were mentioned by all users as easy to use and intuitive. Simple issues such as the location of the continue search command, one of the most frequently used system feature, was a problem as users search throughout the whole page to find this icon.    5. Conclusion This paper has reported a quantitative analysis framework for the evaluation of users' interaction with an online thesaurus and the ways in which it affects their search term selection behaviour. The framework suggests a basis for detailed analysis of steps and moves users adopt during a thesaurus-based search session within the context of a web-based information retrieval system. A Thesaurus Interaction Impact Matrix (TI2M) has been developed to provide an approach to modelling the interaction of different user groups with the thesaurus using quantitative analysis of the number of search terms entered, viewed and selected. A larger study is currently being undertaken to evaluate detailed user-thesaurus interaction and to find possible thesaurus browsing and searching behaviours, as well as issues affecting users' attitudes in selecting search terms for query formulation and expansion.    Acknowledgement The authors acknowledge CAB International for providing access to the Ovid CAB Abstracts database on the web as the experimental platform used within this study. References Aitchison, J., Gilchrist, A. & Bawden, D. (1997). Thesaurus construction and use: a practical manual. Aslib, London . Anderson, James D. & Rowley, Fredrick A. (1991). Building end-user thesauri from full-text. Barbara H. Kwasink and Raya Fidel. In Advances in Classification Research: Proceedings of the 2nd ASIS SIG/CR Classification Research Workshop.  27-31 October, 1991. Medford, NJ: Learned Information. Pp. 1-13. Bates, M.J. (1986). Subject access in online catalogs: a design model.  Journal of the American Society for Information Science, 37(6): 357-376. Beaulieu, M. (1997). Experiments of interfaces to support query expansion.  Journal of Documentation, 53(1): 8-19. Jones, S., Gatford, M., Hancock-Beaulieu, M., Robertson, S.E., Walker, W. & Secker, J. (1995). Interactive thesaurus navigation: intelligence rules Ok?  Journal of the American Society for Information Science, 46(1):52-59.  Lopez-Huertas, M.J. (1997). Thesaurus structure design: a conceptual approach for improved interaction.  Journal of Documentation, 53(3):139-177. Milstead, Jessica L. (1998). Use of thesauri in the full-text environment. In Cochrane, Pauline A. and Johnson, Eric H (Eds.) Visualizing Subject Access for 21st Century Information Resources: Proceedings of the 34th Annual Clinic on Library Applications of Data Processing. 2-4 March, 1997. Graduate School of Library and Information Science, University of Illinois at Urbana–Champaign. Pp. 28-38. Pollitt, A. S, Ellis, G. P., &  Smith, M. P. (1994). Improving search quality using thesauri for query specification and the presentation of search results. Knowledge Organization and Quality Management, Proceedings of the Third international ISKO Conference. June 1994. Frankfurt, Indeks Verlag. Pp.382-389. Shiri, A. A. & Revie, C. (2001). User - thesaurus interaction in a web-based database: an evaluation of users' s term selection behaviour" In Proceedings of  Infotech Oulu International Workshop on Information Retrieval. September 19-21, 2001. Oulu, Finland: University of Oulu, Machine Vision and Media Processing Unit. Pp. 23-32.   Strong, G.W. & Drott, M.C. (1986). A thesaurus for end-user indexing and retrieval.  Information Processing and Management, 22(6): 487-492. Williamson, Nancy J. (2000). Thesauri in the digital age: stability and dynamism in their development and use. Dynamism and Stability in Knowledge Organization, Proceedings of the sixth International ISKO Conference. 10-13 July, 2000. Germany: Ergon Verlag. Pp.268-274.  

A thesaurus for end-user indexing and retrieval.

Building end-user thesauri from full-text.

Experiments of interfaces to support query expansion.

Improving search quality using thesauri for query specification and the presentation of search results. Knowledge Organization and Quality Management,

Interactive thesaurus navigation: intelligence rules Ok?

Subject access in online catalogs: a design model.

Thesauri in the digital age: stability and dynamism in their development and use.

Thesaurus construction and use: a practical manual. Aslib,

Thesaurus structure design: a conceptual approach for improved interaction.

Use of thesauri in the full-text environment. In

User - thesaurus interaction in a web-based database: an evaluation of users' s term selection behaviour&quot;

Assessing the impact of user interaction with thesaural knowledge structures : a quantitative analysis framework

Shiri, Ali Asghar

Revie, Crawford

Chowdhury, Gobinda

The University of Arizona

Assessing the Impact of User Interaction with Thesaural Knowledge Structures: a Quantitative Analysis Framework

Name not available

Assessing the impact of user interaction with thesaural knowledge structures: a quantitative analysis framework

http://www.cis.strath.ac.uk/research/publications/papers/strath_cis_publication_25.pdf

Assessing the impact of user interaction with thesaural knowledge structures: a quantitative analysis framework

Abstract

Similar works

Full text

Available Versions

University of Strathclyde Institutional Repository

The University of Arizona

Name not available