Relevance in Information Systems Research

Information Systems as an academic discipline makes two contributions to society. The first, knowledge exploration, is the creation of new knowledge that is not -- and should not be -- relevant to today\u27s practitioner. The goal of knowledge exploration is to change the future, not improve the present. The second, knowledge exploitation, is the dissemination of knowledge to serve current practice (and to train future practitioners, our students). While I believe we have done a good job of knowledge exploration, I believe we need develop new vehicles to promote, nurture, and validate knowledge exploitation much like our academic cousins in Medicine, Engineering, and Computer Science

Rigour versus Relevance in Information Systems Research in South Africa

Rigour and relevance division is as a result of many reasons. The gap between the two has promoted debate and argument that has lasted for years. Many believed that IS research is effective and others opposed the argument. Others within or outside the discipline are considering whether IS research output is affecting and impacting decision making in the industry. Meanwhile, the debate on rigour and relevance has lasted for decades but in reality, the debate and the gap still persist, in spite of efforts by researchers. Their efforts and hard-work seems ineffective. The study determined whether the needs of practitioners through rigour and relevance of IS/academic research and also to determine whether this lingering debate over these decades has worth from an academic viewpoint. There is also an on-going criticism that IS research lacks rigour, relevance, effective communication and acceptance in the field as noted in the literature

RELEVANCE VERSUS RIGOR IN INFORMATION SYSTEMS RESEARCH: AN ISSUE OF QUALITY

Information Systems Working Papers Serie

The relevance of results in interpretive research in information systems and technology

The rigor and relevance of the results is central to the process of scientific investigation, even in areas where the practice prevails, as is the case of the scientific area of information systems and technology. This issue is also particularly relevant when the underlying epistemological orientation is the interpretivism. Based on a literature review focused on interpretive research in the field of information systems and technology, we find that the generalization of research resulting under the interpretive paradigm are valid and are not exclusive to the positivist orientation. This paper explores the importance of interpretative research in the information systems and technology field. As a result we discuss the different perspectives around the generalization and its interpretation in an interpretative research, supporting the investigator in the grounds of validation of their results.- (undefined

Evaluation Measures for Relevance and Credibility in Ranked Lists

Recent discussions on alternative facts, fake news, and post truth politics have motivated research on creating technologies that allow people not only to access information, but also to assess the credibility of the information presented to them by information retrieval systems. Whereas technology is in place for filtering information according to relevance and/or credibility, no single measure currently exists for evaluating the accuracy or precision (and more generally effectiveness) of both the relevance and the credibility of retrieved results. One obvious way of doing so is to measure relevance and credibility effectiveness separately, and then consolidate the two measures into one. There at least two problems with such an approach: (I) it is not certain that the same criteria are applied to the evaluation of both relevance and credibility (and applying different criteria introduces bias to the evaluation); (II) many more and richer measures exist for assessing relevance effectiveness than for assessing credibility effectiveness (hence risking further bias). Motivated by the above, we present two novel types of evaluation measures that are designed to measure the effectiveness of both relevance and credibility in ranked lists of retrieval results. Experimental evaluation on a small human-annotated dataset (that we make freely available to the research community) shows that our measures are expressive and intuitive in their interpretation

Structuration Theory in Information Systems Research: Relevance and Rigour from a Pluralist Research Approach

This paper reflects on the theoretical aspects of an earlier ontological study. The study was a single case which explored the use of an agricultural decision support system by women cotton growers in the Australian cotton industry and the effect of its use on their farm management roles on family cotton farms. The study was informed through a multi-paradigmatic conceptual framework with structuration theory as a meta-theory, and diffusion theory and gender relations theory as lower level theories. This pluralistic research approach employed both theory and data triangulation. In this paper, the justification for a multi-paradigmatic framework is discussed as well as the relevance and rigour of the study

Closing the loop: assisting archival appraisal and information retrieval in one sweep

In this article, we examine the similarities between the concept of appraisal, a process that takes place within the archives, and the concept of relevance judgement, a process fundamental to the evaluation of information retrieval systems. More specifically, we revisit selection criteria proposed as result of archival research, and work within the digital curation communities, and, compare them to relevance criteria as discussed within information retrieval's literature based discovery. We illustrate how closely these criteria relate to each other and discuss how understanding the relationships between the these disciplines could form a basis for proposing automated selection for archival processes and initiating multi-objective learning with respect to information retrieval

Combining relevance information in a synchronous collaborative information retrieval environment

Traditionally information retrieval (IR) research has focussed on a single user interaction modality, where a user searches to satisfy an information need. Recent advances in both web technologies, such as the sociable web of Web 2.0, and computer hardware, such as tabletop interface devices, have enabled multiple users to collaborate on many computer-related tasks. Due to these advances there is an increasing need to support two or more users searching together at the same time, in order to satisfy a shared information need, which we refer to as Synchronous Collaborative Information Retrieval. Synchronous Collaborative Information Retrieval (SCIR) represents a significant paradigmatic shift from traditional IR systems. In order to support an effective SCIR search, new techniques are required to coordinate users' activities. In this chapter we explore the effectiveness of a sharing of knowledge policy on a collaborating group. Sharing of knowledge refers to the process of passing relevance information across users, if one user finds items of relevance to the search task then the group should benefit in the form of improved ranked lists returned to each searcher. In order to evaluate the proposed techniques we simulate two users searching together through an incremental feedback system. The simulation assumes that users decide on an initial query with which to begin the collaborative search and proceed through the search by providing relevance judgments to the system and receiving a new ranked list. In order to populate these simulations we extract data from the interaction logs of various experimental IR systems from previous Text REtrieval Conference (TREC) workshops

Visualizing recommendations to support exploration, transparency and controllability

Research on recommender systems has traditionally focused on the development of algorithms to improve accuracy of recommendations. So far, little research has been done to enable user interaction with such systems as a basis to support exploration and control by end users. In this paper, we present our research on the use of information visualization techniques to interact with recommender systems. We investigated how information visualization can improve user understanding of the typically black-box rationale behind recommendations in order to increase their perceived relevance and meaning and to support exploration and user involvement in the recommendation process. Our study has been performed using TalkExplorer, an interactive visualization tool developed for attendees of academic conferences. The results of user studies performed at two conferences allowed us to obtain interesting insights to enhance user interfaces that integrate recommendation technology. More specifically, effectiveness and probability of item selection both increase when users are able to explore and interrelate multiple entities - i.e. items bookmarked by users, recommendations and tags. Copyright © 2013 ACM