Humans optional? Automatic large-scale test collections for entity, passage, and entity-passage retrieval

Manually creating test collections is a time-, effort-, and cost-intensive process. This paper describes a fully automatic alternative for deriving large-scale test collections, where no human assessments are needed. The empirical experiments confirm that automatic test collection and manual assessments agree on the best performing systems. The collection includes relevance judgments for both text passages and knowledge base entities. Since test collections with relevance data for both entity and text passages are rare, this approach provides a cost-efficient way for training and evaluating ad hoc passage retrieval, entity retrieval, and entity-aware text retrieval methods

Strategic preferences to outsourcing IT processes and their relationships with organization's past outsourcing experience

Due to current unstable economic conditions organizations have to take every possible step to minimize their operational cost. Hence, outsourcing seems to be one of the major decisions they have to consider to reduce their cost. Due to complexity of the outsourcing process many organizations trepidation to get outsourcing decision when they don't have past outsourcing experience. It prevents an organization to take timely decision for outsourcing business processes in order to obtain advantages of outsourcing. This research explores how past outsourcing experience can relate to outsourcing process. If they don't relate with each other, organizations can directly initiate outsourcing processes with out any prior experience. Preference for outsourcing business processes depend on several factors. Two main factors are strategic importance of a process towards achieving organizational goals and maturity level of the process with in the organization. The main objective of the study is to identify how an organizations' past outsourcing experience relates with preference to outsource strategically important IT processes and matured IT processes. The aim of the outsourcing is to reduce operational cost. With the outsourcing decision organizations are trying to find solutions to several internal problems such as unavailability of required skills, excess headcount and repeated investments to operations. When one considers all these facts IT processes become the first candidate for outsourcing. Scope of this research was limited to IT processes belonging to telecommunication, finance and insurance business domains. Usages of IT within an organization differ from one organization to another. E-maturity level of an organization is determined by factors such as IT practices for day to day business operations, amount of information extract from IT systems for strategic decisions, complexity and customizability of IT systems. Models used to measure e-maturity level were identified during literature survey. Commonly used model was selected to check relationship between e-maturity level and sourcing model. Sourcing models describe options available for an organization to execute outsourcing operation. Quantitative approach was used during the research. Data gathered through questionnaire survey was analyzed using hypothesis testing. Getting precise information was a major constraint in the research because outsourcing internal processes to a third party is a confidential strategic decision. The conclusion of the hypothesis testing was that the past outsourcing experiences were not related to selecting processes to outsource. This implies organization can outsource strategically core or non-core IT process irrespective of their past outsourcing experience. Results were same for mature and immature IT processes as well. Research also had indicated that e-maturity level and sourcing models are independent from each other. Therefore organizations can execute their outsourcing plans even though they don't have any prior outsourcing experience. Key Words: Outsourcing, IT Processes, e-maturit

Evaluating non-deterministic retrieval systems

The use of sampling, randomized algorithms, or training based on the unpredictable inputs of users in Information Retrieval often leads to non-deterministic outputs. Evaluating the effectiveness of systems incorporating these methods can be challenging since each run may produce different effectiveness scores. Current IR evaluation techniques do not address this problem. Using the context of distributed information retrieval as a case study for our investigation, we propose a solution based on multivariate linear modeling. We show that the approach provides a consistent and reliable method to compare the effectiveness of non-deterministic IR algorithms, and explain how statistics can safely be used to show that two IR algorithms have equivalent effectiveness. Copyright 2014 ACM