Evaluating epistemic uncertainty under incomplete assessments

The thesis of this study is to propose an extended methodology for laboratory based Information Retrieval evaluation under incomplete relevance assessments. This new methodology aims to identify potential uncertainty during system comparison that may result from incompleteness. The adoption of this methodology is advantageous, because the detection of epistemic uncertainty - the amount of knowledge (or ignorance) we have about the estimate of a system's performance - during the evaluation process can guide and direct researchers when evaluating new systems over existing and future test collections. Across a series of experiments we demonstrate how this methodology can lead towards a finer grained analysis of systems. In particular, we show through experimentation how the current practice in Information Retrieval evaluation of using a measurement depth larger than the pooling depth increases uncertainty during system comparison

Performance evaluation of phase-angle gradient method for phase retrieval based on low-frequency amplitude-only near-field data

The Phase Angle Gradient Method (PAGM) is a recent technique developed for phase retrieval based on amplitude-only measurement data. Preliminary results have shown that the PAGM is able to perform phase retrieval at 100MHz with accurate phase information based on measured fIeld components on three planar surfaces. In this paper, a performance evaluation of the PAGM under different conFigurations is conducted. Phase retrieval based on field measurements for different plane sizes and separations between the planes are studied rigorously. In addition, the PAGM is tested for different initial phase distributions. The results show that the PAGM is capable of retrieving phase information even if the separation between the measurement planes is small in terms of wavelengths

Evaluating the quality of library portals

To investigate ways of demonstrating how portal implementations positively alter user information retrieval behaviour. Design/methodology/approach - An opinion piece reflecting on existing evidence about the nature of portal implementations, which extrapolates trends in user behaviour on the basis of these reflections. Findings - Although portal technologies probably do offer a way for libraries to create information tools that can compete with "one-stop shop" Internet search engines, there are likely difficulties in their pattern of usage which will have to be detected by effective quality measurement techniques. Research limitations/implications - An expression of opinion about the possible pitfalls of using portals to optimise users' information retrieval activity. Practical implications - This opinion piece gives some clear and practical guidelines for the evaluation of the success of library portal implementations. Originality/value - This editorial points out that, because the portal can be defined as a deliberate clone of a typical successful Internet search engine and may be presented to the naïve user in the same terms, the danger is that library portals might also clone the same information habits as Internet search engines, because of their ease of use. In trying to produce a tool that can meet Google on its own terms but with better content, we might reproduce some of the same educational disbenefits as Google: quality information retrieval is not purely a function of content, it is also a function of the user's perceptions and information habits

Retrieval interval mapping: a tool to visualize the impact of the spectral retrieval range on differential optical absorption spectroscopy evaluations

Remote sensing via differential optical absorption spectroscopy (DOAS) has become a standard technique to identify and quantify trace gases in the atmosphere. Due to the wide range of measurement conditions, atmospheric compositions and instruments used, a specific challenge of a DOAS retrieval is to optimize the retrieval parameters for each specific case and particular trace gas of interest. Of these parameters, the retrieval wavelength range is one of the most important ones. Although for many trace gases the overall dependence of common DOAS retrieval on the evaluation wavelength interval is known, a systematic approach for finding the optimal retrieval wavelength range and quantitative assessment is missing. Here we present a novel tool to visualize the effect of different evaluation wavelength ranges. It is based on mapping retrieved column densities in the retrieval wavelength space and thus visualizing the consequences of different choices of spectral retrieval ranges caused by slightly erroneous absorption cross sections, cross correlations and instrumental features. Based on the information gathered, an optimal retrieval wavelength range may be determined systematically

Dynamic Thresholding Mechanisms for IR-Based Filtering in Efficient Source Code Plagiarism Detection

To solve time inefficiency issue, only potential pairs are compared in string-matching-based source code plagiarism detection; wherein potentiality is defined through a fast-yet-order-insensitive similarity measurement (adapted from Information Retrieval) and only pairs which similarity degrees are higher or equal to a particular threshold is selected. Defining such threshold is not a trivial task considering the threshold should lead to high efficiency improvement and low effectiveness reduction (if it is unavoidable). This paper proposes two thresholding mechanisms---namely range-based and pair-count-based mechanism---that dynamically tune the threshold based on the distribution of resulted similarity degrees. According to our evaluation, both mechanisms are more practical to be used than manual threshold assignment since they are more proportional to efficiency improvement and effectiveness reduction.Comment: The 2018 International Conference on Advanced Computer Science and Information Systems (ICACSIS

An Integrated Approach for Characterizing Aerosol Climate Impacts and Environmental Interactions

Aerosols exert myriad influences on the earth's environment and climate, and on human health. The complexity of aerosol-related processes requires that information gathered to improve our understanding of climate change must originate from multiple sources, and that effective strategies for data integration need to be established. While a vast array of observed and modeled data are becoming available, the aerosol research community currently lacks the necessary tools and infrastructure to reap maximum scientific benefit from these data. Spatial and temporal sampling differences among a diverse set of sensors, nonuniform data qualities, aerosol mesoscale variabilities, and difficulties in separating cloud effects are some of the challenges that need to be addressed. Maximizing the long-term benefit from these data also requires maintaining consistently well-understood accuracies as measurement approaches evolve and improve. Achieving a comprehensive understanding of how aerosol physical, chemical, and radiative processes impact the earth system can be achieved only through a multidisciplinary, inter-agency, and international initiative capable of dealing with these issues. A systematic approach, capitalizing on modern measurement and modeling techniques, geospatial statistics methodologies, and high-performance information technologies, can provide the necessary machinery to support this objective. We outline a framework for integrating and interpreting observations and models, and establishing an accurate, consistent, and cohesive long-term record, following a strategy whereby information and tools of progressively greater sophistication are incorporated as problems of increasing complexity are tackled. This concept is named the Progressive Aerosol Retrieval and Assimilation Global Observing Network (PARAGON). To encompass the breadth of the effort required, we present a set of recommendations dealing with data interoperability; measurement and model integration; multisensor synergy; data summarization and mining; model evaluation; calibration and validation; augmentation of surface and in situ measurements; advances in passive and active remote sensing; and design of satellite missions. Without an initiative of this nature, the scientific and policy communities will continue to struggle with understanding the quantitative impact of complex aerosol processes on regional and global climate change and air quality

Towards open corpus adaptive hypermedia: A study of novelty detection approaches

Classic adaptive hypermedia systems are able to track a user's knowledge of the subject and use it to evaluate the novelty and difficulty of content encountered by the user. Our goal is to implement this functionality in an open corpus context where a domain model is not available nor is the content indexed with domain concepts. We examine methods for novelty measurement based on automatic text analysis. To compare these methods, we use an evaluation approach based on knowledge encapsulated in the structure of a textbook. Our study shows that a knowledge accumulation method adopted from the domain of intelligent tutoring systems offers a more meaningful novelty measurement than methods adapted from the area of personalized information retrieval. © 2011 Springer-Verlag

A Theory for the Measurement of Internet Information Retrieval

The purpose of this study was to develop and evaluate a measurement model for Internet information retrieval strategy performance evaluation whose theoretical basis is a modification of the classical measurement model embodied in the Cranfield studies and their progeny. Though not the first, the Cranfield studies were the most influential of the early evaluation experiments. The general problem with this model was and continues to be the subjectivity of the concept of relevance. In cyberspace, information scientists are using quantitative measurement models for evaluating information retrieval performance that are based on the Cranfield model. This research modified this model by incorporating enduser relevance judgment rather than using objective relevance judgments, and by adopting a fundamental unit of measure developed for the cyberspace of Internet information retrieval rather than using recall and precision-type measures. The proposed measure, the Content-bearing Click (CBC) Ratio, was developed as a quantitative measure reflecting the performance of an Internet IR strategy. Since the hypertext "click" is common to many Internet IR strategies, it was chosen as the fundamental unit of measure rather than the "document." The CBC Ratio is a ratio of hypertext click counts that can be viewed as a false drop measure that determines the average number of irrelevant content-bearing clicks that an enduser check before retrieving relevant information. After measurement data were collected, they were used to evaluate the reliability of several methods for aggregating relevance judgments. After reliability coefficients were calculated, measurement model was used to compare web catalog and web database performance in an experimental setting. Conclusions were the reached concerning the reliability of the proposed measurement model and its ability to measure Internet IR performance, as well as implications for clinical use of the Internet and for future research in Information Science