Cheap IR Evaluation: Fewer Topics, No Relevance Judgements, and Crowdsourced Assessments

To evaluate Information Retrieval (IR) effectiveness, a possible approach is to use test collections, which are composed of a collection of documents, a set of description of information needs (called topics), and a set of relevant documents to each topic. Test collections are modelled in a competition scenario: for example, in the well known TREC initiative, participants run their own retrieval systems over a set of topics and they provide a ranked list of retrieved documents; some of the retrieved documents (usually the first ranked) constitute the so called pool, and their relevance is evaluated by human assessors; the document list is then used to compute effectiveness metrics and rank the participant systems. Private Web Search companies also run their in-house evaluation exercises; although the details are mostly unknown, and the aims are somehow different, the overall approach shares several issues with the test collection approach. The aim of this work is to: (i) develop and improve some state-of-the-art work on the evaluation of IR effectiveness while saving resources, and (ii) propose a novel, more principled and engineered, overall approach to test collection based effectiveness evaluation. [...

Improving neural networks by preventing co-adaptation of feature detectors

When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition

BioSecure: white paper for research in biometrics beyond BioSecure

This report is the output of a consultation process of various major stakeholders in the biometric community to identify the future biometrical research issues, an activity which employed not only researchers but representatives from the entire biometrical community, consisting of governments, industry, citizens and academia. It is one of the main efforts of the BioSecure Network of Excellence to define the agenda for future biometrical research, including systems and applications scenarios

A CASE STUDY ON THE EFFECT OF NEWS ON CRUDE OIL PRICE

Crude oil price volatility has an impact on the global economy and oil-dependent industries and is influenced by supply and demand, geopolitical tensions, and the global economy. Every day, a massive amount of textual information flows in the form of news articles, which humans use to forecast future trends. News articles can have a significant impact on the price of crude oil because they contain information about recent events, trends, and advancements in the industry. The purpose of this work is to investigate how news articles may affect crude oil prices, using the concept of topic modeling and its potential for handling data. Using the webscraping method, the data for the study comes from a large dataset of news articles about the crude oil industry. These news articles were published between January 1 and December 31, 2022, and come from four different sources. The data was compiled using the source Exchange Rates UK to demonstrate how the price of crude oil fluctuated during this period. After the cleaning process was completed, the dataset contained a total of 1532 news articles. The Latent Dirichlet Allocation (LDA) technique is suggested for extracting relevant keywords from news articles and then using the findings as input features to forecast the crude oil price. The forecasting methods employed in the study were the Ridge model, the Random Forest and XGBoost techniques, and the time series method ARIMAX. The outcomes of the experiment indicate that the association between the meaning of the news articles and the crude oil price is not sufficiently strong. It is additionally concluded that the XGBoost algorithm reveal superior predictive performance in the training set. As a result, XGBoost models for each month of 2022 were developed to investigate the impact of features and determine the most important ones for the problem.A volatilidade dos preços do petróleo bruto tem um impacto na economia global e nas indústrias dependentes do petróleo e é influenciada pela oferta e procura, por tensões geopolíticas e pela economia global. Todos os dias uma enorme quantidade de informação flui sob a forma de artigos de notícias e é utilizada pelo ser humano para prever tendências futuras. Os artigos de notícias podem influenciar significativamente o preço do petróleo bruto porque contêm informação sobre eventos recentes, tendências e avanços na indústria. O objetivo deste trabalho é investigar como os artigos de notícias podem afetar os preços do petróleo bruto, utilizando o conceito de modelação de tópicos. Utilizando o método web-scraping, os dados para o estudo provêm de um grande conjunto de artigos de notícias sobre a indústria do petróleo bruto. Estes artigos foram publicados entre 1 de janeiro e 31 de dezembro de 2022 e resultam de quatro fontes diferentes. Os dados foram compilados usando a fonte Exchange Rates UK para demonstrar como o preço do petróleo bruto flutuou ao longo deste período. Após a conclusão do processo de limpeza, obteve-se um total de 1532 artigos de notícias. A técnica Latent Dirichlet Allocation (LDA) é sugerida para extrair as palavras-chave pertinentes dos artigos de notícias. Os seus resultados foram depois utilizados como variáveis de entrada para prever o preço do petróleo bruto. Os métodos de previsão utilizados no estudo foram os modelos Ridge, Random Forest, XGBoost e ARIMAX. Os resultados indicam que a relação entre os artigos de notícias e o preço do petróleo bruto não é suficientemente forte. Conclui-se que o algoritmo XGBoost revela um desempenho preditivo superior no conjunto de treino. Como resultado, foram desenvolvidos modelos XGBoost para cada mês de 2022 para investigar o impacto das características e determinar as mais importantes para o problema

Information Retrieval with Finnish Case Law Embeddings

In this work, five text vectorisation models' capability in embedding Finnish case law texts to vector space for inter-textual similarity computation is studied. The embeddings and their computed similarities are used to create a Finnish case law retrieval system that allows effective querying with full documents. A working web application is presented as a part of the work. The case law data for the work is provided by the Finnish Ministry of Justice, and the studied models are: TF-IDF, LDA, Word2Vec, Doc2Vec and Doc2vecC

Improving Public Services by Mining Citizen Feedback: An Application of Natural Language Processing

Research on user satisfaction has increased substantially in recent years. To date, most studies have tested the significance of predefined factors thought to influence user satisfaction, with no scalable means of verifying the validity of their assumptions. Digital technology has created new methods of collecting user feedback where service users post comments. As topic models can analyse large volumes of feedback, they have been proposed as a feasible approach to aggregating user opinions. This novel approach has been applied to process reviews of primary care practices in England. Findings from an analysis of more than 200,000 reviews show that the quality of interactions with staff and bureaucratic exigencies are the key drivers of user satisfaction. In addition, patient satisfaction is strongly influenced by factors that are not measured by state‐of‐the‐art patient surveys. These results highlight the potential benefits of text mining and machine learning for public administration

Evaluating embodied conversational agents in multimodal interfaces

Based on cross-disciplinary approaches to Embodied Conversational Agents, evaluation methods for such human-computer interfaces are structured and presented. An introductory systematisation of evaluation topics from a conversational perspective is followed by an explanation of social-psychological phenomena studied in interaction with Embodied Conversational Agents, and how these can be used for evaluation purposes. Major evaluation concepts and appropriate assessment instruments – established and new ones – are presented, including questionnaires, annotations and log-files. An exemplary evaluation and guidelines provide hands-on information on planning and preparing such endeavours

Combining implicit and explicit topic representations for result diversification

Result diversification deals with ambiguous or multi-faceted queries by providing documents that cover as many subtopics of a query as possible. Various approaches to subtopic modeling have been proposed. Subtopics have been extracted internally, e.g., from retrieved documents, and externally, e.g., from Web resources such as query logs. Internally modeled subtopics are often implicitly represented, e.g., as latent topics, while externally modeled subtopics are often explicitly represented, e.g., as reformulated queries. We propose a framework that: i) combines both implicitly and explicitly represented subtopics; and ii) allows flexible combination of multiple external resources in a transparent and unified manner. Specifically, we use a random walk based approach to estimate the similarities of the explicit subtopics mined from a number of heterogeneous resources: click logs, anchor text, and web n-grams. We then use these similarities to regularize the latent topics extracted from the top-ranked documents, i.e., the internal (implicit) subtopics. Empirical results show that regularization with explicit subtopics extracted from the right resource leads to improved diversification results, indicating that the proposed regularization with (explicit) external resources forms better (implicit) topic models. Click logs and anchor text are shown to be more effective resources than web n-grams under current experimental settings. Combining resources does not always lead to better results, but achieves a robust performance. This robustness is important for two reasons: it cannot be predicted which resources will be most effective for a given query, and it is not yet known how to reliably determine the optimal model parameters for building implicit topic models