On the evaluation of aggregated web search

Aggregating search results from a variety of heterogeneous sources or so-called verticals such as news, image and video into a single interface is a popular paradigm in web search. This search paradigm is commonly referred to as aggregated search. The heterogeneity of the information, the richer user interaction, and the more complex presentation strategy, make the evaluation of the aggregated search paradigm quite challenging. The Cranfield paradigm, use of test collections and evaluation measures to assess the effectiveness of information retrieval (IR) systems, is the de-facto standard evaluation strategy in the IR research community and it has its origins in work dating to the early 1960s. This thesis focuses on applying this evaluation paradigm to the context of aggregated web search, contributing to the long-term goal of a complete, reproducible and reliable evaluation methodology for aggregated search in the research community. The Cranfield paradigm for aggregated search consists of building a test collection and developing a set of evaluation metrics. In the context of aggregated search, a test collection should contain results from a set of verticals, some information needs relating to this task and a set of relevance assessments. The metrics proposed should utilize the information in the test collection in order to measure the performance of any aggregated search pages. The more complex user behavior of aggregated search should be reflected in the test collection through assessments and modeled in the metrics. Therefore, firstly, we aim to better understand the factors involved in determining relevance for aggregated search and subsequently build a reliable and reusable test collection for this task. By conducting several user studies to assess vertical relevance and creating a test collection by reusing existing test collections, we create a testbed with both the vertical-level (user orientation) and document-level relevance assessments. In addition, we analyze the relationship between both types of assessments and find that they are correlated in terms of measuring the system performance for the user. Secondly, by utilizing the created test collection, we aim to investigate how to model the aggregated search user in a principled way in order to propose reliable, intuitive and trustworthy evaluation metrics to measure the user experience. We start our investigations by studying solely evaluating one key component of aggregated search: vertical selection, i.e. selecting the relevant verticals. Then we propose a general utility-effort framework to evaluate the ultimate aggregated search pages. We demonstrate the fidelity (predictive power) of the proposed metrics by correlating them to the user preferences of aggregated search pages. Furthermore, we meta-evaluate the reliability and intuitiveness of a variety of metrics and show that our proposed aggregated search metrics are the most reliable and intuitive metrics, compared to adapted diversity-based and traditional IR metrics. To summarize, in this thesis, we mainly demonstrate the feasibility to apply the Cranfield Paradigm for aggregated search for reproducible, cheap, reliable and trustworthy evaluation

Evaluating intuitiveness of vertical-aware click models

Modeling user behavior on a search engine result page is important for understanding the users and supporting simulation experiments. As result pages become more complex, click models evolve as well in order to capture additional aspects of user behavior in response to new forms of result presentation. We propose a method for evaluating the intuitiveness of vertical-aware click models, namely the ability of a click model to capture key aspects of aggregated result pages, such as vertical selection, item selection, result presentation and vertical diversity. This method allows us to isolate model components and therefore gives a multi-faceted view on a model's performance. We argue that our method can be used in conjunction with traditional click model evaluation metrics such as log-likelihood or perplexity. In order to demonstrate the power of our method in situations where result pages can contain more than one type of vertical(e.g., Image and News) we extend the previously studied Federated Click Model such that it models user clicks on such pages. Our evaluation method yields non-trivial yet interpretable conclusions about the intuitiveness of click models, highlighting their strengths and weaknesses

Comparing Representations of Contribution Labels in Goal Models

Goal models have been proposed to be an effective method to support decision making in early requirements engineering. Key to using them is the concept of contribution links that represent how the satisfaction of one goal affects that of another. Multiple proposals have been offered for representing contribution; however, the degree to which users can intuitively understand the meaning behind contribution representations and utilize them appropriately has not been thoroughly studied. This work reports the results of an experimental study that compares the intuitiveness of two contribution representation approaches by measuring the performance of untrained users and exploring the role of individual differences (cognitive styles and arithmetic attitude and ability) in establishing the right intuition. Results show significant differences between the two representations as well as effects of various levels of individual factors. The results inspire further research on contribution links and support the operationalizability of intuitiveness as a criterion for evaluating conceptual modelling language designs

VisuaLeague III: Visual Analytics of Multiple Games

Tese de mestrado, Informática, Universidade de Lisboa, Faculdade de Ciências, 2020Digital data available has been growing over the last years and with it, the need to create representative ways to understand and make use of its potential with visualization techniques that can be applied in different purposes. One of these cases are eSports (electronic sports), considered nowadays a sport with high growth expectation, and for which data analyses can have a significant impact. One of the most popular game type practiced in eSports is the Multiplayer Online Battle Arena (MOBA) genre represented by one of the most popular competitive games, League of Legends (LoL), which will be the case study for this thesis. As many traditional sports, there are various events to have in consideration when observing performance of gameplay. In addition to statistics for each game there is relevant information on players’ positions (spatial data), in a specific period in time (temporal data). Specific events in a game, related with objectives, can also be considered, such as purchasing an item, player kills, destroying towers, or complete objectives. Having a way to analyze and visualize this data helps not only programmers and game designers to improve gameplay but also players, coaches and analysts to improve player performance. The objective of this work is to redesign the previous prototype VisuaLeague II, and propose a new version, VisuaLeague III in order to explore techniques to implement analysis for multiple games, team searches and access to professional games’ training sections, scrims. Common problems presented in the analysis with voluminous amount of data, like cluttering and overlapping, are addressed by adding filters to searches, interaction with the visualizations, aggregation of data, and clustering. The developed prototype, VisuaLeague III was evaluated by professional coaches to understand if the searches and visualization techniques implemented are adequate for analysing players’ performance in a competitive environment. The results demonstrate overall positive attitude with particular interest in analysis for custom games and multiple games analysis as those provide visualizations that do not exist in common tools, specially, regarding spatiotemporal data

X-RAY VISION: APPLICATION OF AUGMENTED REALITY IN AVIATION MAINTENANCE TO SIMPLIFY TASKS INHIBITED BY OCCLUSION

This thesis examined the potential applications of augmented or mixed reality (AR/MR) technology and leveraging them in the context of the aviation maintenance community. Specifically, we examined whether using the 3D mapping and real-time space tracking technology of devices like the Microsoft HoloLens 2 can be used to make maintenance tasks easier in environments where the maintainer is not able to see into their workspace. With the complexities of aircraft construction, the prevalence of narrow, tight fitting spaces that are blocked by walls or obstructions is common. In the past, aviation maintainers have had to rely on memorizing 2D diagrams and feeling around dark, cramped spaces in order to determine where certain parts are located. Previous research in the field of AR primarily focuses on comparing AR methods to traditional methods for different types of tasks in simulacra. There is a lack of research in the specific application of AR that addresses occlusion introduced into these tasks. By conducting trials of simulated maintenance in an occluded area using AR technology, we found that the novice maintainer increased the accuracy of performance and decreased maintenance time when compared to traditional methods, while providing a subjectively easier method for instruction.Captain, United States Marine CorpsApproved for public release. Distribution is unlimited

Digital Twins Approaches and Methods Review

© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. This is the accepted manuscript version of a conference paper which has been published in final form at https://doi.org/10.1109/ITC-Egypt58155.2023.10206196This paper investigates the recent advances in Digital Twin technologies. The aim is to compare the approaches, available open source and proprietary technologies and methods, their features, and their integration capabilities. The motivation is to enable better design decisions based on the available literature and case studies. Various tools for 3D reconstruction and visualisation, IoT and sensor integration, Physical simulations and other complete platforms provide complete solutions. A conclusion of current challenges and future work identified that the lack of standardisation and interoperability makes the lifetime of a digital twin short, with a high cost and time to build and rebuild if required