Establishing Trustworthiness: Rethinking Tasks and Model Evaluation

Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing (NLP) community has striven to model computationally for decades. Traditionally, facets of linguistic intelligence have been compartmentalized into tasks with specialized model architectures and corresponding evaluation protocols. With the advent of large language models (LLMs) the community has witnessed a dramatic shift towards general purpose, task-agnostic approaches powered by generative models. As a consequence, the traditional compartmentalized notion of language tasks is breaking down, followed by an increasing challenge for evaluation and analysis. At the same time, LLMs are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups, increasing the need for trustworthy and reliable systems. Therefore, we argue that it is time to rethink what constitutes tasks and model evaluation in NLP, and pursue a more holistic view on language, placing trustworthiness at the center. Towards this goal, we review existing compartmentalized approaches for understanding the origins of a model's functional capacity, and provide recommendations for more multi-faceted evaluation protocols.Comment: Accepted at EMNLP 2023 (Main Conference), camera-read

Navigational Drift Analysis for Visual Odometry

Visual odometry estimates a robot's ego-motion with cameras installed on itself. With the advantages brought by camera being a sensor, visual odometry has been widely adopted in robotics and navigation fields. Drift (or error accumulation) from relative motion concatenation is an intrinsic problem of visual odometry in long-range navigation, as visual odometry is a sensor based on relative measurements. General error analysis using ``mean'' and ``covariance'' of positional error in each axis is not fully capable to describe the behavior of drift. Moreover, no theoretic drift analysis is available for performance evaluation and algorithms comparison. Drift distribution is established in the paper, as a function of the covariance matrix from positional error propagation model. To validate the drift model, experiment with a specific setting is conducted

Evaluation of a portable retinal imaging device: towards a comparative quantitative analysis for morphological measurements of retinal blood vessels

This study investigated the possibility of using low-cost, handheld, retinal imaging devices for the automatic extraction of quantifiable measures of retinal blood vessels. Initially, the available handheld devices were compared using a Zeiss model eye incorporating a USAF resolution test chart to assess their optical properties. The only suitable camera of the five evaluated was the Horus DEC 200. This device was then subjected to a detailed evaluation in which images in human eyes taken from the handheld camera were compared in a quantitative analysis with those of the same eye from a Canon CR-DGi retinal desktop camera. We found that the Horus DEC 200 exhibited shortcomings in capturing images of human eyes by comparison with the Canon. More images were rejected as being unevaluable or suffering failures in automatic segmentation than with the Canon, and even after exclusion of affected images, the Horus yielded lower measurements of vessel density than the Canon. A number of issues affecting handheld cameras in general and some features of the Horus in particular have been identified that might contribute to the observed differences in performance. Some potential mitigations are discussed which might yield improvements in performance, thus potentially facilitating use of handheld retinal imaging devices for quantitative retinal microvascular measurements

Computational Methods for Image Acquisition and Analysis with Applications in Optical Coherence Tomography

The computational approach to image acquisition and analysis plays an important role in medical imaging and optical coherence tomography (OCT). This thesis is dedicated to the development and evaluation of algorithmic solutions for better image acquisition and analysis with a focus on OCT retinal imaging. For image acquisition, we first developed, implemented, and systematically evaluated a compressive sensing approach for image/signal acquisition for single-pixel camera architectures and an OCT system. Our evaluation outcome provides a detailed insight into implementing compressive data acquisition of those imaging systems. We further proposed a convolutional neural network model, LSHR-Net, as the first deep-learning imaging solution for the single-pixel camera. This method can achieve better accuracy, hardware-efficient image acquisition and reconstruction than the conventional compressive sensing algorithm. Three image analysis methods were proposed to achieve retinal OCT image analysis with high accuracy and robustness. We first proposed a framework for healthy retinal layer segmentation. Our framework consists of several image processing algorithms specifically aimed at segmenting a total of 12 thin retinal cell layers, outperforming other segmentation methods. Furthermore, we proposed two deep-learning-based models to segment retinal oedema lesions in OCT images, with particular attention on processing small-scale datasets. The first model leverages transfer learning to implement oedema segmentation and achieves better accuracy than comparable methods. Based on the meta-learning concept, a second model was designed to be a solution for general medical image segmentation. The results of this work indicate that our model can be applied to retinal OCT images and other small-scale medical image data, such as skin cancer, demonstrated in this thesis

An In-Depth Study on Open-Set Camera Model Identification

Camera model identification refers to the problem of linking a picture to the camera model used to shoot it. As this might be an enabling factor in different forensic applications to single out possible suspects (e.g., detecting the author of child abuse or terrorist propaganda material), many accurate camera model attribution methods have been developed in the literature. One of their main drawbacks, however, is the typical closed-set assumption of the problem. This means that an investigated photograph is always assigned to one camera model within a set of known ones present during investigation, i.e., training time, and the fact that the picture can come from a completely unrelated camera model during actual testing is usually ignored. Under realistic conditions, it is not possible to assume that every picture under analysis belongs to one of the available camera models. To deal with this issue, in this paper, we present the first in-depth study on the possibility of solving the camera model identification problem in open-set scenarios. Given a photograph, we aim at detecting whether it comes from one of the known camera models of interest or from an unknown one. We compare different feature extraction algorithms and classifiers specially targeting open-set recognition. We also evaluate possible open-set training protocols that can be applied along with any open-set classifier, observing that a simple of those alternatives obtains best results. Thorough testing on independent datasets shows that it is possible to leverage a recently proposed convolutional neural network as feature extractor paired with a properly trained open-set classifier aiming at solving the open-set camera model attribution problem even to small-scale image patches, improving over state-of-the-art available solutions.Comment: Published through IEEE Access journa

Oriental oculopalpebral dimensions: Quantitative comparison between Orientals from Japan and Brazil

Rodrigo U Takahagi1, Silvana A Schellini1, Carlos R Padovani1, Shinji Ideta2, Nobutada Katori2, Yasuhisa Nakamura21Department of Ophthalmology, Faculdade de Medicina de Botucatu, Botucatu, Sao Paulo State, Brazil; 2Department of Oculoplastic and Orbital Surgery, Hamamatsu Seirei General Hospital, Hamamatsu, Shizuoka-ken, JapanObjectives: Quantitative evaluation of palpebral dimensions of Japanese residents in Japan and Japanese descendant (Nikkeis) who live in Brazil, in order to define if environmental factors may influence these parameters.Methods: A prospective study evaluating 107 Nikkeis from Brazil and 114 Japanese residents in Japan, aged 20 years or older. Exclusion criteria were those with palpebral position alterations, prior palpebral surgery, and crossbreeding. Images were obtained with a digital camera, 30 cm from the frontal plane at pupil height, with the individual in a primary position and the eye trained on the camera lens. Images were transferred to computer and processed by the Scion Image program. Measurements were made of distance between medial canthi, distance between pupils (IPD), superior eyelid crease position, distance between the superior lid margin and corneal reflexes (MRD), horizontal width, height, area, and obliquity of the palpebral fissure. Data were analyzed using analysis of variance for a three factor model and respective multiple comparison tests.Results: Japanese residents and Nikkeis living in Brazil have similar measurements. Statistical differences were found for some age groups concerning distance between pupils, horizontal, and vertical fissures, palpebral fissure area, and obliquity with native Japanese presenting discretely higher measurements than Nikkeis.Conclusion: Environmental factors do not affect palpebral dimensions of Nikkeis living in Brazil.Keywords: eyelid dimensions, Japanese, Nikkeis, digital imag

On Offline Evaluation of Vision-based Driving Models

Autonomous driving models should ideally be evaluated by deploying them on a fleet of physical vehicles in the real world. Unfortunately, this approach is not practical for the vast majority of researchers. An attractive alternative is to evaluate models offline, on a pre-collected validation dataset with ground truth annotation. In this paper, we investigate the relation between various online and offline metrics for evaluation of autonomous driving models. We find that offline prediction error is not necessarily correlated with driving quality, and two models with identical prediction error can differ dramatically in their driving performance. We show that the correlation of offline evaluation with driving quality can be significantly improved by selecting an appropriate validation dataset and suitable offline metrics. The supplementary video can be viewed at https://www.youtube.com/watch?v=P8K8Z-iF0cYComment: Published at the ECCV 2018 conferenc

Beyond standard benchmarks: Parameterizing performance evaluation in visual object tracking

Object-to-camera motion produces a variety of apparent motion patterns that significantly affect performance of short-term visual trackers. Despite being crucial for designing robust trackers, their influence is poorly explored in standard benchmarks due to weakly defined, biased and overlapping attribute annotations. In this paper we propose to go beyond pre-recorded benchmarks with post-hoc annotations by presenting an approach that utilizes omnidirectional videos to generate realistic, consistently annotated, short-term tracking scenarios with exactly parameterized motion patterns. We have created an evaluation system, constructed a fully annotated dataset of omnidirectional videos and the generators for typical motion patterns. We provide an in-depth analysis of major tracking paradigms which is complementary to the standard benchmarks and confirms the expressiveness of our evaluation approach