Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrap

Monitoring machine learning models once they are deployed is challenging. It is even more challenging to decide when to retrain models in real-case scenarios when labeled data is beyond reach, and monitoring performance metrics becomes unfeasible. In this work, we use non-parametric bootstrapped uncertainty estimates and SHAP values to provide explainable uncertainty estimation as a technique that aims to monitor the deterioration of machine learning models in deployment environments, as well as determine the source of model deterioration when target labels are not available. Classical methods are purely aimed at detecting distribution shift, which can lead to false positives in the sense that the model has not deteriorated despite a shift in the data distribution. To estimate model uncertainty we construct prediction intervals using a novel bootstrap method, which improves upon the work of Kumar & Srivastava (2012). We show that both our model deterioration detection system as well as our uncertainty estimation method achieve better performance than the current state-of-the-art. Finally, we use explainable AI techniques to gain an understanding of the drivers of model deterioration. We release an open source Python package, doubt, which implements our proposed methods, as well as the code used to reproduce our experiments.Comment: 7+6 pages. Accepted at AAAI'23 Safe and Robust AI trac

Model Agnostic Explainable Selective Regression via Uncertainty Estimation

With the wide adoption of machine learning techniques, requirements have evolved beyond sheer high performance, often requiring models to be trustworthy. A common approach to increase the trustworthiness of such systems is to allow them to refrain from predicting. Such a framework is known as selective prediction. While selective prediction for classification tasks has been widely analyzed, the problem of selective regression is understudied. This paper presents a novel approach to selective regression that utilizes model-agnostic non-parametric uncertainty estimation. Our proposed framework showcases superior performance compared to state-of-the-art selective regressors, as demonstrated through comprehensive benchmarking on 69 datasets. Finally, we use explainable AI techniques to gain an understanding of the drivers behind selective regression. We implement our selective regression method in the open-source Python package doubt and release the code used to reproduce our experiments

El carácter práctico de la verdad: J. Dewey

El concepto de verdad de J. Dewey es útil para entender cómo, frente a lo que señalan sus críticos, una perspetiva pragmática sobre la verdad puede contener implicaciones normativas. En este sentido representa una vía alternativa a las teorías de la correspondencia y de la coherencia sin los límites que supone la adopción de una posición convencionalista. Desde la perspectiva de Dewey el problema de la verdad, en el marco de la capacidad humana de transformar las situaciones, es el problema de controlar la experiencia humana para hacerla más libre y significativa.Dewey’s concept of truth is useful to understand how a pragmatically approach to knowledge may contain, in opposite to the critics of pragmatism, normative implications. In this sense represents an alternative way to the correspondence’s and coherency’s theories without the limits that suppose a conventionalist position. From Dewey’s perspective, the problem of truth, in the framework of human ability of transforming the situations, is the problem of control the human experience in order to do it more meaningful and free

Dewey: el significado democrático de la primacía de los hábitos.

La relevancia que Dewey concede a los hábitos en la deliberación sirve de clave para precisar la manera en que entendió la democracia. Aunque es posible encuadrarlo dentro de los defensores de la democracia deliberativa, la importancia que da a los factores no estrictamente epistémicos hace que rechace los supuestos intelectualistas que suelen caracterizar aquellas posiciones. La consecuencia es una teoría política normativa que entiende que la democracia requiere no sólo el cultivo de las capacidades cognitivas, sino el desarrollo de un ethos que haga posible la extensión de los valores democráticos

Ética profesional y ciudadanía democrática: una aproximación pragmatista

The paper understands the rise of ethical codes and professional ethics from the point of view of their contribution to the formation of a democratic citizenship. Various aspects of professional ethics are analyzed from the perspective of an agent-based ethics for which goods, norms and virtues are complementary factors for intelligence and individual judgment development. Through a conception of democracy understood as a way of life, professional ethics acquire a renewed meaning as a central element for individual self-realization and social emancipation. In short, under the label of democratic professionalism, and following Dewey’s moral and political philosophy, it is shown that professional ethics are more than a field of application, a constituent element of a civic ethic within the framework of a democratic society.Este trabajo interpreta el auge de los códigos éticos y de la ética profesional desde el punto de vista de su contribución a la formación de una ciudadanía democrática. Se analizan diversos aspectos de la ética profesional desde la perspectiva de una ética del agente para la que bienes, normas y virtudes resultan factores complementarios para el cultivo de la inteligencia y el juicio individual. A través de una concepción de la democracia que la vincula con los hábitos y modos de vida, la ética profesional adquiere una renovada significación como elemento central para la autorrealización personal y la emancipación social. En definitiva, bajo el rótulo del profesionalismo democrático, y de la mano de la filosofía moral y política de Dewey, se muestra que la ética profesional es más que un campo de aplicación, un elemento constituyente de una ética cívica en el marco de una sociedad democrática

Quantile Encoder: Tackling High Cardinality Categorical Features in Regression Problems

Regression problems have been widely studied in machinelearning literature resulting in a plethora of regression models and performance measures. However, there are few techniques specially dedicated to solve the problem of how to incorporate categorical features to regression problems. Usually, categorical feature encoders are general enough to cover both classification and regression problems. This lack of specificity results in underperforming regression models. In this paper,we provide an in-depth analysis of how to tackle high cardinality categor-ical features with the quantile. Our proposal outperforms state-of-the-encoders, including the traditional statistical mean target encoder, when considering the Mean Absolute Error, especially in the presence of long-tailed or skewed distributions. Besides, to deal with possible overfitting when there are categories with small support, our encoder benefits from additive smoothing. Finally, we describe how to expand the encoded values by creating a set of features with different quantiles. This expanded encoder provides a more informative output about the categorical feature in question, further boosting the performance of the regression model.Comment: Accepted at The 18th International Conference on Modeling Decisions for Artificial Intelligence (MDAI

Ciudadanía democrática y ethos científico: una perspectiva pragmatista

The erosion of confidence in the epistemic capacities of citizenship has at its root the radical separation between facts and values ​​that pragmatism has considered as the key to our culture. Today it is possible to overcome dualism on the basis of a set of virtues and individual dispositions that are both ethical and epistemic. This entanglement between the ethical and the epistemic highlights the overlap between a deliberative conception of democracy with its demand for a civic ethic and the requirements of an ethics of scientific research within the framework of a democratic societyLa erosión de la confianza en las capacidades epistémicas de la ciudadanía tiene en su raíz la radical separación entre hechos y valores que el pragmatismo ha considerado como clave de nuestra cultura.  Hoy es posible superar el dualismo sobre la base de un conjunto de virtudes y disposiciones individuales que son tanto éticas como epistémicas. Este entrelazamiento entre lo ético y lo epistémico pone de manifiesto la imbricación entre una concepción deliberativa de la democracia con su exigencia de una ética cívica y los requerimientos de una ética de la investigación científica en el marco de una sociedad democrática

Demographic Parity Inspector: Fairness Audits via the Explanation Space

Even if deployed with the best intentions, machine learning methods can perpetuate, amplify or even create social biases. Measures of (un-)fairness have been proposed as a way to gauge the (non-)discriminatory nature of machine learning models. However, proxies of protected attributes causing discriminatory effects remain challenging to address. In this work, we propose a new algorithmic approach that measures group-wise demographic parity violations and allows us to inspect the causes of inter-group discrimination. Our method relies on the novel idea of measuring the dependence of a model on the protected attribute based on the explanation space, an informative space that allows for more sensitive audits than the primary space of input data or prediction distributions, and allowing for the assertion of theoretical demographic parity auditing guarantees. We provide a mathematical analysis, synthetic examples, and experimental evaluation of real-world data. We release an open-source Python package with methods, routines, and tutorials

The Role of Large Language Models in the Recognition of Territorial Sovereignty: An Analysis of the Construction of Legitimacy

We examine the potential impact of Large Language Models (LLM) on the recognition of territorial sovereignty and its legitimization. We argue that while technology tools, such as Google Maps and Large Language Models (LLM) like OpenAI's ChatGPT, are often perceived as impartial and objective, this perception is flawed, as AI algorithms reflect the biases of their designers or the data they are built on. We also stress the importance of evaluating the actions and decisions of AI and multinational companies that offer them, which play a crucial role in aspects such as legitimizing and establishing ideas in the collective imagination. Our paper highlights the case of three controversial territories: Crimea, West Bank and Transnitria, by comparing the responses of ChatGPT against Wikipedia information and United Nations resolutions. We contend that the emergence of AI-based tools like LLMs is leading to a new scenario in which emerging technology consolidates power and influences our understanding of reality. Therefore, it is crucial to monitor and analyze the role of AI in the construction of legitimacy and the recognition of territorial sovereignty.Comment: 14 page

Explanation Shift: Investigating Interactions between Models and Shifting Data Distributions

As input data distributions evolve, the predictive performance of machine learning models tends to deteriorate. In practice, new input data tend to come without target labels. Then, state-of-the-art techniques model input data distributions or model prediction distributions and try to understand issues regarding the interactions between learned models and shifting distributions. We suggest a novel approach that models how explanation characteristics shift when affected by distribution shifts. We find that the modeling of explanation shifts can be a better indicator for detecting out-of-distribution model behaviour than state-of-the-art techniques. We analyze different types of distribution shifts using synthetic examples and real-world data sets. We provide an algorithmic method that allows us to inspect the interaction between data set features and learned models and compare them to the state-of-the-art. We release our methods in an open-source Python package, as well as the code used to reproduce our experiments.Comment: arXiv admin note: text overlap with arXiv:2210.1236