On Fodor on Darwin on Evolution

Jerry Fodor argues that Darwin was wrong about "natural selection" because (1) it is only a tautology rather than a scientific law that can support counterfactuals ("If X had happened, Y would have happened") and because (2) only minds can select. Hence Darwin's analogy with "artificial selection" by animal breeders was misleading and evolutionary explanation is nothing but post-hoc historical narrative. I argue that Darwin was right on all counts. Until Darwin's "tautology," it had been believed that either (a) God had created all organisms as they are, or (b) organisms had always been as they are. Darwin revealed instead that (c) organisms have heritable traits that evolved across time through random variation, with survival and reproduction in (changing) environments determining (mindlessly) which variants were successfully transmitted to the next generation. This not only provided the (true) alternative (c), but also the methodology for investigating which traits had been adaptive, how and why; it also led to the discovery of the genetic mechanism of the encoding, variation and evolution of heritable traits. Fodor also draws erroneous conclusions from the analogy between Darwinian evolution and Skinnerian reinforcement learning. Fodor’s skepticism about both evolution and learning may be motivated by an overgeneralization of Chomsky’s “poverty of the stimulus argument” -- from the origin of Universal Grammar (UG) to the origin of the “concepts” underlying word meaning, which, Fodor thinks, must be “endogenous,” rather than evolved or learned

On Fodor on Darwin on Evolution

Jerry Fodor argues that Darwin was wrong about "natural selection" because (1) it is only a tautology rather than a scientific law that can support counterfactuals ("If X had happened, Y would have happened") and because (2) only minds can select. Hence Darwin's analogy with "artificial selection" by animal breeders was misleading and evolutionary explanation is nothing but post-hoc historical narrative. I argue that Darwin was right on all counts. Until Darwin's "tautology," it had been believed that either (a) God had created all organisms as they are, or (b) organisms had always been as they are. Darwin revealed instead that (c) organisms have heritable traits that evolved across time through random variation, with survival and reproduction in (changing) environments determining (mindlessly) which variants were successfully transmitted to the next generation. This not only provided the (true) alternative (c), but also the methodology for investigating which traits had been adaptive, how and why; it also led to the discovery of the genetic mechanism of the encoding, variation and evolution of heritable traits. Fodor also draws erroneous conclusions from the analogy between Darwinian evolution and Skinnerian reinforcement learning. Fodor's skepticism about both evolution and learning may be motivated by an overgeneralization of Chomsky's "poverty of the stimulus argument" -- from the origin of Universal Grammar (UG) to the origin of the "concepts" underlying word meaning, which, Fodor thinks, must be "endogenous," rather than evolved or learned

The Search for Invariance: Repeated Positive Testing Serves the Goals of Causal Learning

Positive testing is characteristic of exploratory behavior, yet it seems to be at odds with the aim of information seeking. After all, repeated demonstrations of one’s current hypothesis often produce the same evidence and fail to distinguish it from potential alternatives. Research on the development of scientific reasoning and adult rule learning have both documented and attempted to explain this behavior. The current chapter reviews this prior work and introduces a novel theoretical account—the Search for Invariance (SI) hypothesis—which suggests that producing multiple positive examples serves the goals of causal learning. This hypothesis draws on the interventionist framework of causal reasoning, which suggests that causal learners are concerned with the invariance of candidate hypotheses. In a probabilistic and interdependent causal world, our primary goal is to determine whether, and in what contexts, our causal hypotheses provide accurate foundations for inference and intervention—not to disconfirm their alternatives. By recognizing the central role of invariance in causal learning, the phenomenon of positive testing may be reinterpreted as a rational information-seeking strategy

Causal Reinforcement Learning: A Survey

Reinforcement learning is an essential paradigm for solving sequential decision problems under uncertainty. Despite many remarkable achievements in recent decades, applying reinforcement learning methods in the real world remains challenging. One of the main obstacles is that reinforcement learning agents lack a fundamental understanding of the world and must therefore learn from scratch through numerous trial-and-error interactions. They may also face challenges in providing explanations for their decisions and generalizing the acquired knowledge. Causality, however, offers a notable advantage as it can formalize knowledge in a systematic manner and leverage invariance for effective knowledge transfer. This has led to the emergence of causal reinforcement learning, a subfield of reinforcement learning that seeks to enhance existing algorithms by incorporating causal relationships into the learning process. In this survey, we comprehensively review the literature on causal reinforcement learning. We first introduce the basic concepts of causality and reinforcement learning, and then explain how causality can address core challenges in non-causal reinforcement learning. We categorize and systematically review existing causal reinforcement learning approaches based on their target problems and methodologies. Finally, we outline open issues and future directions in this emerging field.Comment: 48 pages, 10 figure

Intrinsically Motivated Reinforcement Learning based Recommendation with Counterfactual Data Augmentation

Deep reinforcement learning (DRL) has been proven its efficiency in capturing users' dynamic interests in recent literature. However, training a DRL agent is challenging, because of the sparse environment in recommender systems (RS), DRL agents could spend times either exploring informative user-item interaction trajectories or using existing trajectories for policy learning. It is also known as the exploration and exploitation trade-off which affects the recommendation performance significantly when the environment is sparse. It is more challenging to balance the exploration and exploitation in DRL RS where RS agent need to deeply explore the informative trajectories and exploit them efficiently in the context of recommender systems. As a step to address this issue, We design a novel intrinsically ,otivated reinforcement learning method to increase the capability of exploring informative interaction trajectories in the sparse environment, which are further enriched via a counterfactual augmentation strategy for more efficient exploitation. The extensive experiments on six offline datasets and three online simulation platforms demonstrate the superiority of our model to a set of existing state-of-the-art methods

Explainable reinforcement learning for broad-XAI: a conceptual framework and survey

Broad-XAI moves away from interpreting individual decisions based on a single datum and aims to provide integrated explanations from multiple machine learning algorithms into a coherent explanation of an agent’s behaviour that is aligned to the communication needs of the explainee. Reinforcement Learning (RL) methods, we propose, provide a potential backbone for the cognitive model required for the development of Broad-XAI. RL represents a suite of approaches that have had increasing success in solving a range of sequential decision-making problems. However, these algorithms operate as black-box problem solvers, where they obfuscate their decision-making policy through a complex array of values and functions. EXplainable RL (XRL) aims to develop techniques to extract concepts from the agent’s: perception of the environment; intrinsic/extrinsic motivations/beliefs; Q-values, goals and objectives. This paper aims to introduce the Causal XRL Framework (CXF), that unifies the current XRL research and uses RL as a backbone to the development of Broad-XAI. CXF is designed to incorporate many standard RL extensions and integrated with external ontologies and communication facilities so that the agent can answer questions that explain outcomes its decisions. This paper aims to: establish XRL as a distinct branch of XAI; introduce a conceptual framework for XRL; review existing approaches explaining agent behaviour; and identify opportunities for future research. Finally, this paper discusses how additional information can be extracted and ultimately integrated into models of communication, facilitating the development of Broad-XAI. © 2023, The Author(s)

An Ecological and Longitudinal Perspective

Von der Entscheidung für ein Spiel bis zur Wahl einer Taktik, um die Schlafenszeit hinauszuzögern - wiederholte Entscheidungen sind für Kinder allgegenwärtig. Zwei paradigmatische Entscheidungsphänomene sind probability matching (dt. Angleichen der Wahrscheinlichkeit) und Maximieren. Um Belohnungen zu maximieren, sollte eine Person ausschließlich die Option auswählen, welche die höchste Wahrscheinlichkeit hat. Maximieren wird allgemein al ökonomisch rationales Verhalten angesehen. Probability matching beschreibt, dass eine Person jede Option mit der Wahrscheinlichkeit auswählt, wie deren zugrunde liegende Wahrscheinlichkeit einer Belohnung ist. Ob es sich bei probability matching um einen Fehlschluss oder einen adaptiven Mechanismus handelt, ist umstritten. Frühere Forschung zu probabilistischem Lernen zeigte das paradoxe Ergebnis, dass jüngere Kinder eher maximieren als ältere Kinder. Von älteren Kindern nimmt man hingegen an, dass sie probability matchen. Dabei wurde jedoch kaum berücksichtigt, dass Kinder die Struktur der Umwelt zu ihrem Vorteil nutzen können. Diese Dissertation untersucht die inter- und intraindividuelle Entwicklung des probabilistischen Lernens in der Kindheit unter ökologischen und kognitiven Aspekten. Vier empirischen Kapitel zeigen, dass die Interaktion zwischen heranreifenden kognitiven Funktionen, sowie Merkmalen der Lern- und Entscheidungsumgebung die Entwicklung des adaptiven Entscheidungsverhaltens prägt. Die Entwicklung des probabilistischen Lernens durchläuft in der Kindheit mehrere Phasen: von hoher Persistenz, aber auch hoher interindividueller Variabilität bei jüngeren Kindern zu wachsender Anpassungsfähigkeit durch zunehmende Diversifizierung und Exploration bei älteren Kindern. Die Ergebnisse dieser Dissertation unterstreichen insbesondere den Nutzen einer ökologischen Rationalitätsperspektive bei der Erforschung der Entwicklung des Entscheidungsvermögens.From choosing which game to play to deciding how to effectively delay bedtime—making repeated choices is a ubiquitous part of childhood. Two often contrasted paradigmatic choice behaviors are probability matching and maximizing. Maximizing, described as consistently choosing the option with the highest reward probability, has traditionally been considered economically rational. Probability matching, in contrast, described by proportionately matching choices to underlying reward probabilities, is debated whether it reflects a mistake or an adaptive mechanism. Previous research on the development of probability learning and repeated choice revealed considerable change across childhood and reported the paradoxical finding that younger children are more likely to maximize—outperforming older children who are thought to be more likely to probability match. However, this line of research largely disregarded the mind’s ability to capitalize on the structure of the environment. In this dissertation, I investigate the inter- and intra-individual development of probability learning and repeated choice behavior in childhood under consideration of ecological, cognitive, and methodological aspects. Four empirical chapters demonstrate that the interaction between the maturing mind and characteristics of the learning and choice environment shapes the development of adaptive choice behavior. The development of probability learning and repeated choice behavior in childhood progresses from high persistence but also high inter-individual variability to emerging adaptivity marked by increased diversification and exploration. The present research highlights the benefit of taking an ecological rationality view in research on the development of decision making abilities

Boundedly rational versus optimization-based models of strategic thinking and learning in games

The paper is a comment on the article by R. Harstad and R. Selten and considers the tradeoff between bounded rationality and optimization models in the game-theoretic context. The author shows that in most of the models elements of opimization are still retained and that it is thus more productive to further improve the optimization-based modeling rather than to abandon them altogether in favour of bounded rationality

Counterfactual Explanation for Fairness in Recommendation

Fairness-aware recommendation eliminates discrimination issues to build trustworthy recommendation systems.Explaining the causes of unfair recommendations is critical, as it promotes fairness diagnostics, and thus secures users' trust in recommendation models. Existing fairness explanation methods suffer high computation burdens due to the large-scale search space and the greedy nature of the explanation search process. Besides, they perform score-based optimizations with continuous values, which are not applicable to discrete attributes such as gender and race. In this work, we adopt the novel paradigm of counterfactual explanation from causal inference to explore how minimal alterations in explanations change model fairness, to abandon the greedy search for explanations. We use real-world attributes from Heterogeneous Information Networks (HINs) to empower counterfactual reasoning on discrete attributes. We propose a novel Counterfactual Explanation for Fairness (CFairER) that generates attribute-level counterfactual explanations from HINs for recommendation fairness. Our CFairER conducts off-policy reinforcement learning to seek high-quality counterfactual explanations, with an attentive action pruning reducing the search space of candidate counterfactuals. The counterfactual explanations help to provide rational and proximate explanations for model fairness, while the attentive action pruning narrows the search space of attributes. Extensive experiments demonstrate our proposed model can generate faithful explanations while maintaining favorable recommendation performance