Chasing Unknown Bandits: Uncertainty Guidance in Learning and Decision Making

In repeated decision problems for which it is possible to learn from experience, people should actively seek out uncertain options, rather than avoid ambiguity or uncertainty, in order to learn and improve future decisions. Research on human behavior in a variety of multiarmed-bandit tasks supports this prediction. Multiarmed-bandit tasks involve repeated decisions between options with initially unknown reward distributions and require a careful balance between learning about relatively unknown options (exploration) and obtaining high immediate rewards (exploitation). Resolving this exploration-exploitation dilemma optimally requires considering not only the estimated value of each option, but also the uncertainty in these estimations. Bayesian learning naturally quantifies uncertainty and hence provides a principled framework to study how humans resolve this dilemma. On the basis of computational modeling and behavioral results in bandit tasks, I argue that human learning, attention, and exploration are guided by uncertainty. These results support Bayesian theories of cognition and underpin the fundamental role of subjective uncertainty in both learning and decision making

Simple trees in complex forests: Growing Take The Best by Approximate Bayesian Computation

How can heuristic strategies emerge from smaller building blocks? We propose Approximate Bayesian Computation as a computational solution to this problem. As a first proof of concept, we demonstrate how a heuristic decision strategy such as Take The Best (TTB) can be learned from smaller, probabilistically updated building blocks. Based on a self-reinforcing sampling scheme, different building blocks are combined and, over time, tree-like non-compensatory heuristics emerge. This new algorithm, coined Approximately Bayesian Computed Take The Best (ABC-TTB), is able to recover a data set that was generated by TTB, leads to sensible inferences about cue importance and cue directions, can outperform traditional TTB, and allows to trade-off performance and computational effort explicitly

Follow my example, for better and for worse: The influence of behavioral traces on recycling decisions

Recycling behavior can recover valuable materials and mitigate green house gas emissions from landfills and incinerators. The potential positive impact of individuals' recycling behavior depends on others also making an effort, for instance, avoiding contamination. Knowing what other people have done may therefore influence recycling behavior. Behavioral traces are evidence of other people's behavior in a shared environment. Here, they relate to waste items already placed in one of two bins, a mixed recycling bin and a nonrecyclable waste bin. In two online experiments and one real-life intervention study, we investigate the role of behavioral traces on the willingness to recycle as well as the correctness of recycling. We find that seeing behavioral traces of previous recycling behavior makes recycling generally more likely, and people tend to copy item placement. This in turn increases correctness in groups where the average individual has good knowledge of recycling. Introducing correct items at the start of the day in the intervention study did not increase correctness, possibly because the correct items were soon buried by other items. Implications and future directions are discussed. (PsycInfo Database Record (c) 2023 APA, all rights reserved)

Evidence of ‘Green’ behaviours: Exploring behavioural traces of pro- and anti-environmental behaviors

The current climate crisis requires pro-environmental behaviours (PEBs) to be developed, engaged in, and spread to other people. Behavioural traces, i.e. evidence of other people's pro-environmental behaviour left in the shared environment, have shown to influence people towards being more pro-environmental. However, systematic research into behavioural traces of PEBs is missing. In a set of three surveys, we investigate which behavioural traces correspond to a number of pro- and anti-environmental behaviours identified from previous literature, how frequently these behavioural traces are encountered, their relation with engagement in behaviours, and whether behaviours can be inferred from traces. All studies are survey-based with a mix of open-ended questions (Surveys 1 & 3) and rating scales (Survey 2). We use network analysis to identify partial correlations between behaviours and traces. A total of 66 traces uniquely attributed to 36 pro- and anti-environmental behaviours were identified. On average, each trace is observed monthly. Noticing traces correlated with engaging in related behaviours in 24 instances. Participants report that if they saw a trace more frequently, they expect they would be more likely to adopt the behaviour that produced the trace. Finally, participants were generally able to infer the causing behaviours when only presented the traces. We show that unique behavioural traces exist for a number of pro- and anti-environmental behaviours. Traces are noticed and relate to the constituting behaviours based on correlational and self-report evidence. Because of the wide variation between behaviours and their traces, further research into specific behaviours is warranted. Use of these findings for interventions are discussed

Decision making

This chapter reviews normative and descriptive aspects of decision making. Expected Utility Theory (EUT), the dominant normative theory of decision making, is often thought to provide a relatively poor description of how people actually make decisions. Prospect Theory has been proposed as a more descriptively valid alternative. The failure of EUT seems at least partly due to the fact that people’s preferences are often unstable and subject to various influences from the method of elicitation, decision context, and goals. In novel situations, people need to infer their preferences from various cues such as the context and their memories and emotions. Through repeated experience with particular decisions and their outcomes, these inferences can become more stable, resulting in behavior that is more consistent with EUT

Better safe than sorry: Risky function exploitation through safe optimization

Exploration-exploitation of functions, that is learning and optimizing a mapping between inputs and expected outputs, is ubiquitous to many real world situations. These situations sometimes require us to avoid certain outcomes at all cost, for example because they are poisonous, harmful, or otherwise dangerous. We test participants' behavior in scenarios in which they have to find the optimum of a function while at the same time avoid outputs below a certain threshold. In two experiments, we find that Safe-Optimization, a Gaussian Process-based exploration-exploitation algorithm, describes participants' behavior well and that participants seem to care firstly whether a point is safe and then try to pick the optimal point from all such safe points. This means that their trade-off between exploration and exploitation can be seen as an intelligent, approximate, and homeostasis-driven strategy.Comment: 6 pages, submitted to Cognitive Science Conferenc

Cross-dimensional magnitude interactions arise from memory interference

Magnitudes from different dimensions (e.g., space and time) interact with each other in perception, but how these interactions occur remains unclear. In four experiments, we investigated whether cross-dimensional magnitude interactions arise from memory interference. In Experiment 1, participants perceived a constant-length line consisting of two line segments of complementary lengths and presented for a variable stimulus duration; then they received a cue about which of the two segment lengths to later reproduce. Participants were to first reproduce the stimulus duration and then the cued length. Reproduced durations increased as a function of the cued length if the cue was given before duration was retrieved from memory for reproduction (i.e. before duration reproduction; Experiment 1) but not if it was given after the duration memory had been retrieved from memory (i.e. after the start of duration reproduction; Experiment 2). These findings demonstrate that space-time interaction arises as a result of memory interference when length and duration information co-exist in working memory. Experiment 3 further demonstrated spatial interference on duration memories from memories of filled lengths (i.e. solid line segments) but not from noisier memories of unfilled lengths (demarcated empty spatial intervals), thus highlighting the role of memory noise in space-time interaction. Finally, Experiment 4 showed that time also exerted memory interference on space when space was presented as (relatively noisy) unfilled lengths. Taken together, these findings suggest that cross-dimensional magnitude interactions arise as a result of memory interference and the extent and direction of the interaction depend on the relative memory noises of the target and interfering dimensions. We propose a Bayesian model whereby the estimation of a magnitude is based on the integration of the noisily encoded percept of the target magnitude and the prior knowledge that magnitudes co-vary across dimensions (e.g., space and time). We discuss implications for cross-dimensional magnitude interactions in general

Time pressure changes how people explore and respond to uncertainty

How does time pressure influence exploration and decision-making? We investigated this question with several four-armed bandit tasks manipulating (within subjects) expected reward, uncertainty, and time pressure (limited vs. unlimited). With limited time, people have less opportunity to perform costly computations, thus shifting the cost-benefit balance of different exploration strategies. Through behavioral, reinforcement learning (RL), reaction time (RT), and evidence accumulation analyses, we show that time pressure changes how people explore and respond to uncertainty. Specifically, participants reduced their uncertainty-directed exploration under time pressure, were less value-directed, and repeated choices more often. Since our analyses relate uncertainty to slower responses and dampened evidence accumulation (i.e., drift rates), this demonstrates a resource-rational shift towards simpler, lower-cost strategies under time pressure. These results shed light on how people adapt their exploration and decision-making strategies to externally imposed cognitive constraints