Using informative behavior to increase engagement while learning from human reward

In this work, we address a relatively unexplored aspect of designing agents that learn from human reward. We investigate how an agent’s non-task behavior can affect a human trainer’s training and agent learning. We use the TAMER framework, which facilitates the training of agents by human-generated reward signals, i.e., judgements of the quality of the agent’s actions, as the foundation for our investigation. Then, starting from the premise that the interaction between the agent and the trainer should be bi-directional, we propose two new training interfaces to increase a human trainer’s active involvement in the training process and thereby improve the agent’s task performance. One provides information on the agent’s uncertainty which is a metric calculated as data coverage, the other on its performance. Our results from a 51-subject user study show that these interfaces can induce the trainers to train longer and give more feedback. The agent’s performance, however, increases only in response to the addition of performance-oriented information, not by sharing uncertainty levels. These results suggest that the organizational maxim about human behavior, “you get what you measure”—i.e., sharing metrics with people causes them to focus on optimizing those metrics while de-emphasizing other objectives—also applies to the training of agents. Using principle component analysis, we show how trainers in the two conditions train agents differently. In addition, by simulating the influence of the agent’s uncertainty–informative behavior on a human’s training behavior, we show that trainers could be distracted by the agent sharing its uncertainty levels about its actions, giving poor feedback for the sake of reducing the agent’s uncertainty without improving the agent’s performance

Virtual Weigh Station

Overweight trucks shorten highway life and indirectly increase the costs of maintaining roads. Improved methods for enforcement of commercial vehicle weight laws may increase the number of overweight vehicles caught, thereby prolonging highway life. Improved enforcement may also reduce the number of illegally operating vehicles. This report describes the concept of using existing INDOT Weigh-In-Motion equipment, a laptop computer, and wireless communication equipment, to develop a virtual weigh station screening tool. The Virtual Weigh Station screening tool developed in this project allows officers to read the weights of vehicles crossing WIM scales, in real time, in their patrol cars. Giving officers this information increases the chances that the vehicles selected for weighing on portable scales are indeed overweight. This report documents the accuracy and precision evaluation performed on all the candidate WIM sites as well as the new infrastructure required to implement the Virtual Weigh Station concept. The report describes several cases where significantly overweight vehicles were identified and impounded. For example the procedures described in this report, resulted in the identifying the early morning hours as the best time for enforcement in Merrillville. As a result, on May 18, 2001 vehicles weighing 98,700 lbs and 100,600 lbs were stopped. Those vehicles were impounded and resulted in fines of $1,625.00 and$1,735.50, respectively. In February 2002, Commercial Vehicle Enforcement officers stopped ten trucks on US 24 near Fort Wayne using the virtual weigh station. Eight of the trucks were determined to be overweight and fined. The three heaviest trucks weighed 90,200 lbs, 90,900 lbs, and 91,100 lbs resulting in fines of $1,099.50,$1,169.50, and $1,189.50, respectively. On April 12, 2002, another enforcement was conducted on I-65 near Merrillville. Three trucks were stopped based on the virtual weigh station data. One of the trucks weighed 87,400 lbs, resulting in a$529.50 fine. The report concludes by making several recommendations designed to improve the quality of the WIM data and facilitate wide spread deployment by the Commercial Vehicle Enforcement Division

Effects of the flood regime on the body condition of fish of different trophic guilds in the Upper Paraná River floodplain, Brazil

In this study, we evaluated the influence of various hydrological cycles on the feeding body condition of fish species of different trophic guilds in the Paraná River floodplain, as well as the impacts of upstream impoundments on fish conditions. Attributes of the river floods (duration, time of year, intensity, and variability in a given year) and the body condition, measured by the mean residuals of length-weight ratios, of the detritivorous, herbivorous, insectivorous, invertivores, omnivorous, piscivorous, and planktivorous species were evaluated. Fish were sampled during a period before (1986-1994) and after (2000-2004) the completion of filling of the Porto Primavera Reservoir, which is located upstream from the floodplain area under study. Three sub-basins in the floodplain were sampled: the Ivinheima River, which has no dams; the Paraná River, which has several dams; and the Baia River, which is influenced by the Paraná. A two-way ANOVA identified significant variations in mean body conditions for localities and for the hydrological cycles, and the interactions were also significant. The findings revealed that before the Porto Primavera Reservoir was filled, the body condition of the feeding guilds varied similarly in the three sub basins, but this pattern was not observed after filling was completed. However, in years with minor or no floods, the body condition was high, especially in the sub-basins influenced by Porto Primavera (Paraná and Baía). Pearson's and Spearman's correlations showed that most effects of the floods were unfavorable to the body condition of the guilds, except for the annual variation in water level, which aids herbivores in accessing allochthonous food resources. Detritivores were negatively affected by all flood attributes. A correlation between the relative stomach weight (mean residual of the ratio of total and stomach weights) and the body condition demonstrated the poor relationship between the amount of food intake and weight gain; insectivores were the exception, showing a significant negative correlation. Therefore, for this guild, there is evidence that their food sources during floods have low nutritional value. It is hypothesized that dry years lead to improvement in the nutritional quality of food resources, and that the water released from the upstream dam dilutes and removes these resources without providing new food inputs. This exacerbates the effects of floods on body conditions