Many situations arise in which an interested party wishes to
affect the decisions of an agent; e.g., a teacher that seeks to
promote particular study habits, a Web 2.0 site that seeks to
encourage users to contribute content, or an online retailer
that seeks to encourage consumers to write reviews. In the
problem of environment design, one assumes an interested
party who is able to alter limited aspects of the environment
for the purpose of promoting desirable behaviors. A critical
aspect of environment design is understanding preferences,
but by assumption direct queries are unavailable. We work in
the inverse reinforcement learning framework, adopting here
the idea of active indirect preference elicitation to learn the reward function of the agent by observing behavior in response
to incentives. We show that the process is convergent and
obtain desirable bounds on the number of elicitation rounds.
We briefly discuss generalizations of the elicitation method to
other forms of environment design, e.g., modifying the state
space, transition model, and available actions.Engineering and Applied Science
