Clarifying the underlying user information need by asking clarifying
questions is an important feature of modern conversational search system.
However, evaluation of such systems through answering prompted clarifying
questions requires significant human effort, which can be time-consuming and
expensive. In this paper, we propose a conversational User Simulator, called
USi, for automatic evaluation of such conversational search systems. Given a
description of an information need, USi is capable of automatically answering
clarifying questions about the topic throughout the search session. Through a
set of experiments, including automated natural language generation metrics and
crowdsourcing studies, we show that responses generated by USi are both inline
with the underlying information need and comparable to human-generated answers.
Moreover, we make the first steps towards multi-turn interactions, where
conversational search systems asks multiple questions to the (simulated) user
with a goal of clarifying the user need. To this end, we expand on currently
available datasets for studying clarifying questions, i.e., Qulac and ClariQ,
by performing a crowdsourcing-based multi-turn data acquisition. We show that
our generative, GPT2-based model, is capable of providing accurate and natural
answers to unseen clarifying questions in the single-turn setting and discuss
capabilities of our model in the multi-turn setting. We provide the code, data,
and the pre-trained model to be used for further research on the topic