ABSTRACT This paper proposes a novel method to realize an initiative exchange for robot. A humanoid robot plays vibraphone exchanging initiative with a human performer by perceiving multimodal cues in real time. It understands the initiative exchange cues through vision and audio information. In order to achieve the natural initiative exchange between a human and a robot in musical performance, we built the system and the software architecture and carried out the experiments for fundamental algorithms which are necessary to the initiative exchange