Halfwerk, W.

Hochradel, K.

Mendoza, E.

Riebel, K.

Scharff, C.

Simon, R.

Varkevisser, J.M.

Leiden University Scholary Publications

Development and application of a robotic zebra finch (RoboFinch) tostudy multimodal cues in vocal communicationSimon, R.; Varkevisser, J.M.; Mendoza, E.; Hochradel, K.; Scharff, C.; Riebel, K.; Halfwerk, W.CitationSimon, R., Varkevisser, J. M., Mendoza, E., Hochradel, K., Scharff, C., Riebel, K., & Halfwerk,W. (2019). Development and application of a robotic zebra finch (RoboFinch) to studymultimodal cues in vocal communication. Peerj. doi:10.7287/peerj.preprints.28004v3 Version: Publisher's VersionLicense: Creative Commons CC BY 4.0 licenseDownloaded from: https://hdl.handle.net/1887/4211750 Note: To cite this publication please use the final published version (if applicable). 1 Development and application of a robotic zebra finch (RoboFinch) to study multimodal cues in vocal communication Ralph Simon1,2, Judith Varkevisser2, Ezequiel Mendoza3, Klaus Hochradel4, Constance Scharff3, Katharina Riebel2 & Wouter Halfwerk1 1Department of Ecological Science VU University Amsterdam, Amsterdam, The Netherlands 2Institute of Biology Leiden, Leiden University, Leiden, The Netherlands 3Institut für Biologie, Freie Universität Berlin, Berlin, Germany 4Institute of Measurement and Sensor Technology, UMIT-Private University for Health Sciences, Medical Informatics and Technology GmbH, Hall in Tirol, Austria  Corresponding Author:  Ralph Simon De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands Email address: ralph.simon@vu.nl   ABSTRACT Understanding animal behaviour through psychophysical experimentation is often limited by insufficiently realistic stimulus representation. Important physical dimensions of signals and cues, especially those that are outside the spectrum of human perception, can be difficult to standardize and control separately with currently available recording and displaying techniques (e.g. video displays). Accurate stimulus control is in particular important when studying multimodal signals, as spatial and temporal alignment between stimuli is often crucial. Especially for audiovisual presentations, some of these limitations can be circumvented by the employment of animal robots that are superior to video presentations in all situations requiring realistic 3D presentations to animals. Here we report the development of a robotic zebra finch, called RoboFinch, and how it can be used to study vocal learning in a songbird, the zebra finch.  INTRODUCTION The use of stationary and animated, robotic animal models knows a long tradition. Such models have been used in contexts as diverse as e.g. mate attraction signalling, predator-prey interactions or cooperation to investigate which stimulus properties trigger animals’ reactions1,2,3. Advances in technology involving new materials, small-sized actuators, 3D printing techniques and more computational power have greatly increased the possibilities for a new generation of even more realistic robotic animal models4, which can take the study of animal communication signals to the next level, especially in the context of multimodal signalling. This form of signalling where signals in one modality are either facultatively or obligatorily accompanied by signalling in one or more additional modalities, is widespread in animals especially in the context of mate attraction: Birds sing and dance, mammals show visual display, acoustic and chemical signals, many insects PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.28004v3 | CC BY 4.0 Open Access | rec: 30 Dec 2019, publ: 30 Dec 2019 2 combine acoustic, vibratory and chemical signalling5. However, these signals are usually studied in one modality only, often owing to the technical problems involved in controlling more than one modality during stimulus presentations. Robotic models allow multimodal signal components to be controlled independently. This allows to expand the stimulus range and to produce artificial stimulus combinations testing receivers’ reaction to different combinations of signal components. Robotic applications have already helped to understand multimodal stimulus processing in the context of territory defence or sexual signalling (e.g. in some frog and bird species 6,4,7) but could also open a research window into the development of the perception of multimodal signals. Here we outline the progress in designing a robotic bird, looking and singing like a songbird, the zebra finch Taeniopygia guttata. This species is an important model to study the behavioural and neurobiological aspects of vocal learning. A robotic zebra finch would allow studying the potential role of multimodal cues such as beak movements in vocal learning. To enable such studies, our goal was to create a realistic moving 3D-model of a singing bird showing the fast and sound-specific beak movements accompanying male song.  DEVELOPMENT OF THE RoboFinch To develop the basic form of the RoboFinch, we 3D scanned a taxidermic model of a zebra finch with a handheld 3D scanner (Eva, Artec3D, Luxembourg, Luxembourg). The beak was scanned with high resolution from a prepared skull (ATOS 5X, gom, Braunschweig, Germany, Fig. 1 A, B). These scans were combined in the program Catia V5R20 (Dassault Systèmes), where we also implemented the inner mechanics (Fig. 1 C). We printed the RoboFinch with stereolithography 3D printing (Form 2, Formlabs, Somerville, Massachusetts, US), which uses a laser to cure solid isotropic parts from a liquid photopolymer resin (Grey Pro, Formlabs Resin).            Figure 1. 3D Models of the RoboFinch. High resolution scan of (A) the upper and (B) the lower beak. (C) Catia Model of the RoboFinch showing the inner mechanics PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.28004v3 | CC BY 4.0 Open Access | rec: 30 Dec 2019, publ: 30 Dec 2019 3 The movement of the head and beak was controlled by coils we got from dismantling DigiBirds (Silverlit Toys Manufactory, Hongkong, China). The advantage of using those coils is that they are cost-effective, small and allow fast movements up to 100 Hz. The coils were controlled via a custom build controller board. All the movements and the sound were controlled via a data acquisition card (Measurement Computing USB-3101), which was connected to a small desktop PC (Intel NUC i5). All movements, sound and the schedule of the stimulus presentation was controlled by a custom made LabView (National Instruments) Program. We painted the 3D printed models by hand with mixes of Revell colours (Revell, Bünde, Germany), which we tried to closely fit to the colours of the plumage of live zebra finches. We measured the colour spectra of the RoboFinch and live males (n = 6) with a spectrometer (Flame, Ocean Optics, Largo, Florida, US). To create movement files, we did high-speed video recordings (120fps) of singing male finches and deduced their head and beak movements with tracker software and played the recorded movements on the robot. The corresponding sound was played via a loudspeaker placed next to the robot.  APPLICATION OF THE RoboFinch IN A TUTORING SITUATION We tested the acceptance of the robots with two groups of young zebra finches, from 45-75 post-hatching, each group consisting of one male and one female. The setup consisted of a big cage with a black wall on one side. The black wall had a mesh window (20 x 15 cm), the robot was placed directly behind this window and the birds inside the cage could sit on a perch directly at that window. The birds were accustomed to the non-moving robot            Figure 2. Images of the experimental cages of the two experimental groups of young birds summarising their movements in the cage with frame differencing. We plotted the frame differences in a colour gradation, the darker the red, the stronger were the cumulative pixel differences of adjacent frames and the stronger the movement. Each of the four plots is based on 12 events (2 tutor sessions on each of the 6 first exposure days) and for each event we analysed 600 frames (60s, 10fps). The images above are based on sequences right before the robot moved and vocalized, below directly after the robot started moving and vocalizing. Note that the movement of the robot is also visible in the lower images as dark dots. PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.28004v3 | CC BY 4.0 Open Access | rec: 30 Dec 2019, publ: 30 Dec 2019 4 model for about 12 hours (afternoon and night) and then in the following morning (8 am) the robot started moving for the first time. The robot was programmed to move 6 times a day for half an hour displaying head and beak movements associated with short calls and song. Experiments were approved by the Leiden University Animal Welfare Body and all zebra finches were housed and cared for in accordance with the Experiments on Animals Act (Wod, 2014). We observed the birds with webcams (10fps), recorded their behavior and exemplarily analyzed sequences of video frames (600 greyscale frames, 1min) before and after the robot started moving. We did that for two sessions per day over the 6 initial days. As a measure of movement, we used a frame differencing method and calculated the mean difference between adjacent frames (see  Figure 2). Before the robot started moving, the young birds moved preferentially in the horizontal direction between the upper perches (the two upper images in Figure 2) indicating that the birds were not particularly interested in the robot when it wasn’t moving and singing. As soon as the robot started moving and vocalizing however, the birds showed more interest and were approaching the robot as can be seen by the green to dark red pixels closer to the robot (Fig. 2, lower images, right end of the cage). Throughout the song tutoring, during the sensitive phase for song learning, the robot remained placed outside the cage, but at day 74 and 75 we also tested the acceptance of the robot in the cage and noticed the birds interacting with it, in particular throughout the phases when it was moving and singing (see Fig. 3).                 Figure 3. Young zebra finches interacting with the RoboFinch inside the cage. PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.28004v3 | CC BY 4.0 Open Access | rec: 30 Dec 2019, publ: 30 Dec 2019 5 CONCLUSION Our work builds upon and adds to previous work suggesting that robotic models help to uncover the role of visual cues in song learning in zebra finches8. At least one behavioural experiment with zebra finches and a non-moving robot showed that the birds vocally interact with this model9. Moreover, plastic models are already used as tutors in zebra finch song learning experiments10. Our preliminary data demonstrates that our RoboFinch is accepted by young zebra finches nearly instantaneously and that they also interact with it. These observations suggest that the RoboFinch could successfully be used for song tutoring experiments. As we now are able to control visual and acoustic stimuli independently and find out how the combination of these can influence vocal learning, our experiments could help to uncover general principles of multimodal sensory integration in animal communication. ACKNOWLEDGEMENT We greatly acknowledge the help of Dré Kampfraath, Rogier Elsinga, Peter Wiersma and Wesley Delmeer during the development of the RoboFinch. This work was funded by the Human Frontier Science Program (No RGP0046/2016).  REFERENCES 1 Krause, J., Winfield, A. F. & Deneubourg, J.-L. Interactive robots in experimental biology. Trends in Ecology & Evolution 26, 369-375 (2011). 2 Webb, B. What does robotics offer animal behaviour? Animal Behaviour 60, 545-558 (2000). 3 Webb, B. Using robots to understand animal behavior. Advances in the Study of Behavior 38, 1-58 (2008). 4 Taylor, R. C., Klein, B. A., Stein, J. & Ryan, M. J. Faux frogs: multimodal signalling and the value of robotics in animal behaviour. Animal Behaviour 76, 1089-1097 (2008). 5 Halfwerk, W. et al. Toward Testing for multimodal perception of mating Signals. Frontiers in Ecology and Evolution 7, 124 (2019). 6 Narins, P. M., Grabul, D. S., Soma, K. K., Gaucher, P. & Hödl, W. Cross-modal integration in a dart-poison frog. Proceedings of the National Academy of Sciences 102, 2425-2429 (2005). 7 Ręk, P. & Magrath, R. D. Multimodal duetting in magpie-larks: how do vocal and visual components contribute to a cooperative signal's function? Animal behaviour 117, 35-42 (2016). 8 Derégnaucourt, S., Poirier, C., Van der Kant, A., Van der Linden, A. & Gahr, M. Comparisons of different methods to train a young zebra finch (Taeniopygia guttata) to learn a song. Journal of Physiology-Paris 107, 210-218 (2013). 9 Benichov, J. I. et al. The forebrain song system mediates predictive call timing in female and male zebra finches. Current Biology 26, 309-318 (2016). 10 Tchernichovski, O., Mitra, P. P., Lints, T. & Nottebohm, F. Dynamics of the vocal imitation process: how a zebra finch learns its song. Science 291, 2564-2569 (2001). PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.28004v3 | CC BY 4.0 Open Access | rec: 30 Dec 2019, publ: 30 Dec 2019

Development and application of a robotic zebra finch (RoboFinch) to study multimodal cues in vocal communication

https://scholarlypublications.universiteitleiden.nl/access/item%3A4211751/download

Development and application of a robotic zebra finch (RoboFinch) to study multimodal cues in vocal communication

Abstract

Similar works

Full text

Available Versions

Leiden University Scholary Publications