Hough J, Schlangen D. PentoRob: A Puzzle-Playing Robot for Dialogue Experiments. In: Hunter J, Simons M, Stone M, eds. Proceedings of JerSem: The 20th Workshop on the Semantics and Pragmatics of Dialogue. New Brunswick, NJ;  2016: 114-115

Hough, Julian

Hunter, Julie

Schlangen, David 

Simons, Mandy

Stone, Matthew

Publications at Bielefeld University

PentoRob: A Puzzle-Playing Robot for Dialogue ExperimentsJulian Hough and David SchlangenDialogue Systems Group // CITEC // Faculty of Linguistics and LiteratureBielefeld Universityfirstname.lastname@uni-bielefeld.deAbstractWe present a simple puzzle-playing inter-active robot, PentoRob, which allows in-vestigation into real-time, real-world di-alogue. The dialogue control frameworkconsists of a combination of interactiveHarel statecharts and the Incremental Unitframework. We outline its architectureand potential use cases for dialogue andhuman-robot interaction.1 IntroductionIn embodied dialogue systems research, there is aneed for simple robots that do not require heavymechanical maintenance or robotics experts whendeveloping functionality of interest. Here wepresent a system to fulfil these needs: PentoRob,a simple pick-and-place robot controlled by an in-cremental dialogue framework.2 PentoRobPentoRob is a puzzle-playing robot which ma-nipulates Pentomino pieces– see Fig. 1. Itsdialogue control consists of Harel statecharts(Harel, 1987) and the Incremental Unit framework(Schlangen and Skantze, 2011), and is imple-mented with the dialogue toolkit InproTK (Bau-mann and Schlangen, 2012). Here we describe itscomponents in terms of input information or In-cremental Units (IUs), processing, and output IUs.Hardware For the robotic arm, we use theShapeOko2,1 a heavy-duty 3-axis CNC machine,which we modified with a rotatable electromag-net, whereby its movement and magnetic field iscontrolled via two Arduino boards. The sensorsare a webcam and microphone.Incremental Speech Recognizer (ASR) Weuse Google’s web-based ASR API which pack-ages hypotheses into individual WordIUs. While1http://www.shapeoko.com.Understanding Perception Actuation World System wordIU ADDs actionIU  COMMITs actionRequestIU ADDs actionIU COMMITs actionIU ADDs sceneIU ADDs Action generation   User   State Machine  Reference Resolution  ASR Vision Robot Control sceneIU ADDs Game  State Machine Robot State Machine G-code Figure 1: PentoRob’s architecture.its incremental performance is not as responsive asmore inherently incremental local systems such asKaldi or Sphinx-4, this does not incur great costsfor many interesting applications.Computer Vision (CV) We use OpenCV ina Python module to track objects in the cam-era’s view. This information is relayed to In-proTK from Python via the Robotics Service Bus(RSB),2 which outputs IDs and positions of ob-jects detected along with their low-level features(e.g., RGB/HSV values, x,y coordinates, numberof edges, etc.), converting these into SceneIUswhich the reference resolution module consumesand the Robot State Machine uses for obtainingthe positions of objects it plans to grab.Reference resolution (WAC) The referenceresolution component consists of a WordsAs Classifiers (WAC) model (Kennington andSchlangen, 2015) trained on real-world objectsusing low-level vision features from SceneIUs.2https://code.cor-lab.de/projects/rsb.Appears in  JerSem: The 20th Workshop on the Semantics and Pragmatics of Dialogue, edited by Julie Hunter, Mandy Simons and Matthew Stone. New Brunswick, NJ USA, July 16-18, 2016, pages 114-115.user_repairing_robot_action   user_committed_to_goal user_showing_ commitment_to_goal w [u ⋅ w : request,   𝐸𝐸 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈 ≥ 𝛿] user_uncommited robot_committed_to_goal robot_uncommited action [𝐸𝐸 𝑅𝑈𝑅𝑈𝑅𝑈𝑈𝑈𝑈 ≥ 𝜖] action[user_committed_to_goal] [𝐸𝐸 𝑅𝑈𝑅𝑈𝑅𝑈𝑈𝑈𝑈 <  𝜖,  user_uncommitted] robot_repairing_robot_action w[u ⋅ w: confirm, robot_showing_commtment_to_goal] [𝐸𝐸 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈 <  𝛿] User     Robot w [u ⋅ w : request, 𝐸𝐸 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈 ≥ 𝛿] w [ u ⋅ w: repair, robot_showing_ commitment_to goal] action[𝐸𝐸 𝑅𝑈𝑅𝑈𝑅𝑈𝑈𝑈𝑈 ≥ 𝜖] robot_showing_ commitment_to_goal action [ user_repairing_ robot_action] Figure 2: PentoRob’s Interactive Statechart with two parallel, concurrent statesDuring application, as a referring expression isrecognised, each classifier for the words in the ex-pression are applied to the puzzle pieces in view,which after normalisation, results in a probabilitydistribution over pieces.User and Robot State Machines For dialoguecontrol, we use an Interactive Statechart– seeFig. 2. Rather than comprising a single dialoguestate, there are concurrent states for each agent inthe interaction with their own variables. The UserState Machine has access to the estimated currentuser goal UserGoal and a strength-of-evidencefunction Ev(UserGoal), both of which can bedefined by the designer. In our domain UserGoalis the taking of the most likely object accordingto WAC’s output distribution given the utteranceu so far and the Ev function as the probabilityvalue of the highest ranked object in WAC’s distri-bution over its second highest rank. If UserGoalis changed or instantiated, a new ActionRequestIUis made available in its right buffer with the goal.The Robot State Machine gets access to itstransition conditions involving the user throughthe ActionRequestIUs. Through a simple plan-ning function, a number of ActionIUs are cuedto achieve the goal. It sends these as RSB mes-sages to the PentoRob control module and onceconfirmed, via RSB, that the action has begun, theActionIU is committed. For estimation of its ownstate, the robot state has a strength-of-evidencefunctionEv(RobotGoal) defined by the designer.PentoRob control module The module control-ling the robotic actuation of the ShapeOKO arm isa Python module with an Arduino board G-codeinterface. This sends RSB feedback messages tothe Robot State Machine to the effect that actionshave been successful or unsuccessful.3 Use casesWe are currently experimenting with achievingmore fluidity for grounding behaviour in human-robot interaction. The statechart in Fig. 2 hasparameters δ and  which are thresholds for theRobot and User that must be reached by theEv functions to show sufficient evidence of eachagent’s goal. Early results show that lower thresh-olds allow more fluid grounding behaviour, whilehigher thresholds are ‘safer’ for task success. Weare planning a series of related experiments. Otherareas where our setup could be used is learninggrounded semantics for verbs.AcknowledgmentsWe thank Oliver Eickmeyer, Casey Kennington and LiviaDia for contributions to software and hardware. This workwas supported by the Cluster of Excellence Cognitive Inter-action Technology ‘CITEC’ (EXC 277) at Bielefeld Univer-sity, funded by the German Research Foundation (DFG), andthe DFG-funded DUEL project (grant SCHL 845/5-1).ReferencesTimo Baumann and David Schlangen. 2012. The inprotk2012 release. In NAACL-HLT Workshop on Future Direc-tions and Needs in the Spoken Dialog Community: Toolsand Data. ACL.David Harel. 1987. Statecharts: A visual formalism forcomplex systems. Science of computer programming,8(3).Casey Kennington and David Schlangen. 2015. Simplelearning and compositional application of perceptuallygrounded word meanings for incremental reference reso-lution. Proceedings of the Conference for the Associationfor Computational Linguistics (ACL). ACL.David Schlangen and Gabriel Skantze. 2011. A General,Abstract Model of Incremental Dialogue Processing. Di-aloge & Discourse, 2(1).115

PentoRob: A Puzzle-Playing Robot for Dialogue Experiments

https://pub.uni-bielefeld.de/download/2904760/2906205

PentoRob: A Puzzle-Playing Robot for Dialogue Experiments

Abstract

Similar works

Full text

Available Versions

Publications at Bielefeld University