Large populations of simple robots can solve complex tasks, but controlling them is still a challenging problem, due to limited communication and computation power. In order to assemble objects, have shown that a human controller can solve such a task. Instead, we investigate how to learn the assembly of multiple objects with a single central controller. We propose splitting the assembly process in two sub-tasks -- generating a top-level assembly policy and learning an object movement policy. The assembly policy plans the trajectories for each object and the object movement policy controls the trajectory execution.The resulting system is able to solve assembly tasks with varying object shapes being assembled as shown in multiple simulation scenarios

Daun, Kevin

Gebhardt, Gregor H. W.

Hendrich, Alexander

Kauth, Daniel

Neumann, Gerhard

Schnaubelt, Marius

Gebhardt, G. H. W.

Daun, K.

Schnaubelt, M.

Hendrich, A.

Kauth, D.

Neumann, G.

TUbiblio

Learning to Assemble Objects with a Robot Swarm

University of Lincoln Institutional Repository

Learning to Assemble Objects with a Robot Swarm(Extended Abstract)Gregor H.W. GebhardtComputational Learning forAutonomous Systems,Technische Universität DarmstadtKevin DaunComputational Learning forAutonomous Systems,Technische Universität DarmstadtMarius SchnaubeltComputational Learning forAutonomous Systems,Technische Universität DarmstadtAlexander HendrichComputational Learning forAutonomous Systems,Technische Universität DarmstadtDaniel KauthComputational Learning forAutonomous Systems,Technische Universität DarmstadtGerhard NeumannLincoln Centre for AutonomousSystems, School of ComputerScience, University of Lincoln1. INTRODUCTIONNature provides us with a multitude of examples thatshow how swarms of simple agents are much richer in theirabilities than a single individual. This insight is a main prin-ciple that swarm robotics tries to exploit. In the last years,large swarms of low-cost robots such as the Kilobots [6] havebecome available. This allows to bring algorithms developedfor swarm robotics from simulations to the real world. Re-cently, the Kilobots have been used for an assembly taskwith multiple objects [7]: a human operator controlled alight source to guide the swarm of light-sensitive robots suchthat they successfully assembled an object of multiple parts.However, hand-coding the control of the light source for au-tonomous assembly is not straight forward as the interac-tions of the swarm with the object or the reaction to thelight source are hard to model.In this paper, we investigate how to learn a policy thatcontrols the light source such that the Kilobots solve the as-sembly task autonomously. We subdivide the assembly pro-cess in two sub-tasks: the high-level assembly plan is givenby waypoints for each object while the learned low-level ob-ject movement policy controls the trajectory execution byguiding the Kilobots with the light source.Learning to move objects is a complex task as we have tocoordinate a large number of agents, which results in manystate variables. Eventough we need information about theconfiguration of the swarm (i.e. the positions of the agents)in our state representation, it is not important which individ-ual of the swarm is at which position and, furthermore, ourpolicy should be as well invariant to the number of agents inthe swarm. Hence, we represent the state of the swarm as adistribution over the agent locations embedded into a repro-ducing kernel Hilbert space [8]. Our reinforcement learningalgorithm is based on the recently introduced actor-criticrelative entropy policy search (AC-REPS) algorithm [9] andlearns a nonparametric Gaussian process (GP) policy forcontrolling the light source. We evaluate our approach insimulation on assembly tasks with different object shapes.Appears in: Proc. of the 16th International Conference onAutonomous Agents and Multiagent Systems (AAMAS 2017),S. Das, E. Durfee, K. Larson, M. Winikoff (eds.),May 8–12, 2017, Sa˜o Paulo, Brazil.Copyright c© 2017, International Foundation for Autonomous Agentsand Multiagent Systems (www.ifaamas.org). All rights reserved.Figure 1: We propose a new reinforcement learning approachto enable a robot swarm (grey circles) to assemble objects(top: squares, bottom: T- and C-shape). The swarm isguided by a light cone (yellow circle) which follows a learnedcontrol policy.2. SOLVING AN ASSEMBLY TASKOur solution to the assembly task consists of three com-ponents: (1) a path planning strategy to guide the swarmaround the objects and to arrange them for the next push-ing task, (2) an assembly policy that describes how the partsshould move such that they in the end form a whole unit, and(3) an object movement policy that realizes basic movementsof an object part indirectly by controlling a light source thatguides the robots.Learning the Object Movement Policy.We use AC-REPS [9] to learn the continuous, nonpara-metric object movement policies from samples. This model-free reinforcement learning approach based on relative en-tropy policy search (REPS) [5] consists of the following threesteps: (1) Estimate the Q-function using the observed statetransitions with least-squares temporal difference learning(LSTD). (2) Improve the policy by maximizing the expectedQ-value for the sampled data using REPS. (3) Obtain a con-tinuous policy by matching a weighted Gaussian process.We learn multiple policies to control the light source whichdiffer in their objective: one policy for pure translation inpositive x-direction, one for pure counterclockwise (ccw) ro-tation of the object, and arbitrary many policies for dif-ferent ratios w of combined translation and rotation. Thereward function reflects the setting of the control parameterw ∈ [0, 1] which sets the ratio of translation and rotation.The function rewards a ccw rotation for w = 1 and a trans-lation in x direction for w = 0. For values w ∈ ]0, 1[ theterms are weighted accordingly. Later, we will apply theobject movement policies to arbitrary target directions andorientations by rotating and flipping the state of the world.We represent the state relative to the center of mass of theobject part that we want to push. Given a swarm configura-tion with n agents and their relative positions bi = (xi, yi)as well as the relative light position l = (xl, yl), we definethe state vector as s := [l, b1, . . . , bn]. The action vectora = (ax, ay) contains the desired shift of the center of thelight cone.To learn the object movement policy independently fromthe number of individuals in the swarm, we represent theswarm as distribution embedded into a reproducing ker-nel Hilbert space (RKHS). Let H be an RKHS of func-tions, uniquely defined by a positive definite kernel func-tion k(x, x′) := 〈φ(x), φ(x′)〉H [2]. Here, the feature map-pings φ(x) are in general intrinsic to the kernel functions andmight map into an infinite dimensional feature space. Wecan embed now a marginal distribution p(X) as the expectedfeature mapping of its random variable [8] µX := EX [φ(X)].In practice we need to estimate the embedding from samplesas µˆX =1m∑mi=1 k(xi, ·).Thus, the state of the swarm relative to the center ofmass of the part we want to push is represented as µ(k) =1n∑ni=1 k(bi, ·), where k is a kernel function (e.g., the Gaus-sian kernel) and the bi are the locations of the individualagents. With this representation, we can compute the dif-ference between two swarm distributions by computing thesquared difference of their embeddings1n2n,n∑i=1j=1k(bi, bj)− 2nmn,m∑i=1j=1k(bi, b′j) +1m2m,m∑i=1j=1k(b′i, b′j),with the two swarm configurations b and b′ with n and mindividuals, respectively. Based on this distance metric, wedefine a feature vector using radial basis functions (RBF)centered around our samples. The remaining state-actionvariables (position and desired shift of the light) are com-pared by a squared distance. We can now combine these twodistance measures into an exponential product kernel basedon which we define the feature functions for approximatingthe Q-function and for matching the weighted GP as thenew policy.Assembly Policy and Path Planning.The assembly policy contains the construction informa-tion stored as a list of oriented way points with requiredaccuracies for each object. These way points are processedconsecutively by applying either the object movement pol-icy or the path planning strategy. The desired translation-rotation ratio wdes = erot/(etrans + erot) obtained from therotational and translational error determines which of thelearned object movement policies is executed.After reaching a way point, the swarm has to be arrangedat the next object. We use a path planning strategy toguide the swarm around objects and arrange them for thenext pushing task. This strategy is a combination of A*with potential fields.A* is a heuristic search algorithm commonly applied forgraph search problems [3]. The algorithm selects which nodens to expand by minimizing the sum of the cost we have tospend for reaching node ns from the start and a heuristicFigure 2: The value function plots around the object depicthow our approach successfully adapts to different settings oftranslation-rotation-ratio w.20 40 60 8002468·10− 3#KilobotsReward/Stepw = 0w = 1(a) Up to 50 agents, the perfor-mance benefits from the largerswarm size.(b) If the swarm gets too big, itspreads around the object anddisturbs the movement.Figure 3: Performance of a policy learned with 15 individu-als applied to different swarm sizes.lower bound to the cheapest costs from ns to the goal state.Potential fields [4] are a fast planning method for mobilerobots. The robots move along a hypothetical force field,being attracted towards the goal position and repulsed awayfrom the obstacles. In our approach, we use the repulsivepotential in the cost term for the path segments c(ns1 , ns2)of A*.3. EXPERIMENTAL RESULTSTo conduct our experiments, we developed a 2D Kilobotsimulator based on Box2D [1]. By applying a heuristic sam-pling strategy for the initial state of the kilobots, we assurea stable learning process. We learned the object movementpolicy with 15 Kilobots for a square object over 20 iterations.Figure 2 shows the value functions for three different settingsof the translation-rotation ratio w after 15 iterations.To evaluate how well the learned policy generalizes to dif-ferent swarm sizes, we applied policies learned with 15 in-dividuals to different numbers of Kilobots. Figure 3a showshow the average reward per step benefits from a growingswarm size up to about 50 individuals. As shown in Fig-ure 3b, the reward drops with a bigger swarm size since theindividuals start to disperse around the object and disturbor block the movement.In combination with hand-coded assembly policies and thepath planning strategies described above, we applied thelearned object movement policies to different assembly tasks.The top row in Figure 1 depicts the assembly process of com-posing a larger square out of four small squares. In a secondtask, the kilobots had to assemble a C-shaped object anda T-shaped object. This process is depicted in the bottomrow of Figure 1. This second task also demonstrates that thelearned object movement policies—to a certain extend—canbe transferred to different object shapes.REFERENCES[1] Box2d, a 2d physics engine for games.http://box2d.org/.[2] N. Aronszajn. Theory of reproducing kernels.Transactions of the American mathematical society,pages 337–404, 1950.[3] P. E. Hart, N. J. Nilsson, and B. Raphael. A formalbasis for the heuristic determination of minimum costpaths. IEEE transactions on Systems Science andCybernetics, 4(2):100–107, 1968.[4] O. Khatib. Real-time obstacle avoidance formanipulators and mobile robots. The internationaljournal of robotics research, 5(1):90–98, 1986.[5] J. Peters, K. Mu¨lling, and Y. Altun. Relative entropypolicy search. In AAAI. Atlanta, 2010.[6] M. Rubenstein, C. Ahler, N. Hoff, A. Cabrera, andR. Nagpal. Kilobot: A low cost robot with scalableoperations designed for collective behaviors. Roboticsand Autonomous Systems, 62(7):966–975, 2014.[7] M. Rubenstein, A. Cabrera, J. Werfel, G. Habibi,J. McLurkin, and R. Nagpal. Collective transport ofcomplex objects by simple robots: Theory andexperiments. In Proceedings of the 2013 InternationalConference on Autonomous Agents and Multi-agentSystems, AAMAS ’13, 2013.[8] A. Smola, A. Gretton, L. Song, and B. Scho¨lkopf. Ahilbert space embedding for distributions. In InAlgorithmic Learning Theory: 18th InternationalConference, pages 13–31. Springer-Verlag, 2007.[9] C. Wirth, J. Fu¨rnkranz, and G. Neumann. Model-freepreference-based reinforcement learning. In ThirtiethAAAI Conference on Artificial Intelligence, 2016.

Learning to assemble objects with a robot swarm

Learning to assemble objects with a robot swarm

Abstract

Similar works

Full text

Available Versions

TUbiblio

University of Lincoln Institutional Repository