Hierarchical Mask2Former: Panoptic Segmentation of Crops, Weeds and Leaves

Advancements in machine vision that enable detailed inferences to be made from images have the potential to transform many sectors including agriculture. Precision agriculture, where data analysis enables interventions to be precisely targeted, has many possible applications. Precision spraying, for example, can limit the application of herbicide only to weeds, or limit the application of fertiliser only to undernourished crops, instead of spraying the entire field. The approach promises to maximise yields, whilst minimising resource use and harms to the surrounding environment. To this end, we propose a hierarchical panoptic segmentation method to simultaneously identify indicators of plant growth and locate weeds within an image. We adapt Mask2Former, a state-of-the-art architecture for panoptic segmentation, to predict crop, weed and leaf masks. We achieve a PQ{\dag} of 75.99. Additionally, we explore approaches to make the architecture more compact and therefore more suitable for time and compute constrained applications. With our more compact architecture, inference is up to 60% faster and the reduction in PQ{\dag} is less than 1%.Comment: 6 pages, 5 figures, 2 tables, for code, see https://github.com/madeleinedarbyshire/HierarchicalMask2Forme

Auction-based Task Allocation Mechanisms for Managing Fruit Harvesting Tasks

Multi-robot task allocation mechanisms are de-signed to distribute a set of activities fairly amongst a set of robots. Frequently, this can be framed as a multi-criteria optimisation problem, for example minimising cost while maximising rewards. In soft fruit farms, tasks, such as picking ripe fruit at harvest time, are assigned to human labourers. The work presented here explores the application of multi-robot task allocation mechanisms to the complex problem of managing a heterogeneous workforce to undertake activities associated with harvesting soft fruit

Exploring Co-creative Drawing Workflows

This article presents the outcomes from a mixed-methods study of drawing practitioners (e.g., professional illustrators, fine artists, and art students) that was conducted in Autumn 2018 as a preliminary investigation for the development of a physical human-AI co-creative drawing system. The aim of the study was to discover possible roles that technology could play in observing, modeling, and possibly assisting an artist with their drawing. The study had three components: a paper survey of artists' drawing practises, technology usage and attitudes, video recorded drawing exercises and a follow-up semi-structured interview which included a co-design discussion on how AI might contribute to their drawing workflow. Key themes identified from the interviews were (1) drawing with physical mediums is a traditional and primary way of creation; (2) artists' views on AI varied, where co-creative AI is preferable to didactic AI; and (3) artists have a critical and skeptical view on the automation of creative work with AI. Participants' input provided the basis for the design and technical specifications of a co-creative drawing prototype, for which details are presented in this article. In addition, lessons learned from conducting the user study are presented with a reflection on future studies with drawing practitioners

Modelling variable communication signal strength for experiments with multi-robot teams

Reliable communication is a critical factor for ensuring robust performance of multi-robot teams. A selection of results are presented here comparing the impact of poor network quality on team performance under several conditions. Two different processes for emulating degraded network signal strength are compared in a physical environment: modelled signal degradation (MSD), approximated according to increasing distance from a connected network node (ie robot), versus effective signal degradation (ESD). The results of both signal strength processes exhibit similar trends, demonstrating that ESD in a physical environment can be modelled relatively well using MSD

A Non-linear Model for Predicting Tip Position of a Pliable Robot Arm Segment Using Bending Sensor Data

Using pliable materials for the construction of robot bodies presents new and interesting challenges for the robotics community. Within the EU project entitled STIFFness controllable Flexible & Learnable manipulator for surgical Operations (STIFF-FLOP), a bendable, segmented robot arm has been developed. The exterior of the arm is composed of a soft material (silicone), encasing an internal structure that contains air-chamber actuators and a variety of sensors for monitoring applied force, position and shape of the arm as it bends. Due to the physical characteristics of the arm, a proper model of robot kinematics and dynamics is difficult to infer from the sensor data. Here we propose a non-linear approach to predicting the robot arm posture, by training a feed-forward neural network with a structured series of pressures values applied to the arm's actuators. The model is developed across a set of seven different experiments. Because the STIFF-FLOP arm is intended for use in surgical procedures, traditional methods for position estimation (based on visual information or electromagnetic tracking) will not be possible to implement. Thus the ability to estimate pose based on data from a custom fiber-optic bending sensor and accompanying model is a valuable contribution. Results are presented which demonstrate the utility of our non-linear modelling approach across a range of data collection procedures

Multi-agent Task Allocation for Fruit Picker Team Formation (Extended Abstract)

Multi-agent task allocation methods seek to distribute a set of tasks fairly amongst a set of agents. In real-world settings, such as fruit farms, human labourers undertake harvesting tasks, organised each day by farm manager(s) who assign workers to the fields that are ready to be harvested. The work presented here considers three challenges identified in the adaptation of a multi-agent task allocation methodology applied to the problem of distributing workers to fields. First, the methodology must be fast to compute so that it can be applied on a daily basis. Second, the incremental acquisition of harvesting data used to make decisions about worker-task assignments means that a data-backed approach must be derived from incomplete information as the growing season unfolds. Third, the allocation must take “fairness” into account and consider worker motivation. Solutions to these challenges are demonstrated, showing statistically significant results based on the operations at a soft fruit farm during their 2020 and 2021 harvesting seasons

Multi-agent task allocation for harvest management

Multi-agent task allocation methods seek to distribute a set of tasks fairly amongst a set of agents. In real-world settings, such as soft fruit farms, human labourers undertake harvesting tasks. The harvesting workforce is typically organised by farm manager(s) who assign workers to the fields that are ready to be harvested and team leaders who manage the workers in the fields. Creating these assignments is a dynamic and complex problem, as the skill of the workforce and the yield (quantity of ripe fruit picked) are variable and not entirely predictable. The work presented here posits that multi-agent task allocation methods can assist farm managers and team leaders to manage the harvesting workforce effectively and efficiently. There are three key challenges faced when adapting multi-agent approaches to this problem: (i) staff time (and thus cost) should be minimised; (ii) tasks must be distributed fairly to keep staff motivated; and (iii) the approach must be able to handle incremental (incomplete) data as the season progresses. An adapted variation of Round Robin (RR) is proposed for the problem of assigning workers to fields, and market-based task allocation mechanisms are applied to the challenge of assigning tasks to workers within the fields. To evaluate the approach introduced here, experiments are performed based on data that was supplied by a large commercial soft fruit farm for the past two harvesting seasons. The results demonstrate that our approach produces appropriate worker-to-field allocations. Moreover, simulated experiments demonstrate that there is a “sweet spot” with respect to the ratio between two types of in-field workers