Design og styring av smarte robotsystemer for applikasjoner innen biovitenskap: biologisk prøvetaking og jordbærhøsting

This thesis aims to contribute knowledge to support fully automation in life-science applications, which includes design, development, control and integration of robotic systems for sample preparation and strawberry harvesting, and is divided into two parts. Part I shows the development of robotic systems for the preparation of fungal samples for Fourier transform infrared (FTIR) spectroscopy. The first step in this part developed a fully automated robot for homogenization of fungal samples using ultrasonication. The platform was constructed with a modified inexpensive 3D printer, equipped with a camera to distinguish sample wells and blank wells. Machine vision was also used to quantify the fungi homogenization process using model fitting, suggesting that homogeneity level to ultrasonication time can be well fitted with exponential decay equations. Moreover, a feedback control strategy was proposed that used the standard deviation of local homogeneity values to determine the ultrasonication termination time. The second step extended the first step to develop a fully automated robot for the whole process preparation of fungal samples for FTIR spectroscopy by adding a newly designed centrifuge and liquid-handling module for sample washing, concentration and spotting. The new system used machine vision with deep learning to identify the labware settings, which frees the users from inputting the labware information manually. Part II of the thesis deals with robotic strawberry harvesting. This part can be further divided into three stages. i) The first stage designed a novel cable-driven gripper with sensing capabilities, which has high tolerance to positional errors and can reduce picking time with a storage container. The gripper uses fingers to form a closed space that can open to capture a fruit and close to push the stem to the cutting area. Equipped with internal sensors, the gripper is able to control a robotic arm to correct for positional errors introduced by the vision system, improving the robustness. The gripper and a detection method based on color thresholding were integrated into a complete system for strawberry harvesting. ii) The second stage introduced the improvements and updates to the first stage where the main focus was to address the challenges in unstructured environment by introducing a light-adaptive color thresholding method for vision and a novel obstacle-separation algorithm for manipulation. At this stage, the new fully integrated strawberry-harvesting system with dual-manipulator was capable of picking strawberries continuously in polytunnels. The main scientific contribution of this stage is the novel obstacle-separation path-planning algorithm, which is fundamentally different from traditional path planning where obstacles are typically avoided. The algorithm uses the gripper to push aside surrounding obstacles from an entrance, thus clearing the way for it to swallow the target strawberry. Improvements were also made to the gripper, the arm, and the control. iii) The third stage improved the obstacle-separation method by introducing a zig-zag push for both horizontal and upward directions and a novel dragging operation to separate upper obstacles from the target. The zig-zag push can help the gripper capture a target since the generated shaking motion can break the static contact force between the target and obstacles. The dragging operation is able to address the issue of mis-capturing obstacles located above the target, in which the gripper drags the target to a place with fewer obstacles and then pushes back to move the obstacles aside for further detachment. The separation paths are determined by the number and distribution of obstacles based on the downsampled point cloud in the region of interest.Denne avhandlingen tar sikte på å bidra med kunnskap om automatisering og robotisering av applikasjoner innen livsvitenskap. Avhandlingen er todelt, og tar for seg design, utvikling, styring og integrering av robotsystemer for prøvetaking og jordbærhøsting. Del I omhandler utvikling av robotsystemer til bruk under forberedelse av sopprøver for Fourier-transform infrarød (FTIR) spektroskopi. I første stadium av denne delen ble det utviklet en helautomatisert robot for homogenisering av sopprøver ved bruk av ultralyd-sonikering. Plattformen ble konstruert ved å modifisere en billig 3D-printer og utstyre den med et kamera for å kunne skille prøvebrønner fra kontrollbrønner. Maskinsyn ble også tatt i bruk for å estimere soppens homogeniseringsprosess ved hjelp av matematisk modellering, noe som viste at homogenitetsnivået faller eksponensielt med tiden. Videre ble det foreslått en strategi for regulering i lukker sløyfe som brukte standardavviket for lokale homogenitetsverdier til å bestemme avslutningstidspunkt for sonikeringen. I neste stadium ble den første plattformen videreutviklet til en helautomatisert robot for hele prosessen som forbereder prøver av sopprøver for FTIR-spektroskopi. Dette ble gjort ved å legge til en nyutviklet sentrifuge- og væskehåndteringsmodul for vasking, konsentrering og spotting av prøver. Det nye systemet brukte maskinsyn med dyp læring for å identifisere innstillingene for laboratorieutstyr, noe som gjør at brukerne slipper å registrere innstillingene manuelt.Norwegian University of Life SciencespublishedVersio

An autonomous strawberry‐harvesting robot: Design, development, integration, and field evaluation

This paper presents an autonomous robot capable of picking strawberries continuously in polytunnels. Robotic harvesting in cluttered and unstructured environment remains a challenge. A novel obstacle‐separation algorithm was proposed to enable the harvesting system to pick strawberries that are located in clusters. The algorithm uses the gripper to push aside surrounding leaves, strawberries, and other obstacles. We present the theoretical method to generate pushing paths based on the surrounding obstacles. In addition to manipulation, an improved vision system is more resilient to lighting variations, which was developed based on the modeling of color against light intensity. Further, a low‐cost dual‐arm system was developed with an optimized harvesting sequence that increases its efficiency and minimizes the risk of collision. Improvements were also made to the existing gripper to enable the robot to pick directly into a market punnet, thereby eliminating the need for repacking. During tests on a strawberry farm, the robots first‐attempt success rate for picking partially surrounded or isolated strawberries ranged from 50% to 97.1%, depending on the growth situations. Upon an additional attempt, the pick success rate increased to a range of 75–100%. In the field tests, the system was not able to pick a target that was entirely surrounded by obstacles. This failure was attributed to limitations in the vision system as well as insufficient dexterity in the grippers. However, the picking speed improved upon previous systems, taking just 6.1 s for manipulation operation in the one‐arm mode and 4.6 s in the two‐arm mode

Strawberry detection using a heterogeneous multi-processor platform

Over the last few years, the number of precision farming projects has increased specifically in harvesting robots and many of which have made continued progress from identifying crops to grasping the desired fruit or vegetable. One of the most common issues found in precision farming projects is that successful application is heavily dependent not just on identifying the fruit but also on ensuring that localisation allows for accurate navigation. These issues become significant factors when the robot is not operating in a prearranged environment, or when vegetation becomes too thick, thus covering crop. Moreover, running a state-of-the-art deep learning algorithm on an embedded platform is also very challenging, resulting most of the times in low frame rates. This paper proposes using the You Only Look Once version 3 (YOLOv3) Convo-lutional Neural Network (CNN) in combination with utilising image processing techniques for the application of precision farming robots targeting strawberry detection, accelerated on a heterogeneous multiprocessor platform. The results show a performance acceleration by five times when implemented on a Field-Programmable Gate Array (FPGA) when compared with the same algorithm running on the processor side with an accuracy of 78.3% over the test set comprised of 146 images

Design and control of a loader mechanism for the NMBU agricultural robot

Despite the development of new technologies, manual labour still continuous to play a large role within most modern agricultural operations, especially during harvest. Consequently, there is an increasing demand for new machines to reduce labour as a mean to limit costs, while increasing efficiency in a sustainable manner. This thesis concern itself with the design of a mechanism and control system for a robot arm that can substitute workers in logistical operations during strawberry harvest. More specifically, by lifting berry crates onto a robot platform and transporting them from the fields and to the packaging facilities. The robot arm is to be mounted on the platform composing a vehicle- manipulator system. As this thesis is connected to a general research project on agricultural robotics at the Norwegian University of Life Sciences, the chosen platform is the associated field robot Thorvald II. The thesis is divided into two parts, where Part I concerns the mechanical design of the robot arm, while Part II propose a system for controlling the mechanism. The design development process has involved assessments of available solutions before selecting components on the basis of controllability, mechanical properties and costs. The process of selection in Part II is however, based on finding solutions that are compatible with the robot platform’s network (Controller Area Network) and operating system (Robotic Operating System). Part I: Design and Mechanics The design of the robot arm presented in this thesis begun with a preliminary feasibility study conducted by Bjurbeck in September 2016. Following the assessment of this study, the robot arm is designed to have two degrees of freedom operating in the xz-plane. When mounted on the platform, the arm will be free to operate in a 3-dimensional space, as the platform moves in x and y-direction, and rotates around the z-axis. The arm is assembled from two parallel link pairs made from rectangular aluminium tubes, and a revolute and prismatic joint. Both joints are actuated by LinAk LA36 linear electric actuators. The end effector of the arm is a gripper head designed to grasp the handles of the strawberry crate. The gripper head is self-aligning with the crate’s orientation in order to reduce the precision of control needed to envelop and grasp the crate. The frame of the gripper head is made from aluminium angle profiles and sheet metal. A worm drive DC motor actuate the gripper claws via a double link mechanism. Part II: Modeling and Control The geometry of the design presented in Part I is modelled mathematically and the inverse kinematics solved analytically. The kinematics will be used in future implementation of a position control system. Two RoboteQ SDC2160 dual-channel controllers are chosen to control all four actuator mo- tors. The linear actuators are controlled in closed loop position tracking mode with absolute feedback. The gripper motor is controlled in open loop mode with end stop switches detecting the position of the claws. Experiments was conducted to match the controllers with the actuator motors. The experiments revealed firmware issues with the controller. The experiments also affirmed the controller need a script to operate the actuators efficiently. The thesis provides the foundations to build a prototype and write an operating script to test the mechanical design and control system.Til tross for den stadige utviklingen av ny teknologi spiller manuelt arbeid fortsatt en stor rolle i moderne landbruk, særlig i innhøsting. På grunn av den store arbeidkraften som trengs er det en stadig større etterspørsel etter nye maskiner som kan redusere behovet for manuelt arbeid for å redusere utgifter og effektivisere gårdsbruk på en bærekraftig måte. Denne masteroppgaven omhandler det mekaniske designet og reguleringssystemet til en robotarm laget for å kunne erstatte arbeidere i oppgaver tilknyttet logistikk ved innhøsting av jordbær. Dette gjøres ved at armen løfter kasser med bær opp på en robotplattform som transporterer kassene fra jordet og til et pakkeri. Robotarmen er da montert oppå plattformen. Siden oppgaven er tilknyttet et forskningsprosjekt i landbruksrobotikk ved Norges miljø- og biovitenskapelige universitet, var det naturlig å velge den universitetets robot Thorvald II som plattform.submittedVersionM-MP