A spatio-temporal recurrent network for salmon feeding action recognition from underwater videos in aquaculture

Recent developments have shown that Deep Learning approaches are well suited for Human Action Recognition. On the other hand, the application of deep learning for action or behaviour recognition in other domains such as animal or livestock is comparatively limited. Action recognition in fish is a particularly challenging task due to specific research challenges such as the lack of distinct poses in fish behavior and the capture of spatio-temporal changes. Action recognition of salmon is valuable in relation to managing and optimizing many aquaculture operations today such as feeding, as one of the most costly operations in aquaculture. Inspired by these application domains and research challenges we introduce a deep video classification network for action recognition of salmon from underwater videos. We propose a Dual-Stream Recurrent Network (DSRN) to automatically capture the spatio-temporal behavior of salmon during swimming. The DSRN combines the spatial and motion-temporal information through the use of a spatial network, a 3D-convolutional motion network and a LSTM recurrent classification network. The DSRN shows an accuracy that is suitable for industrial use in prediction of salmon behavior with a prediction accuracy of 80%, validated on the task of predicting Feeding and NonFeeding behavior in salmon at a real fish farm during production. Our results show that the DSRN architecture has high potential in feeding action recognition for salmon in aquaculture and for applications domains lacking distinct poses and with dynamic spatio-temporal changes.publishedVersio

Robotic Handling of Compliant Food Objects by Robust Learning from Demonstration

The robotic handling of compliant and deformable food raw materials, characterized by high biological variation, complex geometrical 3D shapes, and mechanical structures and texture, is currently in huge demand in the ocean space, agricultural, and food industries. Many tasks in these industries are performed manually by human operators who, due to the laborious and tedious nature of their tasks, exhibit high variability in execution, with variable outcomes. The introduction of robotic automation for most complex processing tasks has been challenging due to current robot learning policies. A more consistent learning policy involving skilled operators is desired. In this paper, we address the problem of robot learning when presented with inconsistent demonstrations. To this end, we propose a robust learning policy based on Learning from Demonstration (LfD) for robotic grasping of food compliant objects. The approach uses a merging of RGB-D images and tactile data in order to estimate the necessary pose of the gripper, gripper finger configuration and forces exerted on the object in order to achieve effective robot handling. During LfD training, the gripper pose, finger configurations and tactile values for the fingers, as well as RGB-D images are saved. We present an LfD learning policy that automatically removes inconsistent demonstrations, and estimates the teacher's intended policy. The performance of our approach is validated and demonstrated for fragile and compliant food objects with complex 3D shapes. The proposed approach has a vast range of potential applications in the aforementioned industry sectors.Comment: 8 pages, 7 figures,IROS 201

Case Nortura/Norilia.Improving the utilisation of co-streams in poultry processing

Industrialised chicken production is far from organic agriculture prinicples. Still of interest is a more sustainable utilisation of by-products, e.g. hydrolysation of feathers for proteins, or extraction of food grade oil from chicken bones. Such approaches were studied in the bioeconomy-project "CYCLE" (2013-2017)

Computer vision for quality grading in fish processing

High labour costs, due to the existing technology that still involves a high degree of manually based processing, incur overall high production costs in the fish processing industry. Therefore, a higher degree of automation of processing lines is often desirable, and this strategy has been adopted by the Norwegian fish processing industry to cut-down production costs. In fish processing, despite a slower uptake than in other domains of industry, the use of computer vision as a strategy for automation is beginning to gain the necessary maturity for online grading and evaluation of various attributes related to fish quality. This can enable lower production costs and simultaneously increase quality through more consistent and non-destructive evaluation of the fish products. This thesis investigates the possibility for automation of fish processing operations by the application of computer vision. The thesis summarises research conducted towards the development of computer vision-based methods for evaluation of various attributes related to whole fish and flesh quality. A brief summary of the main findings is presented here. By application of computer vision, a method for the inspection of the presence of residual blood in the body cavity of whole Atlantic salmon was developed to determine the adequacy of washing. Inadequate washing of fish after bleeding is quite common in commercial processing plants. By segmenting the body cavity and performing a colour analysis, it was shown that the degree of bleeding correlated well with colour parameters, resulting in correct classification of the fish with residual blood. The developed computer vision-based classifier showed a good agreement with the manual classification of the fish that needed re-washing. The proposed method has potential to automate this type of inspection in fish processing lines. In addition, a computer vision-based classifier for quality grading of whole Atlantic salmon in different grading classes, as specified by the industrial standard, was developed. In the proposed solution, after segmentation of the salmon from the image scene, with the use of the computer vision techniques, it was possible to extract non-redundant geometrical features describing the size and shape of fish. Based on these features, a classifier was developed for classification of fish into respective grading classes. The average correct rate of classification was in good agreement with the manual labelling, and the method has a potential for grading of Atlantic salmon in fish processing lines. Regarding fillet grading, a computer vision-based sorting method for Atlantic salmon fillets according to their colour score was developed. The method and classifier/matching algorithm was based on the present industrial standard NS 9402 for evaluation of fillets by colour according to Roche Cards. As a result, fillets or parts of fillets, could be classified into different colour grades. This is important for the industry since different markets tend to have different preferences for fillet colour. This classification method is suitable for online industrial purposes. In addition, the method gives colour evaluation of fresh and smoked fillets in the CIELab space, similar to the L, a, and b values generated by a Minolta Chromameter, for different parts of fillets as well as for the entire fillet. The advantage of the computer vision-based method derives from the flexibility in the choice of the size of the region of interest of the fillet for colour measurement, as opposed to the Chromameter, where the Minolta generated values are obtained by interrogating a very small area of the fillet (8 mm). The method can also be used for detection of colour non-uniformities (discoloration) in both fresh and smoked fillets. A method for computer vision-based measurements and monitoring of transient 2D and 3D changes in the size and shape of fillets during the rigor process and ice storage was developed. The method successfully measured the size (length, width, area) and shape (roundness) of Atlantic salmon and cod fillets, and monitored changes to these during ice storage with high precision. This was demonstrated by comparison of the exhausted and anesthetized fillets. By laser scanning of the fillet, it was possible to obtain size changes in the height (mm) and area of the fillet in cross-section. The method can be used not only for size and shape analysis of fillets but also for other fish products, both in on-line, as well as off-line conditions as a tool for monitoring 2D/3D size and shape changes. The method can also be used for determination of fillet yield measured in thickness, which is an important parameter for the industry. Together with the colour grading ability, this method can also be used for full feature evaluation and classification of any fish or food product from a single image (colour, size and shape in 2D/3D). If filleting of fish is done pre-rigor, care should be exercised during colour grading since transient colour changes occur in the post-mortem period. As these changes are more pronounced than those that occur during ice storage, incorrect colour grading can occur. The computer vision method developed for evaluation of colour changes in fillets during rigor, ice storage, and due to effects of perimortem handling stress was considered as the most suitable method for industrial purposes when compared to both the Minolta Chromamater and sensory analysis by a panel. A computer vision-based method for evaluation of fresh and smoked fillets with respect to bleeding was developed. This form of evaluation is important for the industry as residual blood in fillets may lead to reduced visual acceptance of the product. The method was considered suitable for the purpose of this type of evaluation. The developed computer vision methods have potential for automation of the mentioned grading operations in the commercial fish processing lines. Application of the proposed solutions would lower the production costs, while simultaneously increasing the quality of the products through a more consistent and non-destructive evaluation of these products

A spatio-temporal recurrent network for salmon feeding action recognition from underwater videos in aquaculture

Recent developments have shown that Deep Learning approaches are well suited for Human Action Recognition. On the other hand, the application of deep learning for action or behaviour recognition in other domains such as animal or livestock is comparatively limited. Action recognition in fish is a particularly challenging task due to specific research challenges such as the lack of distinct poses in fish behavior and the capture of spatio-temporal changes. Action recognition of salmon is valuable in relation to managing and optimizing many aquaculture operations today such as feeding, as one of the most costly operations in aquaculture. Inspired by these application domains and research challenges we introduce a deep video classification network for action recognition of salmon from underwater videos. We propose a Dual-Stream Recurrent Network (DSRN) to automatically capture the spatio-temporal behavior of salmon during swimming. The DSRN combines the spatial and motion-temporal information through the use of a spatial network, a 3D-convolutional motion network and a LSTM recurrent classification network. The DSRN shows an accuracy that is suitable for industrial use in prediction of salmon behavior with a prediction accuracy of 80%, validated on the task of predicting Feeding and NonFeeding behavior in salmon at a real fish farm during production. Our results show that the DSRN architecture has high potential in feeding action recognition for salmon in aquaculture and for applications domains lacking distinct poses and with dynamic spatio-temporal changes.publishedVersio

Automatisk etterkontroll av restpinnebein i pre-rigor laksefileter

Det ikke mulig å avgjøre om en laksefilet er beinfri etter pinnebeinplukking ved bare å se og kjenne på filetens overflate. Det kan fortsatt sitte igjen avrevne bein nede i fiskekjøttet. Disse laksebeina er sannsynligvis ikke helsefarlige, men kan være ubehagelig og medføre tap av matlyst hos mange forbrukere. Dette er et viktig argument for å fjerne alle bein i lakseprodukter som blir markedsført som beinfrie. Ved a skjære i fileten er det mulig å kontrollere om fileten er beinfri, men det er naturligvis ikke en aktuell metode, verken som 100 % -kontroll eller som stikkprøvekontroll av større parti fileter. I mange tiår har røntgen vært benyttet som kvalitetskontroll med hensyn på restbein i hvitfisk. Det har ikke vært vanlig å bruke røntgen på laks fordi laksebeina har gitt for dårlig kontrast i røntgenbildet. Imidlertid er det nå på markedet maskiner med lavenergirøntgen (LER) som med ny og bedre sensorteknologi gjør det mulig å kontrollere og automatisk sortere ut både pre- og post-rigor laksefileter med for mye bein. Dette prosjektet, Automatisk etterkontroll av restpinnebein i pre-rigor laksefileter, har vist at avbildning med LER gir god nok kontrast til "online" deteksjon av pinnebein i størrelser som de færreste forbrukere er i stand til å finne når de spiser et godt laksemåltid.Fiskeri og havbruksnpublishedVersio

A spatio-temporal recurrent network for salmon feeding action recognition from underwater videos in aquaculture

Recent developments have shown that Deep Learning approaches are well suited for Human Action Recognition. On the other hand, the application of deep learning for action or behaviour recognition in other domains such as animal or livestock is comparatively limited. Action recognition in fish is a particularly challenging task due to specific research challenges such as the lack of distinct poses in fish behavior and the capture of spatio-temporal changes. Action recognition of salmon is valuable in relation to managing and optimizing many aquaculture operations today such as feeding, as one of the most costly operations in aquaculture. Inspired by these application domains and research challenges we introduce a deep video classification network for action recognition of salmon from underwater videos. We propose a Dual-Stream Recurrent Network (DSRN) to automatically capture the spatio-temporal behavior of salmon during swimming. The DSRN combines the spatial and motion-temporal information through the use of a spatial network, a 3D-convolutional motion network and a LSTM recurrent classification network. The DSRN shows an accuracy that is suitable for industrial use in prediction of salmon behavior with a prediction accuracy of 80%, validated on the task of predicting Feeding and NonFeeding behavior in salmon at a real fish farm during production. Our results show that the DSRN architecture has high potential in feeding action recognition for salmon in aquaculture and for applications domains lacking distinct poses and with dynamic spatio-temporal changes

Towards automated sorting of Atlantic cod (Gadus morhua) roe, milt, and liver - Spectral characterization and classification using visible and near-infrared hyperspectral imaging

Technological solutions regarding automated sorting of food according to their quality parameters are of great interest to food industry. In this regard, automated sorting of fish rest raw materials remains as one of the key challenges for the whitefish industry. Currently, the sorting of roe, milt, and liver in whitefish fisheries is done manually. Automated sorting could enable higher profitability, flexibility in production and increase the potential for high value products from roe, milt and liver that can be used for human consumption. In this study, we investigate and present a solution for classification of Atlantic cod (Gadus morhua) roe, milt and liver using visible and near-infrared hyperspectral imaging. Recognition and classification of roe, milt and liver from fractions is a prerequisite to enabling automated sorting. Hyperspectral images of cod roe, milt and liver samples were acquired in the 400–2500 nm range and specific absorption peaks were characterized. Inter- and intra-variation of the materials were calculated using spectral similarity measure. Classification models operating on one and two optimal spectral bands were developed and compared to the classification model operating on the full VIS/NIR (400–1000 nm) range. Classification sensitivity of 70% and specificity of 94% for one-band model, and 96% and 98% for two-band model (sensitivity and specificity respectively) were achieved. Generated classification maps showed that sufficient discrimination between cod liver, roe and milt can be achieved using two optimal wavelengths. Classification between roe, milt and liver is the first step towards automated sorting.acceptedVersio

Fast and accurate GPU accelerated, high resolution 3D registration for robotic 3D reconstruction of compliant Food objects

If we are to develop robust robot-based automation in primary production and processing in the agriculture and ocean space sectors, we have to develop solid vision-based perception for the robots. Accurate vision-based perception requires fast 3D reconstruction of the object in order to extract the geometrical features necessary for robotic manipulation. To this end, we present an accurate, real-time and high-resolution ICP-based 3D registration algorithm for eye-in-hand configuration using an RBG-D camera. Our 3D reconstruction, via an efficient GPU implementation, is up to 33 times faster than a similar CPU implementation, and up to eight times faster than a similar library implementation, resulting in point clouds of 1 mm resolution. The comparison of our 3D reconstruction with other ICP-based baselines, through trajectories from 3D registration and reference trajectories for an eye-in-hand configuration, shows that the point-to-plane linear least squares optimizer gives the best results, both in terms of precision and performance. Our method is validated for the eye-in-hand robotic scanning and 3D reconstruction of some representative examples of food items and produce of agricultural and marine origin

Grasping Unknown Objects by Coupling Deep Reinforcement Learning, Generative Adversarial Networks, and Visual Servoing

International audienc