International audienceThis article is concerned with the on-line counting of some harmful insects in videos in the framework of in situ video-surveillance that aims at the early detection of pest attacks in greenhouse crops. The challenges mainly concern the tiny resolution and color contrast of the insects of interest in the videos, the outdoor issues and the quasi-real time constraints. Thus, we propose a complete video-processing chain that can cope with such challenges quite satisfactorily while yielding an acceptable performance. The system has been validated off-line against many recorded videos for the whiteflie species (one potential harmful pest species)

Bechar, Ikhlef

Moisan, Sabine

INRIA a CCSD electronic archive server

HAL Id: inria-00515624https://hal.inria.fr/inria-00515624Submitted on 8 Sep 2010HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.On-line counting of pests in a greenhouse usingcomputer visionIkhlef Bechar, Sabine MoisanTo cite this version:Ikhlef Bechar, Sabine Moisan. On-line counting of pests in a greenhouse using computer vision.VAIB 2010 - Visual Observation and Analysis of Animal and Insect Behavior, Bob Fisher, Aug 2010,Istanbul, Turkey. ￿inria-00515624￿On-line counting of pests in a greenhouse using computer visionIkhlef Bechar and Sabine MoisanEPI Pulsar, INRIA Sophia Antipolis-Mediterraneeikhlef.bechar@sophia.inria.fr, sabine.moisan@sophia.inria.frAbstractThis article is concerned with the on-line countingof some harmful insects in videos in the framework ofin situ video-surveillance that aims at the early detec-tion of pest attacks in greenhouse crops. The challengesmainly concern the tiny resolution and color contrast ofthe insects of interest in the videos, the outdoor issuesand the quasi-real time constraints. Thus, we proposea complete video-processing chain that can cope withsuch challenges quite satisfactorily while yielding anacceptable performance. The system has been validatedoff-line against many recorded videos for the whitefliespecies (one potential harmful pest species).1 IntroductionComputer vision has been successfully applied invarious real-life surveillance applications [4] because ofits various advantages among which one can mention:non-invasiveness, autonomy, accuracy and objective-ness. Recently, motivated by the growing interest fora sustainable Agriculture and a safer alimentation, itsarea of application has been extended to the automationof the pest monitoring process in greenhouse crops[1, 6]One commonly used idea consists in endowing agreenhouse with a network of video-cameras that senseduring day time some sticky traps (cf. Fig.1(a)) dis-tributed, for instance, uniformly in the greenhouse oraccording to some priori knowledge about focus regionsof harmful insects. Then a video processing makesit possible to recognize the trapped insects belongingto the harmful species of interest, and to describe sta-tistically their spatiotemporal presence within a green-house. Such an information is used to predict a pestattack in order to allow the cultivator to use the fightingactions that fall at the early stage.Despite the used big image size, the objects of in-terest (harmful insects) look in the videos only liketiny and lowly contrasted objects with unclearly de-fined borders (see Fig.1(a)-(b)), moreover because ofhazardous in situ events (illumination variations, re-flectances, outliers), this makes it very challenging torecognize on-line the insects of interest. Therefore, wepropose to combine some image processing operationswith some classical video processing algorithms in or-der to achieve a final operative system for the on-linecounting of pests.2 Frame-wise detection of harmful insectsInitially, the zone of the sticky trap with respect toeach video is extracted automatically once and for allfrom the first video-frame by using some mere assump-tions about similarity of color and compacity. There-fore, all subsequent image and video-processing oper-ations are performed only in this zone of interest in avideo (cf. Fig.2(a)).In the reminder, we firstly describe the two-step al-gorithm that we developed for the recognition of theharmful insects of interest in individual video-frames,and secondly we show how it can be speeded upand made more robust using some standard video-processing algorithms in order to achieve an on-line op-erative video-surveillance application.2.1 RGB-into-gray linear transformationThe first step consists in transforming each RGBvideo-frame into a gray image where the zones of in-sects of interest will look as much brighter as possiblethan their surrounding background. To do so, we con-sider a linear transformation of the form I := trR +tgG+ tbB, and we estimate the linear coefficients tr, tgand tb in such a way to maximize the following (SNR)ratio between the mean contrast over a sample of NIinsect intensities SI ={(Ri, Gi, Bi); i = 1, · · · , NI}and the mean contrast over a sample of NB backgroundintensities SB ={(rj , gj , bj); j = 1, · · · , NB}as fol-(a) (b)Figure 1. (a) An example of a typi-cal video-frame (resolution: 1280 × 960px). The central yellow zone correspondsto the zone of the sticky trap, and thetrapped harmful insects (whiteflies) corre-spond to the tiny white spots fixed on itssurface ; (b) A zoom on the imagette ofone insect of interest in the video-frame.lows:∑NIi=1(trRi + tgGi + tbBi)2∑NBj=1(trrj + tggj + tbbj)2→ maxtr,tg,tb(1)Both color samples SI and SB were previously ob-tained by manual sampling of insect imagettes in sam-ple videos (here, a dataset of 500 insect imagettes isused) in such a way that the linear coefficients tr, tg,tb can be estimated off-line by solving the generalizedeigen value problem which is deduced from (1).(a) (b)Figure 2. (a) Projection of an RGBvideo-frame on the recognized zone of thesticky trap ; (b) Its RGB-into-gray transfor-mation.2.2 Recognition of potential locations of in-sects of interest in a video-frameNow, we would like to automatically extract thebright spots in a given gray video-frame (cf. Fig.2(b))that may correspond to the insects of interest in theoriginal video-frame. Such a problem of automatic ex-traction of small spots from unspecific backgrounds hasbeen extensively studied in various application contextssuch as fluorescence videomicroscopy [3], genetic spotarray images [5], and feature extraction in active vi-sion amounting to extracting interest points in video-frames[2]. In this paper, we propose a new parametricapproach for the extraction of tiny spots in videos andwe apply it to the problem of the recognition of harfmulinsects of interest in videos.To do so, we model a bright spot as a contrasted rect-angular pattern R := R(r, w, θ, f(·, ·)), with r and wstanding for its half-width and its half-length respec-tively, θ which stands for its tilt angle, and f(x, y)which stands for a 2D function describing the gray in-tensity level at any point (x, y) of the plane. For sim-plicity’s sake, we shall assume that f(x, y) is a piece-wise constant function which is equal to a constant h+ainside R, and to a constant a outside R as follows:f(x, y) =h + a, if |x cos(θ) + y sin(θ)| ≤ w and| − x sin(θ) + y cos(θ)| ≤ r;a, otherwise.where h stands for the gray contrast of R and a standsfor the gray level of its surrounding background. Now,in order to yield a continuously differentiable 2D im-age which can show “singularities”, namely local max-ima at the rectangular zones of interest in the imageand which can then be extracted efficiently by using ageometric differential technique, for instance by usingthe Karush-Kuhn-Tucker (KKT) local maximality cri-terion, we propose to convolve f(x, y) with a gaussiankernel Kσ(x, y) = 1√2πσ e−(x2+y2)2σ2 to obtain the fol-lowing 2D scale space intensity profilefσ(x, y) = h×(Φσ(x cos(θ) + y sin(θ) + w)− Φσ(x cos(θ) + y sin(θ)− w))(Φσ(−x sin(θ)+y cos(θ)+ r)−Φσ(−x sin(θ)+y cos(θ)− r))with Φσ(t) = 1√2πσ∫ t−∞ e−u22σ2 du .We show then that a condition on σ which is of theform: σ ≥ max(w,r)√3is sufficient in order for fσ(x, y)to show a clearly defined local maximum at the centroidof R. Therefore σ is chosen as follows σ := `1√3, with`1 standing for the prior about maximum half-width ormaximum half-length of an insect of interest in a video,and the latter can be obtained from a sample of manu-ally segmented insect imagettes.Now, one expresses the KKT sufficient conditionsof local maximimality of fσ(u, v) at the centroid point(0, 0) of R as follows:∇fσ(0, 0) = 0 (2)∇2fσ(0, 0) < 0 (3)20 50 100 150 200 250 300 350 400 450 5000.511.522.533.544.55x 10−3training sample indexsample value (s*i )(a)Figure 3. Sample values ofmin(s1(σ,w, r, h), s2(σ,w, r, h)) computedfor 500 insects imagettes and arranged inan ascending order.with ∇ and ∇2 standing respectively for the gradientand the Hessian operators with respect to x and y. Fordetection purposes, we shall focus more on the sec-ond KKT condition of local maximality (3) of fσ(u, v)at the centroid of some rectangular pattern which canalso be seen as a measure of its saliency. Such a cri-terion amounts then to saying that both eigen valuesof ∇2fσ(0, 0) are negative. However, because of thepresence of noise in the image, such a criterion shouldbe replaced with a robust one consisting in saying thatboth eigen values of ∇2fσ(0, 0) should lie below acertain negative threshold ’−s∗’, which in plain lan-guage means that only salient enough rectangular pat-terns which may correspond to insects of interest insome video-frame should be considered. Note, in par-ticular, the invariance of such criterion w.r.t. to θ. Asa consequence, knowledge of the tilt angle θ of a rect-angular pattern is not necessary. In practise, s∗ maybe estimated off-line from manually segmented sampleimagettes of the insect of interest (cf. Fig.1(b)) as weshall explain it hereafter. First of all, one shows easilythat the respective eigen values of ∇2fσ(0, 0) are givenby:s1(σ, w, r, h) :=−2hw√2πσ3e− w22σ2(2Φσ(r)− 1)s2(σ, w, r, h) :=−2hr√2πσ3e− r22σ2(2Φσ(w)− 1)Therefore, given a sample of N manually segmentedimagettes (insect pixels versus background pixels), onecan easily estimate the parameters r, w and h (we es-timate h as the difference between the mean gray in-tensity over insect pixels and the mean gray intensityover background pixels in an imagette) to obtain, us-ing the latter formulae, a set of candidate values fors∗ as follows S∗ = {si∗, i = 1, · · · , N}, with si∗ =min(si1(σ,w, r, h), si2(σ,w, r, h)). Hence, one mightchoose s∗ as the minimum value of S∗ (cf. Fig.3) .Thus, the algorithm for the extraction of potential in-sect locations in a given video frame that we propose(a) (b)Figure 4. (a) A video-frame ; (b) Extrac-tion of the insect locations in it.consists in two main steps. Firstly, extract the cen-troid locations by using the fact that for some pixel(x, y), the eigen values of matrix ∇2fσ(x, y) lie below−s∗. Secondly, such locations are grouped by means ofthe connected components algorithm and a quick localconquer-and-merge segmentation strategy in the RGBframe allows to complete the detections (cf. Fig.4(c)).3 An on-line video-processing solution3.1 The basic online video-processing ideaIn order to achieve an on-line video-surveillance so-lution, we exploit the fact that the objects of interest in avideo (harmful insects) may appear in it only in a sparseway during day time. Thus, initially a video-frame is di-vided into a number of k×k slightly overlapping virtualimage patches (e.g. k = 10) that can be processed veryquickly by the insect detector described above. Then, aquick background subtraction algorithm in the spirit ofthe Mixture-of-Gaussians algorithm (MoG) [7] whichruns permanently with respect to each video allows todetect any significant intensity changes in it. When achange is detected at some pixel, the pixel undergoesa second test referred as the insect presence detectiontest which will try to classify it as ’likely’ or ’unlikely’to be an insect of interest pixel. This is achieved bylearning off-line the space of color intensities of insectsof interest by means of a Principal Components Analy-sis (PCA). The frame patch with maximum number ofpixels that passed the insect presence detection test isthen submitted to the insect detector described in sec-tion 2 in order to realize independently a precise detec-tion of any recently trapped insect of interest, providedthat this number exceeds some user-defined threshold.A detection is validated if and only if it intersects with aminimal number of pixels that passed the presence de-tection test. Since a recently trapped insect may manageto displace slowly from its initial location in the stickytrap (due to its continuous efforts to escape from it), so3in order to keep track of it without having to detect itagain in subsequent video-frames, hence a quick TBD(Track-Before-Detect) type tracking algorithm is used.Its principle is explained in the following subsection.3.2 Frame-to-Frame insect trackingObviously, one needs firstly to specify which dis-criminative feature vector Y of an object of interestto use for the tracking task. However, since preciseshape information about the insects is lacking in therecorded videos, hence one is left with no many optionsfor choosing Y . Thus, we propose to choose the featurevector Y of any detected insect of interest as follows.Firstly, a square bounding box Bt is computed aroundthe insect in the video-frame ’t’ where it has been firstlydetected as the smallest square bounding box whichcontains it. A feature vector Y t0 is then computed w.r.t.such a bounding box as follows. Firstly, take all RGBintensities lying inside Bt, transform them into gray in-tensities by using the linear transformation as explainedin subsection 2.1, then sort them in an ascending orderand finally arrange them in a vector Y t0 . One notes inparticular the invariance of Y t0 with respect to a rotationor a translation of the insect. In the next video-frames,such a bounding box is updated by sliding its centroidin the neighorhood of its current position (typically, in asmall square window F of some predefined size H×Hpx.), and for each slid bounding box Bt+1, a new fea-ture vector Y t+1 is computed in the same way as ex-plained for Y t0 . Bt might then be updated by choosingfor example which Bt+1 maximizes a similarity crite-rion between Y t and Y t+1 of the form S(Y t, Y t+1) :=min{ ‖Y t0 )‖‖Y t+1‖ ,‖Y t+1‖‖Y t0 ‖}Y tTY t+1‖Y t‖‖Y t+1‖ , whereY tTY t+1‖Y t‖‖Y t+1‖stands for the correlation ratio between vectors Y t andY t+1, and min{ ‖Y t0 )‖‖Y t+1‖ ,‖Y t+1‖‖Y t0 ‖}is a factor which fa-vors vectors with comparable modules so as to fightmore robustly against noise. However, such a track-ing criterion S(Y t, Y t+1) has shown in our tests to besomewhat sensitive to image noise, hence we proposeto replace the maximum of the criterion S(Y t, Y t+1)over F with its empirical mean.4 Method’s evaluationThe currently developed version of our vision appli-cation has been tested off-line against 8 video sequencesrepresenting the whitefly species and recorded underrealistic in situ conditions during daytime for periodsranging from 20 minutes to 1 hour (the insects trap pro-cess was accelerated by placing the cameras near highlyinfested plants). An assessment of the results againstground truth revealed that the false positive rate is negli-gible (namely, one false positive has been found againstabout 250 found others), whereas the false negative rateis of order of 3%, and the latter concerns mainly someinsects that were not detected by the algorithm becauseof their too low signal to noise ratio or because of ahighly illuminated neighboring background. Finally,the system has shown to be able to process videos at asatisfactory video frame rate for the application (nearly2 frames every 3 seconds).5 ConclusionWe developed a new full on-line computer visionprototype for in-situ pest monitoring and we showedits feasibility for the case of one potential harmful pestspecies (the whitefly), nevertheless, its extension toother harmful pest species of interest (e.g. the green-fly species) is straightforward. An in situ systems val-idation at a larger scale has been scheduled with oneproject partner (INRA).6 AcknowledgementsThe authors are grateful to CREAT (La Gaude) andINRA (Avignon) for their participation in this work.7 REFERENCESReferences[1] C. Bauch and T. Rath. A prototype of a vision basedsystem for measurements of whitefly infestation. ActaHorticulturae (ISHS), 0(691):773–780, Jan. 2005.[2] E. Z. G. Loy. A fast radial symmetry transform for de-tecting points of interest. 7 th Euproean Conference onComputer Vision, Oct.. 2002.[3] H. B. H.-Y. Chen, N. Brandle and H. Lapp. Robustspot fitting for genetic spot array images. Image Pro-cessing,Proceedings. International Conference on Pat-tern Recognition and Image Processing, 3:412–415, Oct..2000.[4] L. V. N. Haering, A. Peter and A. Lipton. The evolutionof video surveillance: an overview. Machine Vision andApplications, 19:279–290, Jun. 2008.[5] J. Olivo-Marin. Extraction of spots in biological im-ages using multiscale products. Pattern Recognition,35(9):1989–1996, Sept. 2002.[6] V. M. P. Boissard and S. Moisan. A cognitive vision ap-proach to early pest detection in greenhouse crops. Int.Journ.of Comp. Elect. in Agric., 2(62):81–93, Jul. 2008.[7] C. Stauffer and W. Grimson. Background mixture modelsfor real-time tracking. CVPR, Aug. 1999.4

On-line counting of pests in a greenhouse using computer vision

https://hal.inria.fr/inria-00515624/document

On-line counting of pests in a greenhouse using computer vision

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server