Two years ago, the Croatian Mine Action Center–Center for Testing Development and Training Ltd. tested two Japanese dual-sensor systems for humanitarian demining in Croatia. The test’s results show that these detection systems can potentially increase the accuracy of mine-detecting operations, but several improvements to the sensors may be required before the systems are fully effective

Gaal, Mate

Gulle, Dieter

Takahashi, Kazunori

English

James Madison University

Journal of Conventional Weapons Destruction Volume 13 Issue 3 The Journal of ERW and Mine Action Article 28 November 2009 Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in Croatia Kazunori Takahashi Leibniz Institute for Applied Geophysics Mate Gaal Federal Institute for Materials Research and Testing Dieter Gulle Federal Office of Defense Technology and Procurement Follow this and additional works at: https://commons.lib.jmu.edu/cisr-journal  Part of the Defense and Security Studies Commons, Emergency and Disaster Management Commons, Other Public Affairs, Public Policy and Public Administration Commons, and the Peace and Conflict Studies Commons Recommended Citation Takahashi, Kazunori; Gaal, Mate; and Gulle, Dieter (2009) "Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in Croatia," The Journal of ERW and Mine Action : Vol. 13 : Iss. 3 , Article 28. Available at: https://commons.lib.jmu.edu/cisr-journal/vol13/iss3/28 This Article is brought to you for free and open access by the Center for International Stabilization and Recovery at JMU Scholarly Commons. It has been accepted for inclusion in Journal of Conventional Weapons Destruction by an authorized editor of JMU Scholarly Commons. For more information, please contact dc_admin@jmu.edu. 66         research and development | the journal of ERW and mine action | fall 2009 | 13.3 13.3 | fall 2009 | the journal of ERW and mine action | research and development         67In October 2007, the Croatian Mine Action Center–Center for Testing Development and Training Ltd. (HCR-CTRO), with assistance from the German Federal Institute for Materials Research and Testing (BAM), test-ed two sensor systems: the Advanced Landmine Imaging System, devel-oped by Tohoku University, Japan, and the Gryphon, developed by Tokyo Institute for Technology, Japan. Both systems employ commercial metal detectors (the ALIS with CEIA MIL-D1 and the Gryphon with Minelab F3) and ground-penetrating radar. The metal detector only indicates the presence of metal; it cannot determine if the metal is a mine. The GPR in-dicates objects with a shape that could resemble a mine. The operator of the system decides whether to reject the metal clutter. Together, the sys-by Kazunori Takahashi [ Leibniz Institute for Applied Geophysics ], Mate Gaal [ Federal Institute for Materials Research and Testing ] and Dieter Gülle [ Federal Office of Defense Technology and Procurement ]Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in CroatiaTwo years ago, the Croatian Mine Action Center–Center for Testing Development and Training Ltd. tested two Japanese dual-sensor systems for humanitarian demining in Croatia. The test’s results show that these detection systems can potentially increase the accuracy of mine-detecting operations, but several improvements to the sensors may be required before the systems are fully effective. tems improve the productivity of demining operations.1,2,3 This article dis-cusses the test’s results, the systems’ performances and the data analysis.Test ConditionsThe complete report, detailing the conditions, procedures and results of the two dual-sensor systems is available online.4 The test was carried out at the Benkovac test site in Croatia where previous metal-detector tri-als have taken place, such as the Systematic Test and Evaluation of Metal Detector (STEMD) trial.5 Three soils are available at this site: red bauxite (Lane 1), neutral clay (Lane 3), and red bauxite with neutral stones (Lane 5, local soil), as shown in the figures on the next page.6,7 The Gryphon dual-sensor system evaluated in the test. The Gryphon team consists of two buggies: one with a metal detector (near side) and one with GPR (far side).ALL PHOTOS AnD GRAPHICS COURTESY OF THE AUTHORSLane 1: red bauxite. Lane 3: neutral clay. Lane 5: red bauxite with neutral stones.Table 1: Differences in categorization of sources of alarms for stand-alone met-al detectors and dual sensors.*1 The metal-detector scan in ALIS is performed in the conventional manner (i.e., manual scan with sound alert), while the metal detector on Gryphon is scanned by the robot arm and the detection is according to visual interpretations of the metal-detector image. *2 Scans of the GPR in ALIS are performed for each metal-detector alarm. Gryphon scans both sensors for an area approx. 1 x 2m at once and interpre-tations are done for each scanned area, i.e., Gryphon scans all the area with both sensors.Figure 1: Operation procedure of the dual-sensor systems.Blind tests were conducted in these lanes with real, ren-dered-safe mines (11 were PMA-2 and nine were PMA-3 mines) and metal clutter. The target layout was the same as that in the International Test and Evaluation Program for Humanitarian Demining STEMD trial5 with addition-al small pieces of various metals (nine per lane) placed on the ground surface. Thus, each lane comprised a total of 38 buried targets.Sources of true positivesSources of false positivesStand alone metal detector mines, metals soilMetal detector as part of dual sensor mines metals, soil1Takahashi et al.: Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in CroatiaPublished by JMU Scholarly Commons, 200968         research and development | the journal of ERW and mine action | fall 2009 | 13.3 13.3 | fall 2009 | the journal of ERW and mine action | research and development         69Three deminers from the Croatian Mine Action Center served as operators of the ALIS; the developer had trained them together 10 workdays prior to the test. In the test, each deminer went through each of the three lanes once. The developer’s team of five to six per-sons operated the Gryphon.Both dual-sensor systems employ a metal detector as a primary sen-sor and a GPR as a secondary sensor; the metal detector first detects all the metal objects, and then the GPR identifies objects suspected to be landmines. In the test, red markers indicated positions of objects detect-ed by the metal detector and yellow markers indicated positions of ob-jects confirmed as landmines by the GPR, so that those detections could be classified later. The operation procedure is schematically illustrated in Figure 1 on the previous page. After each run, all the markers’ positions were measured and com-pared to the real positions of mines measured when they were planted. A circular area around a target, called a halo, is defined according to CWA 14747-1:2003.8 A marker is considered a hit (true positive) if it falls into the area, and a marker is counted as a false alarm (false positive) if it is placed outside the area. A target with no markers in its halo is counted as a miss (false negative).Data AnalysisProbability of detection has been commonly used to evaluate perfor-mance of metal detectors. Since the dual sensors employ two kinds of de-tection, two kinds of POD can be defined. The POD for a metal detector is defined as: where              and             denote the number of mines buried and the number of mines detected by the metal detector, respectively. The other POD for GPR is defined as:where          is number of mines correctly confirmed after the use of the metal detector and the GPR. Metal-detector alarms not caused by mines and GPR alarms incorrectly confirmed as mines are considered false calls for a dual sensor. Therefore, the false-alarm rates for the metal detector and for the GPR, respectively, are defined as:where                ,              ,               and               are alarm numbers caused by metal and soil, reported by the metal detector and the GPR, and A denotes an area searched. Note that the definition of the false alarm for metal detectors in this analysis is different from that of stand-alone met-al detectors. Alarms from metal pieces are normally counted as true positives for stand-alone metal detectors, while they are considered false positives for dual sensors because of the detectors’ objective, which is to differentiate between landmines and other objects. The different catego-rizations of alarms are summarized in Table 1. In order to observe how much efficiency is improved, FAR reduction,      , is introduced as follows:If all the false alarms are rejected,              takes a value 1.The GPR could fail to detect mines. It can be acceptable to miss false alarms; however, miss-discrimination for mines threatens the lives of end-users. In order to see the frequency of missed mines, probability of detection reduction,             , is defined as:If the GPR does not reject any mines found by the metal detector, the value becomes 0. Related to the reduction, the discrimination ratio for mines is introduced to find how often mines are correctly confirmed.The ratio is actually given by one minus POD reduction:  Confidence limits of 95% of POD, FAR and their reductions are pro-vided in Figures 2 and 3 (see page 68) to illustrate the accuracy of the es-timations. They are calculated assuming the binomial and the Poisson distributions for POD and FAR, respectively.9,10,11In the following section, the defined quantities above are incorporat-ed into various figures that display the results of the dual-sensor system performance evaluations.ResultsAlthough performances of metal detectors and GPRs can be quite dif-ferent in various types of soil, the results in the three lanes are analyzed to-gether in this article to show the overview. An analysis of each soil can be found in the trial report.4 The ALIS operator whose results differed greatly from the others had his results excluded as an outlier.Figure 2 (see page 68) shows the receiver-operating-characteristic dia-gram in which probability of detections are plotted against false-alarm rates and the 95% confidence limits. Each device has two plots, one from using only the metal detector (primary sensor) and one from using both sensors (metal detector and GPR). It can be observed that the FARs by metal detectors (squares) are shifted toward the left by using GPRs (cir-cles), meaning that FARs are reduced significantly. However, at the same time, reductions of PODs also occur for both devices, which should not happen for safety reasons.The absolute levels of POD and FAR are basically given by the metal detectors, which are commercial ones in both systems. The reductions of FAR and POD can be seen as contributions of the GPR. The reductions are plotted in Figure 3. In this figure, FAR reductions are plotted with respect to POD reductions; therefore an ideal dual-sensor system that can perfectly discriminate targets gives a plot on the upper left portion of the graph. If a system uses random chance to determine whether a mine is present, the plot lies on the diagonal line. Both the ALIS and the Gryphon give plots above the diagonal line, therefore the GPRs in both systems are contributing to the decision-making. The Gryphon gives larger FAR reduction than the ALIS; however, the POD reduction is also larger than that by the ALIS. The difference in the POD reductions is not Figure 2: Receiver-operating-characteristic diagram with 95% confidence limits for each device.Figure 3: FAR reduction versus POD reduction for each device.Figures 4a and 4b: POD given by the metal detector (solid lines) and by both sensors (dotted lines), and discrimination ratio for mines (dotted-broken lines) with respect to target burial depths, given by the ALIS (top) and the Gryphon (bottom).The ALIS dual-sensor system evaluated in the test. 2Journal of Conventional Weapons Destruction, Vol. 13, Iss. 3 [2009], Art. 28https://commons.lib.jmu.edu/cisr-journal/vol13/iss3/2870         research and development | the journal of ERW and mine action | fall 2009 | 13.3 13.3 | fall 2009 | the journal of ERW and mine action | research and development         71In 2007, the Director of the Canadian Centre for Mine Action Tech-nologies received a request to investigate a potentially promising heat-treatment process to extend the operational life of humanitarian-deminer visors through removal of scratches from the field of view. The heat-treatment procedure was developed by undergraduate students as part of a product-design course and was published in The Journal of Mine Action.3 The authors of that article noted that further testing would be required to determine whether the visor properties were ad-versely affected by the scratch-repair procedure. In order to allow for Blast Testing of Visors Used for Humanitarian Demining  by Captain Charlene Fawcett [ DRDC Suffield ]This article discusses experimental results from blast testing of Security Devices Ltd. polycarbonate visors used by humanitarian deminers. Visors used in the blast testing fell into one of three categories: new visors, manually scratched visors, and scratched and heat-gun-repaired visors. Results show that the visors in all three categories failed to meet the draft international standard for blast testing1 relevant at the time, that further research is required to establish pressure profiles for the standard charge size being tested, and that the proposed heat- treatment method does appear to degrade the blast resistance of the visor used in the test.2 an independent assessment of the technique, the authors provided a detailed outline of the procedure in the article that readers could fol-low independently. Trial Objectives and MethodologyThe objective of this research was to assess the blast and ballistic per-formance of deminer visors before and after heat treatment. To ensure compatibility with the original student project, the same type of visors were obtained from Security Devices Ltd.Figure 1: Testing platform and positioning rig.ALL GRAPHICS COURTESY OF DRDC SUFFIELDKazunori Takahashi is a Research Scien-tist formerly employed with Federal Institute for Materials Research and Testing (BAM) and currently at Leibniz Institute for Applied Geophysics. His research activities include development and evaluation of dual-sensor systems for humanitarian demining, signal processing for GPR, and reliability analy-sis of non-destructive testing methods.Kazunori Takahashi Research ScientistLeibniz Institute for Applied GeophysicsStilleweg 230655 Hannover / GermanyTel: +49 511 643 3572Fax: +49 511 643 3665E-mail: kazunori.takahashi@liag-hannover.deWeb site: http://www.liag-hannover.deDieter Gülle is a retired colonel who has worked since 1995 in different positions con-nected with minefield survey, clearance operations and testing of clearance equip-ment. He represented Germany in the In-ternational Test and Evaluation Program for Humanitarian Demining as head of the ITEP Secretariat. He is currently a consul-tant to the Federal Office of Defense Tech-nology and Procurement in Germany.Dieter Gülle ConsultantFederal Office of Defense Technology and ProcurementStubenrauchstr. 65dBerlin 12487 / Germany Tel: +49 306 366 418E-mail: dieter.guelle@xyz.deMate Gaal is a Research Scientist at BAM who previously worked on testing metal de-tectors and dual sensors. He earned his doc-torate on experimental design for tests of demining equipment. Currently he works on developing sensors for ultrasonic testing.Mate Gaal Research ScientistBAM – Federal Institute for Materials     Research and TestingUnter den Eichen 8712205 Berlin / GermanyTel: +49 30 8104 3174Fax: +49 30 8104 1845E-mail: mate.gaal@bam.deWeb site: http://www.bam.deso significant considering the 95% confidence limit, but devices for demining must avoid the POD reduction as much as possible. The re-sults suggest that the Gryphon can reduce FAR more than the ALIS can. However, the abso-lute level of FARs is almost the same as shown in Figure 2 (see page 68) and the larger FAR re-duction is due to a larger number of false alarms given by the metal detector implemented in the Gryphon. Therefore, performances of the whole system as dual sensor in terms of FAR can be characterized as almost the same.Figure 4 (see page 69) shows probability of detections given by the metal detector and by both sensors, along with the discrimination ratio with respect to depth for each device. As the theory in the Das and McFee article12 states and former trials verified, the PODs giv-en by the metal detectors are decreasing with depth. Since the GPRs are always used after the metal detectors, the PODs used by the dual sensors cannot exceed those by the met-al detectors. It can be observed that the PODs by both sensors positively correlate with the PODs by the metal detectors.Furthermore, discrimination ratios tend to increase with depth in these results. This fact cannot be determined conclusively be-cause the number of mines belonging to each depth class is small and the estimation would not be sufficiently accurate. This tendency sup-ports a common theory that GPR has difficul-ties in detecting shallowly buried targets since reflections from the ground surface mask those from targets.13 However, this observed tendency is not as strong as expected; both systems achieved about 0.7 of the discrimina-tion ratio at the depth range from 0–3cm, so the theory cannot clearly be confirmed. This may be because both sensors measured data of GPR as images in terms of horizontal slice and this type of representation may be good at depicting small changes close to the surface, unlike only one-time signals or a vertical slice.Conclusions and DiscussionThe results of the test campaign for the dual-sensor systems tell us that those sys-tems reduced false-alarm rates significantly by more than one-half. However, the systems also reduced probability of detections, which must be avoided in real clearance operations. Useful-ness of the dual sensors may strongly depend on improvements with POD.The full report4 stated that the three de-miners who worked on the ALIS achieved different results in terms of POD, FAR and working hours. The variation may be caused by the way the deminers interpret the output of the sensor and make decisions when oper-ating the ALIS. The visual interpretation of images and decision-making process are en-tirely subject to the operators themselves. In order to avoid unstable and/or unexpected re-sults, further developments/improvements, such as an automatic-recognition algorithm, are recommended.Unfortunately, it was not possible to use stand-alone metal detectors at the same time as a benchmark, making a direct comparison of dual sensors to stand-alone metal detectors unavailable. However, one can roughly com-pare the detectors to those from the STEMD trial,5 taking into account additional metals. The ALIS and the Gryphon needed approxi-mately five and nine minutes, respectively, to survey an average of one square meter. It can be roughly estimated that the ALIS may be two to three times slower and the Gryphon may be four to five times slower than stand-alone metal detectors.14 Even if the search speed in this test is slower than for a stand-alone met-al detector, it is possible that these dual sen-sors would accelerate the clearance operation in total, because rejected alarms from metals would reduce the need for excavation or could be rapidly excavated. Increased search speed would also multiply these benefits. Another dual-sensor trial in Germany was carried out in September 2009 by the Interna-tional Test and Evaluation Program for Hu-manitarian Demining and led by the German Federal Office of Defense Technology and Pro-curement.15 The results are being analyzed and we hope that a more detailed evaluation of dual-sensor performance will be available soon. The authors acknowledge Mr. N. Pavković and Mr. T. V. B. Vondracek from HCR-CTRO for managing the test. We also thank the devel-opers and deminers that participated.See Endnotes, Page 793Takahashi et al.: Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in CroatiaPublished by JMU Scholarly Commons, 2009

Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in Croatia

https://commons.lib.jmu.edu/cgi/viewcontent.cgi?article=1484&context=cisr-journal

Data Analysis and Performance Evaluation of Japanese Dual-sensor Systems Tested in Croatia

Abstract

Similar works

Full text

Available Versions

James Madison University