Automated Dial Reading (ADR) using image processing is a challenging task that has to deal with the dynamics of real time environment. Literature contains limited research work for ADR that is based on background subtraction, object tracking, and pattern recognition. These methods suffer from dynamic environment such as: varying light intensity, poor resolution, and vibrations in capturing device. A valuable contribution to the existing dial reading approaches is made in this paper by deploying convolution method which plays a significant role in needle/hand recognition within a dial. Proposed dial reading approach is successfully used and tested reading analogue aircraft instruments facilitated by the Flight Guardian  (FG) project for automated reading of the cockpit devices in dynamic environments. Performance is evaluated by statistical analysis of the experimental results that proved the robustness of the proposed method

Amina, Mahd

Ansell, Darren

Khan, Wasiq

Kuru, Kaya

CLoK

1  Automated Aircraft Instrument Reading Using Real Time Video Analysis  Wasiq Khan, Darren Ansell, Kaya Kuru, Mahdi Amina Intelligent Systems, School of Engineering University of Central Lancashire Preston, PR1 2HE, United Kingdom Wkhan4@uclan.ac.uk   Abstract—Automated Dial Reading (ADR) using image processing is a challenging task that has to deal with the dynamics of real time environment. Literature contains limited research work for ADR that is based on background subtraction, object tracking, and pattern recognition. These methods suffer from dynamic environment such as: varying light intensity, poor resolution, and vibrations in capturing device. A valuable contribution to the existing dial reading approaches is made in this paper by deploying convolution method which plays a significant role in needle/hand recognition within a dial. Proposed dial reading approach is successfully used and tested reading analogue aircraft instruments facilitated by the Flight Guardian1 (FG) project for automated reading of the cockpit devices in dynamic environments. Performance is evaluated by statistical analysis of the experimental results that proved the robustness of the proposed method.  Keywords-intelligent pattern recognition; aircraft instruments; automated dial reading; image processing; video analysis; object detection I.  INTRODUCTION  Literature contains a variety of the image processing based approaches for ADR of utility meters. Image subtraction also referred as background subtraction, is one of the most common techniques that have extensive use for object detection. The image frame at current time is subtracted from a reference frame to identify the moving object in the scene. A number of approaches exist that are beyond the literal meaning of background subtraction. The most commonly used are running Gaussian average, temporal median filter, mixture of Gaussians kernel density estimation, sequential kernel density approximation, co-occurrence of image variations, and Eigen-backgrounds [1]. Background subtraction has been used in vast application areas that include geo-technical engineering [2], astrophotography [3], medical science, computed tomography scanners, ultrasound techniques, digital radiography, and nuclear magnetic resonance [4]. Some ADR approaches used an image subtraction method which is limited to fixed size static images [5, 6, 7, and 8]. Similarly, Hough transforms for line detection [9] and feature based object detection [10, 11] have also been used for the ADR purpose.                                                           1 NATEP funded project. See Acknowledgment section for further details on FG project. An interesting approach for ADR is presented by [12] that uses polar representation of the dial gauge image to identify the needle. The input image is pre-processed using a Gaussian filter to reduce the illumination effects. Next circular shape is identified by calculating the maximum correlation between rotated versions of connected components. The identified area is converted to polar coordinate system to find out radius of the circle. The start and the needle position of the dial gauge are estimated using the column projection of the polar image. Finally, minimum projected value is selected as identified needle pixels. Although, this approach is more generic comparatively, however, it does not work well in the existence of specular highlights [12]. Furthermore, a number of assumptions are made such as gauge starting point, circle size, and needle length which make it specific to limited dials and gauges.  Research work in [13] used Scale-Invariant Feature Transform (SIFT) features to match the target image with a stored template for electro-mechanical meter reading. The advantage of SIFT features is the independency of scale and orientation. The best matched features were used as the outliers that represent the target needle within the dial. The centroid of the identified pixels is taken as the central point of the needle and is used to measure the angle/position with respect to center of the image. Experiments are conducted over 141 images with 88% accuracy. A major issue with this approach is the existence of specular highlights. In case of white color of needle, the specular highlights will be identified as the most similar outliers. Moreover, the commutation time is also comparatively higher for this approach. Image registration is alternative approach that has been used for ADR.  Literature consist numerous methods for image registration that includes geometrical transformation, image dimensionality, degree of interaction, modalities based registration, optimization procedures, and subject and object based registration [14]. A detailed review and implementation of the aforementioned image registration methods is presented in [15]. However, an intensity based method is the most common that has been used in a number of application areas, specifically for medical imaging. Similarly, rigid registration approach is also popular due to its simplicity and applicability in image-guided surgery [15]. Despite of the efficient performance of image registration in diverse areas of applications, it is not helpful in a real time flight environment 2  due to high computational cost. In addition, this method produces poor performance in varying light intensity and light reflection conditions. In the proposed approach for dial reading in the FG system, cockpit instruments that include airspeed and engine speed (RPM) are monitored by camera devices that produce continuous streams of video data which is analyzed to measure the instrument states at a fixed time interval. A number of filter operations are applied to instant images by performing a special operation called ‘convolution’ with a static vector (kernel). The convolution uses a 1 dimensional kernel that is convolved with a limited set of pixels around the center of the input image. Maximum convolution is achieved at those pixels which are occupied by the target object (i.e. needle). II. PROPOSED METHOD A number of processes are sequentially amalgamated to develop the proposed automatic dial reading approach. The primary objective of dial reading task is to provide instantaneous state information about the cockpit instruments to the FG decision support system for the prediction of hazardous situations. The overall dial reading task is divided into a number of sequential processes as shown in Fig. 1. Data Acquisition&Pre-processingAngle MeasurementFor Identified PixelsAirspeed Video StreamRPM Video StreamAngle Calibration w.r.tAirspeed & RPM InstrumentsMeasured Angle forAirspeed & RPM NeedlesOutput Airspeed & RPM ReadingsProcessed Images Figure 1. Framework for automated dial reading In the first step, two fixed position camera devices are used to record the continuous video streams for airspeed and RPM dial. The recorded video streams are then forwarded to a pre-processing unit to convert the video streams as instantaneous images with 0.5 Hz frequency and forward to ADR process. Because of the circular shape and needle locations in both dials, focus of the camera devices is fixed at center positions of the cockpit instruments as shown in Fig. 2. It can be observed that all pixels are black within a limited circular range (red circle in Fig. 2) around the image center except the white pixels occupied by the needle. Therefore, only a sub-image (within the red circle) is needed to identify the needle position in the cockpit instruments.    In the second step, a convolution function with a static ‘kernel’ is used to identify the needle and hence its position in the dial. Usually, kernel matrix size is predefined and the matrix entries represent the filter type. However; due to nature of the problem in FG system, a one-dimensional kernel vector is convolved with a limited set of pixels occupied by sub-image described above.   Figure 2.  Estimation of needle angular position within the airspeed dial using convolution between circular area around the image center and a one dimensional kernal To enhance the process in terms of execution time, the sub-image pixels are converted to one-dimensional vector as shown in Fig. 3. As the entire pixels occupied by the target needle are white, the entries of the kernel matrix are set to ‘1’ such that it produces maximum convolution output for only white pixels within the cropped sub-image.  Figure 3. Conversion of two dimensional image pixels to one dimensional vector which is convlived with kernal in time domain to get the output convolution vector The entire convolution process between selected sub-image pixels ‘ x ’and a kernel ‘ h ’ over a pre-defined interval is expressed as a pseudo code shown in Fig. 4.    Figure 4. Pseudo code for convolution between cropped image and kernel 3  III. PERFORMANCE EVALUATION  For the proposed ADR performance evaluation, multiple scenarios are generated based on the dynamic situations and the performance is validated using a number of metrics. The evaluation methodology entails experiments for multiple environmental settings such as (1) Camera perspective, (2) Camera resolution, (3) Light reflections, (4) Varying light intensities, and (5) Real time and Lab environmental setting. In a Lab environment, the ADR performance is validated using different methodologies. In first method, the resulted dial reading values are compared with the corresponding instrument readings (i.e. ground truth) extracted during the simulated flight in Microsoft Flight Simulator (FSX). Representative analogue dials were used in the lab, driven by FSX. In addition to Lab testing, the proposed system is tested during live flight environment using two camera devices fixed at different perspectives with respect to cockpit instruments. Dial reading performance is compared using simultaneous data streams from both camera devices. This strategy validates the system accuracy for mounting the camera devices at dynamic position/perspective w.r.t. the target cockpit instrument. Also it validates the dial reading ability in a more reliable and trustable way, compared to a single camera’s performance. In addition to individual performance of proposed dial reading system, overall performance of the FG system (including decision support components) is also evaluated in Lab environment as well as in real time flight environment which is out of scope of this paper.  Figure 5. Ground truth Vs ADR based airSpeed measurements Performance of the proposed ADR is evaluated by statistical measurements (i.e. p-value, root mean squired error, R squared value) between the actual readings for airspeed gauge (i.e. ground truth) extracted from FSX during the simulated flight and predicted airspeed measurements by automatic dial reading system. Fig. 5 demonstrates a comparison between ground truth data stream provided by FSX and predicted measurements by ADR system. It can be analyzed that the proposed system performed robustly in terms of precise overlapping the ground-truth stream. Also, in the proposed approach, convolution of limited numbers of image pixels with the kernel provides robustness in terms of computation time. The proposed approach takes only 0.2 seconds from image acquisition to digital output.  In addition to the aforementioned evaluation method, the ADR is further evaluated by setting up two cameras at different angles with respect to the airspeed gauge in a real time flight environment and simultaneously measuring the needle/hand angle with respect to y-axis. Fig. 6 demonstrates the angular measurements provided by ADR while taking simultaneous video streams by two cameras. The robustness of the proposed ADR can be observed by the accurate overlapping of output streams.   Figure 6. Angular measurements of the needle for multiple camera devices An important factor to be taken into account is setting the tolerance level for airspeed measurements. In terms of the FG system, the tolerance level can be quite flexible because of the flexibility in airspeed ranges. However, the tolerance of 15 knots for airspeed is set in the proposed hazardous situation assessment. Table 1 addresses the detailed statistical results for aforementioned scenarios to evaluate the airspeed measurement performances with varying tolerance level. It can be observed that with a tolerance of 12 knots, 100% significance level (i.e. p=1) for the airspeed measurement is achieved which proves the robustness of the system. In both scenarios, the performance gradually decreases with the decrement in the tolerance level. However, this decrease is due to the manual mounting of the camera devices with respect to the cockpit device to be observed. In practice, 100% accuracy can be achieved for the low tolerance values by setting up the camera devices horizontally and vertically aligned to the cockpit device. For example, a flexible mounting clip would provide rotational and lateral flexibility. Alternatively, a reference point in the cockpit may be used to automatically rotate/align the video streams/images with respect to y-axis as discussed in ‘Conclusions and Future Directions’ Section. 4  TABLE I.  STATISTICAL METRICS FOR AIRSPEED MEASUREMENTS                                                                                            Tolerance (Knots) Cam-Device  Vs.  FSX-Stream (ASp Measurements)  Cam-Device1  Vs.  Cam-Device2 (ASp Measurements) p-value RMSE R2  p-value RMSE R2 12 1 0 0.9999 0.98 0.87 0.9998 11 0.98 0.64 0.9995 0.96 1.25 0.9997 10 0.96 1.05 0.9987 0.94 1.55 0.9995 9 0.95 1.18 0.9984 0.91 1.88 0.9994 8 0.90 1.63 0.9970 0.84 2.41 0.9991 In addition to the aforementioned statistical and graphical results, the dial reading functionality is tested in the presence of sunlight reflections. As the needle color in airspeed instrument is white, it was challenging task to differentiate the light reflections and actual needle as discussed in [12, 13]. However, the convolution method along with the circular constraints makes it more accurate by minimizing the probability of existence of light reflections in such a tiny area of the cockpit. Fig. 7 shows the test cases for the FG performance in the presence of light reflection in the cockpit. It is analyzed that the proposed ADR performed robustly in dynamic light intensity conditions by differentiating the specular highlights and needle efficiently.     Figure 7. Proposed system performance in the presence of specular highlights 5  IV. CONCLUSIONS AND FUTURE DIRECTIONS This paper presents a new approach for automated dial reading using fixed position camera. A number of existing techniques were reviewed and compared with the proposed method which is based on linear convolution. Performance of the proposed method is evaluated using the corresponding validation methodologies while testing the proposed method in dynamic situations. Despite of the robust ADR performance of the proposed approach, there are some limitations that are needed to be considered as future opportunities for improvement. Firstly, the proposed system is restricted to fixed position camera focusing exactly the center of dial. However, a generalized approach to automatically identify the desired gauge in the cockpit, then rotate it with respect to a fixed reference point (e.g. digit recognition can be used for each gauge), and finally measure the needle position within the recognized dial using the approaches introduced in this work, would be preferable. Secondly, the problem of vibrations can be resolved by using a fixed reference pattern within the gauge/dial with respect to center of the dial leading to a more flexible and reliable system. ACKNOWLEDGEMENT Special thanks to The Great Circle Ltd who were the industrial partner, collaborating with UCLan on the development of FG. The FG project was part-funded by the National Aerospace Technology Exploitation Programme (NATEP). The NATEP has been funded by £23m from the Department of Business, Innovation & Skills through its Advanced Manufacturing Supply Chain Initiative (AMSCI) fund, administered by Birmingham City Council & Finance Birmingham. The NATEP is managed by the UK's national aerospace body ADS, supported by the UK regional aerospace alliances – FAC, NWAA, MAA & WEAF. REFERENCES [1] M. Piccardi, “Background Subtraction Techniques: A Review,” IEEE Int. Conf. on Sys. Man and Cyb, vol. 4, Pp. 3099- 3104., 10-13 October 2004.  [2] K. J. Scheibengraber, H. J. Pincus, “Optimal Fourier Analysis of Surface Features of Interest in Geotechnical Engineering: Image Subtraction and Correlation,” Geotechnical Applications of Remote Sensing and Remote Data Transmission, ASTM STP 967, A. I. Johnson and C. B. Pettersson, Eds., American Society for Testing and Materials, Philadelphia, Pp. 151-162, 1988. [3] R. Fisher, S. Perkins, A. Walker,  and E. Wolfart, “Pixel Subtraction,” The University of Edinburhg, School of Informatics, 2003. Online Available at: http://homepages.inf.ed.ac.uk/rbf/HIPR2/pixsub.htm. [4] N. Taleb, L. Jetto, “Image registration for applications in Digital Subtraction Angiography,” Control Engineering Practice, Elsevier, vol. 6, No. 2, Pp. 227-238, 1998. [5] B. Hemming, H. Lehto, “Calibration of Dial Indicators using Machine Vision,” In Meas. Sci. Technol,vol. 13, Pp. 45-49, 2002. [6] H. Feng, J. Zhao, “Application Research of Computer Vision in the Auto-Calibration of Dial Gauges,” International Conference on Computer Science and Software Engineering, vol. 2, Pp. 845-848, 2008.  [7] S. Zhao, B. Li, J. Yuan, and G. Cui, “Research on Remote Meter Automatic Reading Based on Computer Vision,” IEEE/PES Transmission and Distribution Conf.& Exhibition: Asia and Paciﬁc Dalian, 2005. [8] S. Moussa, “Efﬁcient Way of Reading Rotary Dial Utility Meter Using Image Processing,” MS Thesis, Online Available at: http://etd.ohiolink.edu/view.cgi?acc num=case1258511722. [9] D. Antolovic, “Review of the Hough Transform Method With an Implementation of the Fast Hough Variant for Line Detection,” Technical Report, School of Informatics and Computing, Indiana University, 2008. Online Available at: http://www.cs.indiana.edu/pub/techreports/TR663.pdf. [10] R. Szeliski, “Computer Vision, Algorithms and Applications,” Chapter 4: Feature Detection and Matching, Texts in Computer Science, Springer-Verlag London Limited, Pp. 213-269, 2011. [11] R. Fergus, P. Perona  and A. Zisserman, “Object Class Recognition by Unsupervised Scale-Invariant Learning,” 2003, Online Available at: http://people.csail.mit.edu/fergus/papers/fergus03.pdf.  [12] M. K. Gellaboina, G. Swaminathan and V. Venkoparao, “Analog Dial Gauge Reader for Handheld Devices ”, ACS-Advanced Technology Labs Honeywell, 2003. [13] R. O. Vega, G. S. Ante, L. E. F. Morales and H. Sossa, “Automatic Reading of Electro-mechanical Utility Meters,” 12th Mexican International Conference on Artificial Intelligence, Pp. 164-170, 2013. [14] J. B. A. Maintz and M. A. Viergever, “A survey of medical image registration,” Med. Image Anal, vol. 2, Pp. 1–36, 1998. [15] J. M. Fitzpatrick, D. L. G. Hill and C. R. Maurer, “Handbook of Medical Imaging,” Bellingham, WA: SPIE Press, 2000, vol. 2, Ch. 8: Image Registration, Pp. 449–506. 

Automated Aircraft Instrument Reading Using Real Time Video Analysis

Abstract

Similar works

Full text

Available Versions

CLoK