Usage of statistical classifiers, namely AdaBoost and its modifications, is very common in object detection and pattern recognition.
Performance of such classifiers strongly depends on low level features they use. This paper presents an experimental
implementation of the Local Binary Patterns (LBP) that uses SIMD instructions for acceleration. The experiments shows that
the proposed implementation is about six times faster than the plain C implementation (i.e. with no special optimizations) and
superior to optimized implementations of features with similar descriptive power

Herout, Adam

Juránek, Roman

Zemčík, Pavel

University of West Bohemia Digital Library

Implementing the local binary patterns with SIMD instructions of CPU

DSpace at University of West Bohemia

Implementing the Local Binary Patterns with SIMDInstructions of CPURoman Juránek, Pavel Zemcˇík, Adam Herout{ijuranek,zemcik,herout}@fit.vutbr.czGraph@FITDepartment of Computer Graphics and MultimediaFaculty of Information TechnologyBrno University of Technology, Brno, Czech RepublicABSTRACTUsage of statistical classifiers, namely AdaBoost and its modifications, is very common in object detection and pattern recog-nition. Performance of such classifiers strongly depends on low level features they use. This paper presents an experimentalimplementation of the Local Binary Patterns (LBP) that uses SIMD instructions for acceleration. The experiments shows thatthe proposed implementation is about six times faster than the plain C implementation (i.e. with no special optimizations) andsuperior to optimized implementations of features with similar descriptive power.Keywords: LBP, AdaBoost, Object Detection, Feature Extraction, SIMD, SSE, CUDA1 INTRODUCTIONObject detection in images and video sequences haswide range of applications. Several object detectionmethods exist; however, one of the best available meth-ods today is exploitation of statistical classifiers. Thestatistical classifiers are able to distinguish an objectfrom non-object in a small window. To detect the objectin image data or video, it is necessary to scan the imageor video frame and apply the classifier to each possiblewindow location within the scanned image. The clas-sifier works with low-level features extracted from theclassified image window. The features are usually sim-ple functions of selected pixels from the window. Thedesign of the features significantly affects the perfor-mance of the classifier and its speed and thus the speedof detection.The Local Binary Patterns (LBP) [7] discussed in thispaper are widely known to be good features to describelocal areas. They are frequently used in texture analysisand segmentation. The recent studies show that they areusable as features for classification as well.In this paper, the high performance implementationof LBP feature extraction is introduced. The imple-mentation exploits the SIMD instructions (namely SSE)available on contemporary CPUs. The implementationis comparable with similar implementations of otherfeature types – Local Rank Differences (LRD) [13] andLocal Rank Patterns (LRP) [6]. For the experimentsPermission to make digital or hard copies of all or part ofthis work for personal or classroom use is granted withoutfee provided that copies are not made or distributed for profitor commercial advantage and that copies bear this noticeand the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.WaldBoost (a modification of AdaBoost [11]) trainingalgorithm [9] has been used.2 RELATED WORKThe interest in fast feature extraction has been ear-lier pursued by other researchers because the extractionspeed influences the overall performance of the objectdetection. Efficient implementation is thus necessaryfor practical applications of object detection.Innovative approach that constituted a breakthroughin real-time object detection was taken by Viola andJones [11]. They converted the input image into an inte-gral representation which allows for calculation of Haarfeatures in constant time.There has also been much effort to implement the Ad-aBoost based object detection on hardware platformslike FPGA or ASIC chips [12, 10, 5]. Typically theyuse traditional Haar features. Recently there were pro-posed implementations of object detection with LocalRank Differences image features (LRD) on GPU [8]and CUDA [2] which employs resources of moderngraphics cards to accelerate feature extraction. The dis-advantage of this approaches is that they still need spe-cial hardware.In our previous work, we introduced high perfor-mance implementation of the Local Rank Differences[1] and Local Rank Patterns [4] image features. Theseimplementations exploited the SIMD instructions of In-tel CPU. This work shares same framework with thementioned implementations of LRD and LRP.3 LOCAL BINARY PATTERNSLocal Binary Patterns [7, 14], in their basic form, cap-ture information about local textural structures throughthresholding samples from local neighborhood by itscentral value. From the thresholded values a patternWSCG 2010 Poster papers 39code is formed such that each sample is represented bya single bit (Figure 1). The image pixels or response ofsome sampling function can be used as the samples. Inmany applications, circular neighborhood with 8 sam-ples is used (8 bit pattern).sampling samples threshold weights7 4 210 8 513 12 100 0 01 1 11 0LBP = 11110000b = 2401 2 464 32 16128 8Figure 1: Evaluation of the LBP feature. First the sam-ples are taken from the image. The values are thenthresholded by the central value.In this form, the feature response is dependent on thefeature orientation. In some applications, such as tex-ture classification or image segmentation, the rotationalinvariance is needed. In this case, the feature responseis normalized (e.g. by bit shifting).In our approach, the feature consists from 3× 3 reg-ularly spaced rectangular samples. The classifier canhold feature instances of different sizes. However, forpractical reasons the size of a sample is constrained tomaximum of 2× 2 pixels. And therefore four possiblesamplings exist as shown in the Figure 2. This con-straint, as we show in [4], does not introduce any re-duction of classifier precision.Figure 2: Possible configurations of LBP features.4 LBP EVALUATIONThe traditional way of the feature evaluation is to gathersamples from the input image and sequentially con-struct the LBP code bit by bit. The evaluation in theproposed implementation is optimal in two ways. First,it minimizes number of memory accesses by using pre-calculated image representation – interleaved convolu-tion image [1, 4], which allows for fast retrieval of nec-essary data. Second, the evaluation takes advantage ofthe SIMD instructions by processing all values of thefeature in parallel manner.4.1 The SSE Instruction SetBefore the actual implementation of the LBP featureevaluation is described, let us briefly characterize theSSE instruction set on Intel CPUs. Unlike classic x86instructions where an operation is executed on onepiece of data at the time, the SSE set provides means toexecute one operation over multiple data (as shown inthe Figure 3). Therefore the main attribute of the SSEinstruction set is parallel processing of data.A a1 a2 a3 a4B b1 b2 b3 b4A+B a1+b1 a2+b2 a3+b3 a4+b4ADDPSFigure 3: Operation of SSE instruction ADDPS. The Aand B are considered to be vectors of four float val-ues.The SSE set was introduced in 1999 as an exten-sion to x86 and MMX instruction set. The instructionsworks with 128 bit wide registers which can hold a vec-tor of values (e.g. 4× 32 bit float, 16× 8 bit integer,etc.).4.2 Image PreprocessingAs was previously stated, in our approach we use rect-angular samples as an input for feature evaluation. Inthe preprocessing stage, images convolved with all pos-sible shapes of samples are created. This images arelater used as source of samples. Four images are cre-ated in our case as the size of the samples is restrictedto maximum of 2×2 pixels.Every convolved image is arranged in a manner thatfour consequent pixels in the memory (i.e. 32 bit wordas we use 8 bit images) correspond to 2× 2 adjacentconvolution responses. This memory layout requiresthat each convolved image is logically divided intoblocks where each block contains sub-sampled imageconvolved with same modulo position of convolutionkernel. Number of blocks is determined by w×h wherew and h is width and height of the convolution kernel.This is schematically shown in Figure 4 where prepro-cessing of image with 2× 1 pixel kernel is displayedand two memory blocks are formed. Images prepro-cessed with other kernels are created analogically. Thislayout ensures that data for each feature are placed insame block and can be obtained by two memory ac-cesses.0 12 30 12 3Image Convolution Memory0 1 2 3012 332 bitrowblock1 2 301 2 301 2 300 12 3block 0block 1Figure 4: Preprocessing of image with 2×1 pixel ker-nel.Note that the feature evaluation itself is independenton the choice of the convolution kernels. The rectan-gular shape was selected because the implementationWSCG 2010 Poster papers 40of convolution and memory rearrangement can be veryhighly optimized.4.3 The EvaluationThe framework of evaluation of the features is dis-played in the Figure 5. The input image is first pre-processed – the convolution images are created. Eachfeature is parametrized by a position in the image andsize of samples. The convolution image is selected ac-cording to the size of samples and the block and ad-dress within the block is determined from the featureposition.Pre−calculated sampling functionPkPk+1Pk+2Pk,0Pk,1Pk,2Feature32 bit wordconvolveevaluation LBPInput imageload dataFigure 5: Schematic view of the image preprocessingand LBP evaluation. In this case a feature with 2× 2pixel samples is used.By loading eight 32 bit aligned pixels from two sub-sequent rows of a convolution image (i.e. two 64 bitreads), the 4×4 responses of the sampling function areeffectively loaded. The feature data is then located in a3×3 sub-window of this data (feature shift). Note thatthe feature evaluation code is independent on the sizeof the feature as the data are loaded from precalculatedrepresentations.01expandcomparisondatamaskLBP=240maskingsum00 00 00FF00FF00FF00FFFFFFFF00FFFFand>02 04 0080 00 08 0040 20 10 0000 00 00 000000 000000 00 00 0000center_value00 00 0006 010F0F0D0A0F0F0F0C000A0F0A 0A 0A 0A0A 0A 0A 0A0A 0A 0A 0A0A 0A 0A 0A00 00008040 20 10Figure 6: Block diagram of LBP evaluation.Figure 6 shows the feature evaluation step by step.The corresponding code is displayed in the Figure 7.Every grid represents a 128 bit SIMD register holdingsixteen 8 bit values loaded from the pre-calculated sam-pling function. First, the central value (defined by fea-ture shift) is expanded to the full register width andcompared with all other values. The comparison re-sult then serves as a mask for vector of weights (againselected according to the feature shift). The maskedweights (i.e. weights for which the comparison resultedin true) are then summed up producing the LBP value.Note that, although the comparison can result in true onpositions that do not belong to the feature, this will notinfluence the result as the weights are always zero forthem.union {__m128i q;signed short ss[8];} result = {_mm_sad_epu8(_mm_and_si128(lbpWeights[shift].q,_mm_cmpgt_epi8(data,_mm_set1_epi8(*center))),zero)};int lbp = result.ss[4] + result.ss[0];Figure 7: The actual evaluation code (with Intel intrin-sic functions). The lbpWeight[shift] selects oneof four possible weight vectors, data is vector of thedata and center is the pointer to feature central value.5 RESULTSThe experiments were conducted on a PC with IntelCore i7 CPU (eight core), 4 GB DDR3 and CUDA ca-pable ASUS NVidia ENGTX 280 graphics card.The proposed implementation was compared to ex-perimental implementation of LBP on CUDA architec-ture [3] and to the implementation with no special op-timizations (which we refer to as Plain C). And alsocompared to implementations of the LRD and LRP ex-tractors which shares same evaluation framework.The implementations were tested on the task of mul-tiscale face detection. In the first test, we comparethe real detection performance in terms of processedframes per second on a long video (taken from publicTV broadcasting).α = 0.1 560×240px 720×576px 1280×720pxSIMD 61/58/57 22/20/20 10/10/10CUDA 88/73/69 68/56/54 27/24/23Plain C 8/6.4/5.6 3.4/3.5/2.8 1.4/1.3/1.1α = 0.2 560×240px 720×576px 1280×720pxSIMD 87/82/81 28/24/24 13/11/11CUDA 115/91/89 82/63/61 31/27/26Plain C 12/10.4/9.6 4.5/4/3.7 1.9/1.6/1.5Table 1: Object detection performance in frames persecond on different architectures with usage of differentfeature extractors. The values in the table are orderedin following way: LBP/LRP/LRD.Results in Table 1 display performance of process-ing of three videos with different resolutions rangingWSCG 2010 Poster papers 41from low resolution to 720p HD video. Two classifierswith different target error rate (α) were used. The ex-periment shown that the SIMD implementation outper-forms the Plain C. On the low resolutions the SIMDalmost reaches the performance of the multiprocessorCUDA implementation athought on the high resolu-tions the CUDA is almost three times faster.560×240px 720×576px 1280×720pxSIMD 0.32 0.98 2.1CUDA 0.29 0.43 0.82Plain C 2.5 7.8 17.5Table 2: Preprocessing performance [ms/frame]The Table 2 shows comparison of preprocessing onthe three architectures. The preprocessing includes im-age scaling to build image pyramid. Additionally, in theSIMD implementation, each pyramid level is convolvedaccording to description in the Section 4.6 CONCLUSION AND FUTUREWORKThe goal of the work presented in this paper was to ef-ficiently implement the Local Binary Patterns featureextractor in the contemporary CPUs using the advancedSIMD instructions. The goal was fulfilled and high per-formance feature extractor was implemented and exper-imentally evaluated.The extractor needs preprocessing stage which pre-pares convolution images with precalculated samplingfunctions. In the evaluation stage, each feature needsonly two memory accesses to load necesarry data andthe data are then processed by SIMD instructions to cal-culate the LBP feature response.The results show that the SIMD extractor is fasterthan the Plain C extractor by factor of 6.5 and almostreaches to performance of implementation on CUDAarchitecture. And the speed of LBP evaluation is com-parable or better than evaluation of feature types asLRD and LRP. Although the CUDA implementationis faster, it needs special hardware which supports theCUDA architecture. On the other hand, the SIMDinstructions are common in all contemporary CPUswhich makes the SIMD implementation suitable forlarger range of applications. Espetially for embededsystems.The future work includes optimization of featureevaluation on bulk data, for example evaluation ofseveral independent features in parallel or simultaneousevaluation of consequent features. The future workalso includes further dataflow optimizations.ACKNOWLEDGEMENTSThis work was supported by Czech Ministry of Educa-tion, Youth and Sports research project Center of Com-puter Graphics (LC06008).REFERENCES[1] Adam Herout, Michal Hradiš, Roman Juránek, and PavelZemcˇík. Implementation of the "local rank differences" imagefeature using simd instructions of cpu. In Proceedings of SixthIndian Conference on Computer Vision, Graphics and ImageProcessing, page 9, 2008.[2] Adam Herout, Radovan Jošth, Pavel Zemcˇík, and MichalHradiš. Gp-gpu implementation of the "local rank differences"image feature. In Proceedings of International Conference onComputer Vision and Graphics 2008, Lecture Notes in Com-puter Science, pages 1–11. Springer Verlag, 2008.[3] Adam Herout, Radovan Jošth, Pavel Zemcˇík, and MichalHradiš. Gp-gpu implementation of the "local rank differences"image feature. In Proceedings of International Conference onComputer Vision and Graphics 2008, Lecture Notes in Com-puter Science, pages 380–390. Springer Verlag, 2008.[4] Adam Herout, Pavel Zemcˇík, Michal Hradiš, Roman Juránek,Jirˇí Havel, Radovan Jošth, andMartin Žádník. Low-Level ImageFeatures for Real-Time Object Detection, page 25. IN-TECHEducation and Publishing, 2009.[5] M. Hiromoto, K. Nakahara, H. Sugano, Y. Nakamura, andR. Miyamoto. A specialized processor suitable for adaboost-based detection with haar-like features. In Computer Visionand Pattern Recognition, 2007. CVPR ’07. IEEE Conferenceon, pages 1–8, June 2007.[6] Michal Hradiš, Adam Herout, and Pavel Zemcˇík. Local rankpatterns - novel features for rapid object detection. In Pro-ceedings of International Conference on Computer Vision andGraphics 2008, Lecture Notes in Computer Science, pages 1–2,2008.[7] Timo Ojala, Matti Pietikäinen, and Topi Mäenpää. Multiresolu-tion gray-scale and rotation invariant texture classification withlocal binary patterns. IEEE Trans. Pattern Anal. Mach. Intell.,24(7):971–987, 2002.[8] Lukáš Polok, Adam Herout, Pavel Zemcˇík, Michal Hradiš, Ro-man Juránek, and Radovan Jošth. "local rank differences" im-age feature implemented on gpu. In Proceedings of the 10thInternational Conference on Advanced Concepts for IntelligentVision Systems, Lecture Notes In Computer Science; Vol. 5259,pages 170–181. Springer Verlag, 2008.[9] Jan Sochman and Jiri Matas. Waldboost – learning for timeconstrained sequential detection. In CVPR ’05: Proceedingsof the 2005 IEEE Computer Society Conference on ComputerVision and Pattern Recognition (CVPR’05) - Volume 2, pages150–156, Washington, DC, USA, 2005. IEEE Computer Soci-ety.[10] T. Theocharides, N. Vijaykrishnan, and M.J. Irwin. A paral-lel architecture for hardware face detection. In Emerging VLSITechnologies and Architectures, 2006. IEEE Computer SocietyAnnual Symposium on, volume 00, pages 2 pp.–, March 2006.[11] Paul Viola and Michael Jones. Rapid object detection usinga boosted cascade of simple features. Computer Vision andPattern Recognition, IEEE Computer Society Conference on,1:511–I–518 vol.1, 2001.[12] Yu Wei, Xiong Bing, and C. Chareonsak. Fpga implemen-tation of adaboost algorithm for detection of face biometrics.In Biomedical Circuits and Systems, 2004 IEEE InternationalWorkshop on, pages S1/6–17–20, Dec. 2004.[13] Pavel Zemcˇík, Michal Hradiš, and Adam Herout. Local rankdifferences - novel features for image. In Proceedings of SCCG2007, pages 1–12, 2007.[14] Lun Zhang, Rufeng Chu, Shiming Xiang, ShengCai Liao, andStan Z. Li. Face detection based on multi-block lbp representa-tion. In ICB, pages 11–18, 2007.WSCG 2010 Poster papers 42

English

http://dspace5.zcu.cz/bitstream/11025/1111/1/Juranek.pdf

Implementing the local binary patterns with SIMD instructions of CPU

Abstract

Similar works

Full text

Available Versions

University of West Bohemia Digital Library

DSpace at University of West Bohemia