In this paper, we present a graph-based semi-supervised framework for
hyperspectral image classification. We first introduce a novel superpixel
algorithm based on the spectral covariance matrix representation of pixels to
provide a better representation of our data. We then construct a superpixel
graph, based on carefully considered feature vectors, before performing
classification. We demonstrate, through a set of experimental results using two
benchmarking datasets, that our approach outperforms three state-of-the-art
classification frameworks, especially when an extremely small amount of
labelled data is used.Case Studentship with the NP

Aviles-Rivero, AI

Coomes, D

Faul, A

Papadakis, N

Schonlieb, CB

Sellars, P

English

arXiv

Crossref

Semi-Supervised Learning with Graphs: Covariance Based Superpixels For Hyperspectral Image Classification

International audienceIn this paper, we present a graph-based semi-supervised framework for hyperspectral image classification. We first introduce a novel superpixel algorithm based on the spectral covariance matrix representation of pixels to provide a better representation of our data. We then construct a superpixel graph, based on carefully considered feature vectors, before performing classification. We demonstrate, through a set of experimental results using two benchmarking datasets, that our approach outperforms three state-of-the-art classification frameworks, especially when an extremely small amount of labelled data is used

Sellars, Philip

Aviles-Rivero, Angelica I.

Papadakis, Nicolas

Coomes, David

Faul, Anita

Schönlieb, Carola-Bibiane

Archive Ouverte en Sciences de l'Information et de la Communication

Semi-supervised Learning with Graphs: Covariance Based Superpixels For Hyperspectral Image Classification

Apollo (Cambridge)

SEMI-SUPERVISED LEARNING WITH GRAPHS: COVARIANCE BASED SUPERPIXELSFOR HYPERSPECTRAL IMAGE CLASSIFICATIONPhilip Sellars1, Angelica I. Aviles-Rivero2, Nicolas Papadakis3,David Coomes4, Anita Faul5 and Carola-Bibiane Schönlieb11 DAMTP and 2DPMMS, Faculty of Mathematics, University of Cambridge, UK.3 IMB, Université Bordeaux France, 4 Department of Plant Science, University of Cambridge, UK.5 Laboratory for Scientific Computing, University of Cambridge, UK.ABSTRACTIn this paper, we present a graph-based semi-supervisedframework for hyperspectral image classification. We firstintroduce a novel superpixel algorithm based on the spectralcovariance matrix representation of pixels to provide a betterrepresentation of our data. We then construct a superpixelgraph, based on carefully considered feature vectors, beforeperforming classification. We demonstrate, through a set ofexperimental results using two benchmarking datasets, thatour approach outperforms three state-of-the-art classificationframeworks, especially when an extremely small amount oflabelled data is used.Index Terms— Hyperspectral Imaging, Superpixels, Covari-ance, Graphs, Semi-Supervised Learning, Classification1. INTRODUCTIONHyperspectral image (HSI) classification is an active area ofresearch and poses unique challenges. The high dimensionalnature of the data allows for a detailed description of an im-age, and class labels can be assigned on a pixel-by-pixel basis.The majority of classification frameworks for HSIs are super-vised learning (SL) frameworks; during the training phase in-formation is only gained from the initial labelled data. Theseinclude kernel methods [1], deep-learning [2] and sparse rep-resentation methods [3]. However, the problem with SL meth-ods is that they rely upon the existence of a large and accu-rately labelled training set. In application such as HSI classi-fication, label collection is time consuming and expensive.Another set of algorithmic approaches are based on unsuper-vised learning frameworks. However, the intrinsic nature ofthe problem makes the classification task a strongly ill-posedproblem, and therefore, specific assumptions are needed tomitigate the lack of correspondence between the producedclusters and the known classes.In practice, the size of the labelled set is often very small com-pared to the amount of unlabelled data. In such applications,there are large advantages to using semi-supervised learning(SSL) approaches [4]. SSL methods use information presentin the labelled and unlabelled data during the training process,which can lead to much higher classification accuracy. Theseapproaches can be divided into three groups: generative, low-density separation and graph-based methods.This paper follows the graph perspective. This is motivatedby the advantages of using graphs: i) a natural representationfor HSI data, ii) a way to gain scalability and therefore com-putational tractability and iii) a structure with mathematicaldesirable properties (e.g. sparseness). However, a fundamen-tal problem in graph based learning is how to construct thegraph in order to produce an accurate classification? This isof great interest as the performance of the classifier heavilydepends on the feature selection and the structure design, andthis question is addressed in this paper.Contributions. We present a novel graph base frameworkfor HSI classification, which we call hyperspectral superpixelgraph classification (HSGC). Our approach achieves state-of-the-art classification results using a semi-supervised graphbased approach alongside a carefully designed feature space.Our highlights are: 1) spectral covariance based superpix-els for feature extraction which uses local covariance ma-trix representation to include spatial and spectral informa-tion, 2) a graphical representation where each node repre-sents a superpixel and the edges represent the similarity beingthe superpixel feature vectors and 3) a unified SSL frame-work. We show that learning with minimal supervision ishighly beneficial when one removes feature space redundancywhilst strengthening the synergy between feature selectionand graph construction.2. LEARNING WITH MINIMAL SUPERVISIONThis section is divided into three key parts. Firstly, we in-troduce our proposed algorithmic approach to segment HSIs.Secondly, we describe our feature extraction process. Fi-nally, we construct the graphical representation and performthe classification task.2.1. Spectral Covariance Based SuperpixelsLet I = {Ib}, b = 1, ..,B be a HSI with dimensions W ×H × B representing the width, height and number of bandsrespectively and Ib : W × H → D where D is the imagerepresentation of a band. Our framework starts by performingdimensionality reduction, via PCA [5], on I for computationalefficiency. We construct a dimensionally reduced HSI Î ={Îa}, a = 1, ..,A where A  B.We then aim to find a better representation of the HSI datato increase the performance of our classifier. This is an im-portant step as classification accuracy is highly dependent onsetting relevant local regions. Local regions are commonlyset by using either a fixed size or dynamic window. However,this provides a limited representation. An alternative is to usesuperpixels, which has been explored in [1]. Unlike [1] whichrelied on an existing superpixel algorithm, we propose a newsuperpixel approach that is designed with HSIs in mind.Denoting an individual pixel as p ∈ Î, superpixels split theHSI into a family of disjoint sets, Î = ∪Ki=1Si , Si ∩ Sj = ∅,where Si corresponds to an individual superpixel and K isthe number of superpixels, which is initially set by the user.Each superpixel Si is made up of a set of ni connected pix-els, Si = {pi,1, ..., pi,ni}. Our superpixel segmentationsare produced via minimisation of the following objective,Q({S1, ..,SK}) =∑Ki=1∑p∈Î d((p, Î(p)), F (Si)) where dis a distance function and F (Si) is the average of Si.We have built our superpixel algorithm on top of the com-monly used SLIC algorithm [6]. SLIC has drawbacks [7]found in the fixed size localised search range. To improveupon this, we adopt the observation of [7], in which the searchrange is dynamically adjusted by the local content density inthe image. This information is given by the function g whichmaps each pixel to a positive real number. Therefore, in oursuperpixel algorithm, we used the following search ranged((p, Î(p)), F (Si)) if |p− (F (Si))1| ≤ 2√WHK g(F (Si));Otherwise d((p, Î(p)), F (Si)) =∞;where (F (Si))1 represents that spatial part of the featurefunction, and | · | is the euclidean distance on the image grid.Furthermore, instead of using the Euclidean spectral distancefound in [6, 7], we instead use covariance matrix represen-tation [8] and the Log-Euclidean distance (LED) [9] whichis better suited for HSIs. For each pixel p ∈ Î we constructa covariance matrix Cp describing the relationship betweendifferent hyperspectral bands, which extracts powerful spec-tral and spatial information. However, covariance matricesare symmetric positive definite matrices and they do not lieon a Euclidean space but instead on a Riemannian manifold.Therefore, the LED metric is used to construct the spectraldistance between pixels,dspectral(px, py) = ||logm(Cpx)− logm(Cpy )||F , (1)In our superpixel construction, the reduced image Î is passedinto the covariance based superpixel algorithm and a 2-D su-perpixel label map is generated. This map is applied back toÎ to obtain our 3-D superpixel mapping.2.2. Feature ExtractionFrom each superpixel Si we extract three different features.By applying a mean filter to each superpixel we can extractlocalised spatial information. The mean feature vector is de-noted as ~Smi and it is defined as~Smi =∑nij=1 Î(pi,j)ni. (2)To obtain the spatial information surrounding a superpixel, wetake a weighted combination of the information present in theadjacent superpixels. For each given superpixel Si, we definethe set Zi = {z1, z2.., zJ} which contains the J indexes ofthe adjacent superpixels. The weighted feature vector ~Swi isgiven by~Swi =J∑j=1wi,zj~Smzj , (3)where h is a predefined scalar parameter and the weight be-tween adjacent superpixels wi,zj is defined aswi,zj =exp(−|| ~Smzj − ~Smi ||22/h)∑Jj=1 exp(−|| ~Smzj − ~Smi ||22/h) (4)Finally, we extract the location of the centre of each super-pixel ~Spi which we calculate as~Spi =∑nij=1 pi,jni. (5)2.3. Graph Construction and SSL ClassificationUsing these feature vectors we construct a weighted, undi-rected graphical representation G = (V,E,W ) where eachnode is a superpixel and the edge weights reflect the similar-ity between superpixels. Note that a similarity of 1 impliesmost similar and an decreasing number means less similar.The weight between superpixels is given by wij = sij lij ,sij = exp((β − 1) || ~Swi − ~Swj ||22 − β|| ~Smi − ~Smj ||22σ2s),(6)lij = exp(−|| ~Spi − ~Spj ||22σ2l). (7)The parameter β weights the contribution of the mean andweighted feature vectors and σs, σl determine the width of theGaussian kernels. The edge set is constructed using k-nearestneighbours.The initial labelling of the superpixels is specified using thematrix Y ∈ RK×c, where c is the number of classes presentand K is the number of superpixels. Yvl specifies the valueof the seed label l for node v. The initial label informationRGB GT HSGC (ours) LCMR IFRF SVMPAVIA UNIVERSITYINDIAN PINESFig. 1. The classification maps. A comparison of the classification maps produced for the two data sets. From left to right:the RGB image, the ground truth (GT), and classification maps from: HSGC (our approach), LCMR [10], IFRF [11] and SVM[12] methods when ten labelled pixels for each class are used for training. Note that HSGC and LCMR are SSL based methods.for each superpixel is taken as the average initial label of itsset of pixels. If no pixel within a superpixel is labelled thesuperpixel has no initial label. The graph G and the labellingmatrix Y are then fed into the Learning with Local and GlobalConsistency algorithm (LGC) by Zhou et al [13]. LGC propa-gates the labelling information across the graph and producesthe final superpixel labelling matrix F ∈ RK×c. The finallabel for each superpixel yi is given by,yi = argmaxj∈{1,..,c}Fij . (8)Each superpixel label is then passed down to its correspond-ing set of pixels.3. EXPERIMENTAL RESULTSThis section addresses the experimental methodology that weused to validate and assess our proposed approach.Data Description. We use two benchmark datasets. Univer-sity of Pavia: with dimensions of 610× 340× 103, a spectralrange from 0.43 to 0.86µm and a spatial resolution of 1.3m.Indian Pines: dimensions of 145×145×200, a spectral rangeof 0.4 to 2.5µm and a spatial resolution of 20m.Experimental Setup. We compared our proposed approachagainst three state-of-the-art HSI classification methods: lo-cal covariance matrix representation (LCMR) [10], a SVMmethod [12] and image fusion and recursive filtering (IFRF)[11]. The first method is semi-supervised and the last twoare supervised methods. The parameters of the compared ap-Table 1. OA (%) AA (%) and Kappa (%) with ten training samplesper class, and for the two HSI datasets.UNIVERSITY OF PAVIA DATASETHSGC (ours) LCMR [10] IFRF [11] SVM [12]OA 91.5 ± 2.6% 87.9 ± 3.8% 79.1 ± 3.8% 69.7 ± 3.1%AA 90.2 ± 7.8% 90.4 ± 2.6% 73.2 ± 3.1% 77.8 ± 1.3%Kappa 88.9 ± 3.3% 84.3 ± 4.7% 73.0 ± 4.6% 61.9 ± 3.3%INDIAN PINES DATASETOA 89.8 ± 1.7% 83.4 ± 2.1% 80.1 ± 3.2% 54.1 ± 2.7%AA 93.8 ± 7.7% 90.6 ± 1.5% 74.8 ± 2.8% 67.2 ± 1.6%Kappa 88.4 ± 2.0% 81.2 ± 2.3% 77.5 ± 3.5% 48.7 ± 2.7%proaches were set to the default values referenced in the pa-pers. For our method the parameters were determined by anempirical coarse to fine search method. The spectral dimen-sion after PCA was set by demanding that the total explainedvariance ratio was ≥ 0.98. The number of superpixels cho-sen was 1000 for Indian Pines and 2000 for the Universityof Pavia. To demonstrate the performance of our method inminimal supervision settings, we only use very small trainingsets ranging from three to twenty labelled pixels per class.All experiments are repeated ten times and use three commonmetrics to evaluate our performance: overall accuracy (OA),average accuracy (AA) and the Kappa coefficient.Comparison With Other Methods. we start by visuallyevaluating our approach against the three different classifiers.The results are visualised in Fig.1. We can observe that ourmethod produces a smoother output, with significantly feweroutliers than the compared approaches. It deals well with theOA %NUMBER OF TRAINING SAMPLES PER CLASSFig. 2. Overall Classification Accuracy (OA). The lines rep-resent the average data for ten runs whilst the shaded regionsrepresent the standard deviation of: LCMR [10], IFRF [11],SVM [12] and the proposed HSGC.Table 2. OA (%) on the two benchmarking datasets.UNIVERSITY OF PAVIA DATASETLabels\ Class HSGC (ours) LCMR [10] IFRF [11] SVM [12]3 81.2 ± 6.5% 67.5 ± 5.9% 55.6 ± 10.3% 52.0 ± 4.0%7 91.7 ± 2.7% 83.9 ± 3.2% 70.6 ± 7.7% 65.0 ± 3.1%20 95.5 ± 1.4% 92.8 ± 1.9% 87.1 ± 3.2% 76.4 ± 3.3%INDIAN PINES DATASET3 72.4 ± 4.5% 65.4 ± 3.8% 57.2 ± 5.7% 38.5 ± 3.5%7 84.3 ± 2.0% 77.0 ± 2.9% 70.9 ± 5.6% 50.3 ± 3.0%20 94.1 ± 1.5% 90.7 ± 1.1% 89.4 ± 2.3% 62.1 ± 1.7%complex structures present in the Pavia data set and preservesthe boundaries between the different regions in Indian Pines.This is reflected in the numerical results reported in Table 1,where the OA, the AA and Kappa coefficient were calcu-lated using ten labelled pixels per class. We observe that ourapproach outperformed the other methods for both datasets.However, we can observe in Fig. 2 and Table 2 that our ap-proach significantly outperforms the compared methods whenthe number of labelled pixels per class is reduced below 10.We can observe, that for all number of labels counts and forboth datasets, our approach exhibits the highest overall classi-fication accuracy (OA). Note that higher values of OA corre-spond to better classifier performance. Overall, our approachoutperformed the compared state-of-the-art approaches. Themain contribution of our approach is to produce very highclassification accuracy even when the number of labelled pix-els per class is incredibly small.4. CONCLUSIONIn this work, we present a novel framework, HSGC, for hy-perspectral image classification. Our framework combinesa novel, purpose built superpixel algorithm with a semi-supervised graph based approach. We demonstrate state-of-the-art results compared with a recent semi-supervisedand two supervised approaches. Our highlight is that HSGCproduces the highest classification accuracy, even when theamount of labelled data is small. This shows the benefits andpotential of learning with minimal supervision on graphs.5. REFERENCES[1] L Fang, S Li, W Duan, J Ren, and J A Benedikts-son, “Classification of hyperspectral images by exploit-ing spectralspatial information of superpixel via multi-ple kernels,” IEEE Trans. Geosci. Remote Sens., vol. 53,no. 12, pp. 6663–6674, 2015.[2] D Ienco, R Gaetano, C Dupaquier, and P Maurel, “Landcover classification via multitemporal spatial data bydeep recurrent neural networks,” IEEE Geoscience andRemote Sensing Letters, pp. 1685–1689, 2017.[3] Y Chen, N M Nasrabadi, and T D Tran, “Semi-supervised graphbased hyperspectral image classifica-tion,” IEEE Trans. Geosci. Remote Sens., 2011.[4] O Chapelle, A Zien, and B Schölkopf, Semisupervisedlearning, MIT Press, 2006.[5] I Jolliffe, “Principal component analysis,” New York,NY, USA: Wiley, 2005.[6] R Achanta and et al, “SLIC superpixels compared tostate-of-the-art superpixel methods,” IEEE Trans. Pat-tern Anal. Mach. Intell., pp. 2274–2282, 2012.[7] G. Maierhofer, D. Heydecker, A. I. Aviles-Rivero, S. M.Alsaleh, and C. Schönlieb, “Peekaboo-where are theobjects? structure adjusting superpixels,” in IEEE Inter-national Conference on Image Processing (ICIP), 2018.[8] O Tuzel, F Porikli, and P Meer, “Region covariance:A fast descriptor for detection and classification,” Proc.Eur. Conf. Comput. Vis., pp. 589–600, 2006.[9] V Arsigny, P Fillard, X Pennec, and N Ayache, “Geo-metric means in a novel vector space structure on sym-metric positive-definite matrice,” SIAM J. Matrix Anal.Appl., vol. 29, no. 1, pp. 328–347, 2006.[10] L Fang, N He, S Li, and J Plaza, “A new spatial-spectralfeature extraction method for hyperspectral images us-ing local covariance matrix representation,” IEEE Trans.Geosci. Remote Sens., pp. 3534–3546, 2018.[11] X Kang, S Li, and J A Benediktsson, “Feature extractionof hyperspectral images with image fusion and recursivefiltering,” IEEE Trans. Geosci. Remote Sens., 2014.[12] F Melgani and L Bruzzone, “Classification of hyper-spectral remote sensing images with support vector ma-chines,” IEEE Trans. Geosci. Remote Sens., 2004.[13] D Zhou, O Bousquet, T Lal, J Weston, and B Schölkopf,“Learning with local and global consistency,” NIPS, pp.595–602, 2004.

Semi-Supervised Learning with Graphs: Covariance Based Superpixels for Hyperspectral Image Classification

In this paper, we present a graph-based semi-supervised framework for hyperspectral image classification. We first introduce a novel superpixel algorithm based on the spectral covariance matrix representation of pixels to provide a better representation of our data. We then construct a superpixel graph, based on carefully considered feature vectors, before performing classification. We demonstrate, through a set of experimental results using two benchmarking datasets, that our approach outperforms three state-of-the-art classification frameworks, especially when an extremely small amount of labelled data is used.Nonlocal Methods for Arbitrary Data Source

SELLARS, Philip

AVILES-RIVERO, Angelica I.

PAPADAKIS, Nicolas

COOMES, David

FAUL, Anita

SCHÖNLIEB, Carola-Bibiane

Oskar Bordeaux

https://www.repository.cam.ac.uk/bitstream/handle/1810/293841/JapanConference.pdf?sequence=3&isAllowed=y

Semi-Supervised Learning with Graphs: Covariance Based Superpixels for Hyperspectral Image Classification

Abstract

Similar works

Full text

Available Versions

Crossref

Archive Ouverte en Sciences de l'Information et de la Communication

Apollo (Cambridge)

Oskar Bordeaux