Deep learning for agricultural land use classification from Sentinel-2

[ES] En el campo de la teledetección se ha producido recientemente un incremento del uso de técnicas de aprendizaje profundo (deep learning). Estos algoritmos se utilizan con éxito principalmente en la estimación de parámetros y en la clasificación de imágenes. Sin embargo, se han realizado pocos esfuerzos encaminados a su comprensión, lo que lleva a ejecutarlos como si fueran “cajas negras”. Este trabajo pretende evaluar el rendimiento y acercarnos al entendimiento de un algoritmo de aprendizaje profundo, basado en una red recurrente bidireccional de memoria corta a largo plazo (2-BiLSTM), a través de un ejemplo de clasificación de usos de suelo agrícola de la Comunidad Valenciana dentro del marco de trabajo de la política agraria común (PAC) a partir de series temporales de imágenes Sentinel-2. En concreto, se ha comparado con otros algoritmos como los árboles de decisión (DT), los k-vecinos más cercanos (k-NN), redes neuronales (NN), máquinas de soporte vectorial (SVM) y bosques aleatorios (RF) para evaluar su precisión. Se comprueba que su precisión (98,6% de acierto global) es superior a la del resto en todos los casos. Por otra parte, se ha indagado cómo actúa el clasificador en función del tiempo y de los predictores utilizados. Este análisis pone de manifiesto que, sobre el área de estudio, la información espectral y espacial derivada de las bandas del rojo e infrarrojo cercano, y las imágenes correspondientes a las fechas del período de verano, son la fuente de información más relevante utilizada por la red en la clasificación. Estos resultados abren la puerta a nuevos estudios en el ámbito de la explicabilidad de los resultados proporcionados por los algoritmos de aprendizaje profundo en aplicaciones de teledetección.[EN] The use of deep learning techniques for remote sensing applications has recently increased. These algorithms have proven to be successful in estimation of parameters and classification of images. However, little effort has been made to make them understandable, leading to their implementation as “black boxes”. This work aims to evaluate the performance and clarify the operation of a deep learning algorithm, based on a bi-directional recurrent network of long short-term memory (2-BiLSTM). The land use classification in the Valencian Community based on Sentinel-2 image time series in the framework of the common agricultural policy (CAP) is used as an example. It is verified that the accuracy of the deep learning techniques is superior (98.6 % overall success) to that other algorithms such as decision trees (DT), k-nearest neighbors (k-NN), neural networks (NN), support vector machines (SVM) and random forests (RF). The performance of the classifier has been studied as a function of time and of the predictors used. It is concluded that, in the study area, the most relevant information used by the network in the classification are the images corresponding to summer and the spectral and spatial information derived from the red and near infrared bands. These results open the door to new studies in the field of the explainable deep learning in remote sensing applications.Este trabajo ha sido subvencionado gracias al Convenio 2019 y 2020 de colaboración entre la Generalitat Valenciana, a través de la Conselleria d’Agricultura, Medi Ambient, Canvi Climàtic i Desenvolupament Rural, y la Universitat de València – Estudi General.Campos-Taberner, M.; García-Haro, F.; Martínez, B.; Gilabert, M. (2020). Deep learning para la clasificación de usos de suelo agrícola con Sentinel-2. Revista de Teledetección. 0(56):35-48. https://doi.org/10.4995/raet.2020.13337OJS3548056Baraldi, A., Parmiggiani, F. 1995. An investigation of the textural characteristics associated with gray level cooccurrence matrix statistical parameters. IEEE Transactions on Geoscience and Remote Sensing, 33(2), 293-304. https://doi.org/10.1109/36.377929Bengio, Y., Simard, P., Frasconi, P. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157-166. https://doi.org/10.1109/72.279181Breiman, L., Friedman, J., Olshen, R.A., Stone, C.J. 1984. Classification and regression trees. Taylor & Francis: London, UK.Breiman, L. 2001. Random forests. Machine learning, 45(1), 5-32. https://doi.org/10.1023/A:1010933404324Campos-Taberner, M., Romero-Soriano, A., Gatta, C., Camps-Valls, G., Lagrange, A., Le Saux, B., Beaupère, A., Boulch, A., Chan-Hon-Tong, A., Herbin, S., Randrianarivo, H., Ferecatu, M., Shimoni, M., Moser, G., Tuia, D. 2016. Processing of extremely high-resolution Lidar and RGB data: outcome of the 2015 IEEE GRSS data fusion contest-part a: 2-D contest. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(12), 5547-5559. https://doi.org/10.1109/JSTARS.2016.2569162Campos-Taberner, M., García-Haro, F.J., Martínez, B., Sánchez-Ruíz, S., Gilabert, M.A. 2019a. A Copernicus Sentinel-1 and Sentinel-2 Classification Framework for the 2020+ European Common Agricultural Policy: A Case Study in València (Spain). Agronomy, 9(9), 556. https://doi.org/10.3390/agronomy9090556Campos-Taberner, M., García-Haro, F.J., Martínez, B., Sánchez-Ruiz, S., Gilabert, M.A. 2019b. Evaluación del potencial de Sentinel-2 para actualizar el SIGPAC de la Comunitat Valenciana. En: XVIII Congreso de la Asociación Española de Teledetección. Valladolid, España, 24-27, septiembre. pp 11-14.Camps-Valls, G., Tuia, D., Bruzzone, L., Benediktsson, J.A. 2013. Advances in hyperspectral image classification: Earth monitoring with statistical learning methods. IEEE Signal Processing Magazine, 31(1), 45-54. https://doi.org/10.1109/MSP.2013.2279179Chuvieco, E. 2008. Teledetección Ambiental. La observación de la Tierra desde el espacio. Madrid: Ariel.Cover, T., Hart, P. 1967. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21-27. https://doi.org/10.1109/TIT.1967.1053964Estrada, J., Sánchez, H., Hernanz, L., Checa, M.J., Roman, D. 2017. Enabling the Use of Sentinel-2 and LiDAR Data for Common Agriculture Policy Funds Assignment. ISPRS International Journal of Geo-Information, 6(8), 255. https://doi.org/10.3390/ijgi6080255Friedl, M.A., McIver, D.K., Hodges, J.C., Zhang, X.Y., Muchoney, D., Strahler, A.H., Baccini, A. 2002. Global land cover mapping from MODIS: algorithms and early results. Remote sensing of Environment, 83(1-2), 287-302. https://doi.org/10.1016/S0034-4257(02)00078-0Graves, A., Mohamed, A.R., Hinton, G. 2013. Speech recognition with deep recurrent neural networks. En 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645-6649. https://doi.org/10.1109/ICASSP.2013.6638947Gilabert, M.A., González-Piqueras, J., García-Haro, J., 1997. Acerca de los índices de vegetación. Revista de teledetección, 8, 1-10. Disponible en http://www.aet.org.es/?q=revista8-4González-Guerrero, O., y Pons, X., 2020. The 2017 Land Use/Land Cover Map of Catalonia based on Sentinel-2 images and auxiliary data. Revista de Teledetección, 55, 81-92. https://doi.org/10.4995/raet.2020.13112Gregrio, A., Jansen, J. 2000. Land cover classification system (LCCS); Classification concepts and user manual for software version 2. Roma: FAO.Griffiths, P., Nendel, C., Hostert, P. 2019. Intra-annual reflectance composites from Sentinel-2 and Landsat for national-scale crop and land cover mapping. Remote Sensing of Environment, 220, 135-151. https://doi.org/10.1016/j.rse.2018.10.031Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., Yang, G.Z. 2019. XAI-Explainable artificial intelligence. Science Robotics, 4(37). https://doi.org/10.1126/scirobotics.aay7120Haralick, R.M., Shanmugam, K., Dinstein, I.H. 1973. Textural features for image classification. IEEE Transactions on systems, man, and cybernetics, 3(6), 610-621. https://doi.org/10.1109/TSMC.1973.4309314Haykin, S. 1994. Neural networks: a comprehensive foundation. River: Prentice Hall.Immitzer, M., Vuolo, F., Atzberger, C. 2016. First experience with Sentinel-2 data for crop and tree species classifications in central Europe. Remote Sensing, 8(3), 166. https://doi.org/10.3390/rs8030166Hochreiter, S., Schmidhuber, J. 1997. Long short-term memory. Neural computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735Kussul, N., Lavreniuk, M., Skakun, S., Shelestov, A. 2017. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters, 14(5), 778-782. https://doi.org/10.1109/LGRS.2017.2681128Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., Johnson, B.A. 2019. Deep learning in remote sensing applications: A meta-analysis and review. ISPRS Journal of Photogrammetry and Remote Sensing, 152, 166-177. https://doi.org/10.1016/j.isprsjprs.2019.04.015Maggiori, E., Tarabalka, Y., Charpiat, G., Alliez, P. 2016. Convolutional neural networks for large-scale remotesensing image classification. IEEE Transactions on Geoscience and Remote Sensing, 55(2), 645-657. https://doi.org/10.1109/TGRS.2016.2612821Mikolov, T., Kombrink, S., Burget, L., Černocký, J., Khudanpur, S. 2011. Extensions of recurrent neural network language model. En 2011 IEEE International Conference on acoustics, speech and signal processing (ICASSP). Praga, República Checa, 22-27 Mayo. pp. 5528-5531. https://doi.org/10.1109/ICASSP.2011.5947611Montavon, G., Samek, W., Müller, K.R. 2018. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73, 1-15. https://doi.org/10.1016/j.dsp.2017.10.011Mou, L., Ghamisi, P., Zhu, X.X. 2017. Deep recurrent neural networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7), 3639-3655. https://doi.org/10.1109/TGRS.2016.2636241Mountrakis, G., Im, J., Ogole, C. 2011. Support vector machines in remote sensing: A review. ISPRS Journal of Photogrammetry and Remote Sensing, 66(3), 247- 259. https://doi.org/10.1016/j.isprsjprs.2010.11.001Ndikumana, E., Ho Tong Minh, D., Baghdadi, N., Courault, D., Hossard, L. 2018. Deep recurrent neural network for agricultural classification using multitemporal SAR Sentinel-1 for Camargue, France. Remote Sensing, 10(8), 1217. https://doi.org/10.3390/rs10081217Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N. 2019. Deep learning and process understanding for data-driven Earth system science. Nature, 566(7743), 195-204. https://doi.org/10.1038/s41586-019-0912-1Ruiz, L.A., Almonacid-Caballer, J., Crespo-Peremarch, P., Recio, J.A., Pardo-Pascual, J.E., SánchezGarcía, E. 2020. Automated classification of crop types and condition in a mediterranean area using a fine-tuned convolutional neural network. En The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences. Niza, Francia, 31 Agosto - 2 Septiembre (en línea). pp. 1061-1068. https://doi.org/10.5194/isprs-archivesXLIII-B3-2020-1061-2020Samek, W., Montavon, G., Vedaldi, A., Hansen, L. K., Muller, K.R. (Eds.). 2020. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Cham: Springer Nature. https://doi.org/10.1007/978-3-030-28954-6Schmedtmann, J., Campagnolo, M.L. 2015. Reliable crop identification with satellite imagery in the context of common agriculture policy subsidy control. Remote Sensing, 7(7), 9325-9346. https://doi.org/10.3390/rs70709325Schuster, M., Paliwal, K.K. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11), 2673-2681. https://doi.org/10.1109/78.650093Sitokonstantinou, V., Papoutsis, I., Kontoes, C., Lafarga Arnal, A., Armesto Andrés, A.P., Garraza Zurbano, J.A. 2018. Scalable parcel-based crop identification scheme using sentinel-2 data time-series for the monitoring of the common agricultural policy. Remote Sensing, 10(6), 911. https://doi.org/10.3390/rs10060911Story, M., Congalton, R.G. 1986. Accuracy assessment: a user's perspective. Photogrammetric Engineering and Remote Sensing, 52(3), 397-399.Tapsall, B., Milenov, P., Tasdemir, K. 2010. Analysis of RapidEye imagery for annual landcover mapping as an aid to European Union (EU) common agricultural polic. En ISPRS Technical Commission VII Symposium - 100 Years ISPRS. Viena, Austria, 5-7 Julio. pp. 568-573.Van Den Oord, A., Kalchbrenner, N., Kavukcuoglu, K. 2016. Pixel recurrent neural networks. En Proceedings of the 33rd International Conference on International Conference on Machine Learning. New York, EEUU., 20-22 Junio. pp. 1747-1756.Vuolo, F., Neuwirth, M., Immitzer, M., Atzberger, C., Ng, W.T. 2018. How much does multi-temporal Sentinel-2 data improve crop type classification? International Journal of Applied Earth Observation and Geoinformation, 72, 122-130. https://doi.org/10.1016/j.jag.2018.06.007Wardlow, B.D., Egbert, S.L., Kastens, J.H. 2007. Analysis of time-series MODIS 250 m vegetation index data for crop classification in the US Central Great Plains. Remote Sensing of Environment, 108(3), 290-310. https://doi.org/10.1016/j.rse.2006.11.021Watson, R.T., Noble, I.R., Bolin, B., Ravindranath, N.H., Verardo, D.J., Dokken, D.J. 2000. Land use, land-use change and forestry: a special report of the Intergovernmental Panel on Climate Change. Cambridge: Cambridge University Press.Zhan, X., Defries, R., Townshend, J.R.G., Dimiceli, C., Hansen, M., Huang, C., Sohlberg, R. 2000. The 250 m global land cover change product from the Moderate Resolution Imaging Spectroradiometer of NASA's Earth Observing System. International Journal of Remote Sensing, 21(6-7), 1433-1460. https://doi.org/10.1080/014311600210254Zhang, L., Zhang, L., Du, B. 2016. Deep learning for remote sensing data: A technical tutorial on the state of the art. IEEE Geoscience and Remote Sensing Magazine, 4(2), 22-40. https://doi.org/10.1109/MGRS.2016.2540798Zhong, L., Hu, L., Zhou, H. 2019. Deep learning based multi-temporal crop classification. Remote sensing of environment, 221, 430-443. https://doi.org/10.1016/j.rse.2018.11.032Zhu, Z., Woodcock, C.E. 2014. Continuous change detection and classification of land cover using all available Landsat data. Remote Sensing of Environment, 144, 152-171. https://doi.org/10.1016/j.rse.2014.01.011Zhu, X.X., Tuia, D., Mou, L., Xia, G.S., Zhang, L., Xu, F., Fraundorfer, F. 2017. Deep learning in remote sensing: A comprehensive review and list of resources. IEEE Geoscience and Remote Sensing Magazine, 5(4), 8-36. https://doi.org/10.1109/MGRS.2017.276230

Improvement in Land Cover and Crop Classification based on Temporal Features Learning from Sentinel-2 Data Using Recurrent-Convolutional Neural Network (R-CNN)

Understanding the use of current land cover, along with monitoring change over time, is vital for agronomists and agricultural agencies responsible for land management. The increasing spatial and temporal resolution of globally available satellite images, such as provided by Sentinel-2, creates new possibilities for researchers to use freely available multi-spectral optical images, with decametric spatial resolution and more frequent revisits for remote sensing applications such as land cover and crop classification (LC&CC), agricultural monitoring and management, environment monitoring. Existing solutions dedicated to cropland mapping can be categorized based on per-pixel based and object-based. However, it is still challenging when more classes of agricultural crops are considered at a massive scale. In this paper, a novel and optimal deep learning model for pixel-based LC&CC is developed and implemented based on Recurrent Neural Networks (RNN) in combination with Convolutional Neural Networks (CNN) using multi-temporal sentinel-2 imagery of central north part of Italy, which has diverse agricultural system dominated by economic crop types. The proposed methodology is capable of automated feature extraction by learning time correlation of multiple images, which reduces manual feature engineering and modeling crop phenological stages. Fifteen classes, including major agricultural crops, were considered in this study. We also tested other widely used traditional machine learning algorithms for comparison such as support vector machine SVM, random forest (RF), Kernal SVM, and gradient boosting machine, also called XGBoost. The overall accuracy achieved by our proposed Pixel R-CNN was 96.5%, which showed considerable improvements in comparison with existing mainstream methods. This study showed that Pixel R-CNN based model offers a highly accurate way to assess and employ time-series data for multi-temporal classification tasks

Tile2Vec: Unsupervised representation learning for spatially distributed data

Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distributional hypothesis from natural language -- words appearing in similar contexts tend to have similar meanings -- to spatially distributed data. We demonstrate empirically that Tile2Vec learns semantically meaningful representations on three datasets. Our learned representations significantly improve performance in downstream classification tasks and, similar to word vectors, visual analogies can be obtained via simple arithmetic in the latent space.Comment: 8 pages, 4 figures in main text; 9 pages, 11 figures in appendi

Information Technology for Land Degradation Assessment Based on Remote Sensing

National Research Foundation of Ukraine within the project 2020.02/0284 «Geospatial models and information technologies of satellite monitoring of smart city problems» and Horizon 2020 e-shape project (https://e-shape.eu/)Since the launch of ESA Copernicus program, satellite data of high resolution became publicly available and methods and tools for their automated processing to solve a wide range of applications have developed rapidly. An important scientific task is to assess land degradation and achieve zero levels of degradation. There are many methods for determining land degradation. Known approaches to the tasks of environmental land monitoring usually use the same methodology for all types of land cover. The paper represents the approach to the calculation of land degradation based on remote sensing data and modelling results taking into account the specifics of land degradation for different land cover and land use types. Our method is based on the classification of different land cover and land use types from satellite imagery and application of different schemes of land degradation assessment for each of them. We consider forest cuts as land degradation for forests and assess them using deep learning models. Land degradation for croplands is estimated by comparison of real leaf area index (LAI) and ideal LAI, calculated with the bio-physical crop development model. And land degradation for grassland is determined with a traditional approach based on vegetation index NDVI extracted from satellite imagery. The proposed approach was implemented for the territory of Ukraine