The paper deals with processing data obtained using nailfold high-speed videocapillaroscopy. To detect the red blood cells velocity two approaches are used. The deterministic approach is based on pixel intensities analysis for object detection and calculation of the displacement and velocity of red blood cells in a capillary. The obtained data formulate targets for the second approach. The stochastic approach is based on a sequence of artificial neural networks. The semantic segmentation network UNet is used for capillary detection. Then, the classification network GoogLeNet or ResNet is used as a feature extractor to convert masked video frames to a sequence of feature vectors. And finally, the long short-term memory network is used to approximate the red blood cells velocity. The results demonstrated that the accuracy of the mean velocity approximation in the time range of several seconds is up to 0.96. But the accuracy at each specific time moment is less accurate. So, the proposed algorithm allows the determination of the RBCs mean velocity but it doesn't allow determination of the RBCs pulsations accurate enough

Dremin, Viktor V.

Kornaev, Alexey V.

Kornaeva, Elena P.

Volkov, Mikhail V.

Aston Publications Explorer

PROCEEDINGS OF SPIESPIEDigitalLibrary.org/conference-proceedings-of-spieApplication of deep convolutional andlong short-term memory neuralnetworks to red blood cells motiondetection and velocity approximationAlexey Kornaev, Viktor Dremin, Elena Kornaeva, MikhailVolkovAlexey V. Kornaev, Viktor V. Dremin, Elena P. Kornaeva, Mikhail V. Volkov,"Application of deep convolutional and long short-term memory neuralnetworks to red blood cells motion detection and velocity approximation,"Proc. SPIE 12194, Computational Biophysics and Nanobiophotonics,121940C (29 April 2022); doi: 10.1117/12.2626040Event: XXV Annual Conference Saratov Fall Meeting 2021; and IXSymposium on Optics and Biophotonics, 2021, Saratov, Russian FederationDownloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 29 Jun 2022  Terms of Use: https://www.spiedigitallibrary.org/terms-of-use      Application of deep convolutional and long short-term memory neural networks to red blood cells motion detection and velocity approximation  Alexey V. Kornaev*a,b, Viktor V. Dreminc,d, Elena P. Kornaevae, and Mikhail V. Volkovf aDept. Of Mechatronics, Mechanics, and Robotics, Orel State University, 95 Komsomolskaya Street, Orel, Russian Federation, 302026; bLab. of Artificial Intelligence, Innopolis University, 1 Universitetskaya street, Innopolis, Russian Federation, 420500; cResearch and Development Center of Biomedical Photonics, Orel State University, 95 Komsomolskaya Street, Orel, Russian Federation, 302026; dCollege of Engineering and Physical Sciences, Aston University, Birmingham, B4 7ET, UK; eDept. of Information Systems, Orel State University, 95 Komsomolskaya Street, Orel, Russian Federation, 302026; fFaculty of Applied Optics, ITMO University, 49 Kronverksky, St. Petersburg, Russian Federation, 197101; ABSTRACT   The paper deals with processing data obtained using nailfold high-speed videocapillaroscopy. To detect the red blood cells velocity two approaches are used. The deterministic approach is based on pixel intensities analysis for object detection and calculation of the displacement and velocity of red blood cells in a capillary. The obtained data formulate targets for the second approach. The stochastic approach is based on a sequence of artificial neural networks. The semantic segmentation network UNet is used for capillary detection. Then, the classification network GoogLeNet or ResNet is used as a feature extractor to convert masked video frames to a sequence of feature vectors. And finally, the long short-term memory network is used to approximate the red blood cells velocity. The results demonstrated that the accuracy of the mean velocity approximation in the time range of several seconds is up to 0.96. But the accuracy at each specific time moment is less accurate. So, the proposed algorithm allows the determination of the RBCs mean velocity but it doesn't allow determination of the RBCs pulsations accurate enough. Keywords: deep learning, transfer learning, semantic segmentation, feature extraction, approximation, capillary, red blood cells, videocapillaroscopy  1. INTRODUCTION  The red blood cells (RBC) mechanical properties may indicate a healthy or diseased state of an organism1–5. Modern equipment for video recording and processing allows the RBCs motion and deformation observation in vessels in real-time mode6–8. Obviously, real-time in vivo health condition monitoring is a promising method. This paper deals with the RBCs velocity approximation using the long short-term memory (LSTM) networks. The study is based on the results of nailfold videocapillaroscopy (VCS) presented in the paper9. 2. DATA COLLECTION Images of the nailfold capillaries were obtained using a custom videocapillaroscopy setup providing direct visualization of capillary blood flow. In this setup, a long working distance objective (Mitutoyo M Plan APO 5X) with 5x magnification and 0.14 aperture was used together with a Bi-Convex lens with 200 mm focal length. A green LED light source was mounted to the side of the objective to illuminate the area of nailfold bed at a distal phalanx of a finger. The use of green light is also associated with better light absorption by the blood in this part of the spectrum, which provides a better contrast. The video was recorded with a a CMOS camera (IDS UI-3060-C-HQ) at a resolution of 800×800 and 150 fps, allowing the placement of several adjacent capillary loops in the field of view (see Figure 1). The study involved patients with rheumatic diseases and healthy volunteers presented in more detail in the paper by D. Stavtsev and coauthors9. Computational Biophysics and Nanobiophotonics, edited by Dmitry E. Postnov, Boris N. Khlebtsov, Proc. of SPIE Vol. 12194, 121940C · © 2022 SPIE · 1605-7422 · doi: 10.1117/12.2626040Proc. of SPIE Vol. 12194  121940C-1Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 29 Jun 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use       Figure 2. The red blood cells motion detection and velocity approximation procedure includes data collection, preliminary feature extraction, and data fitting. 3. DATA LABELING AND PRELIMINARY FEATURE EXTRACTION Processing of the obtained videos allowed the determination of the velocity of red blood cells in separate capillaries using the algorithm described in detail in the papers by I. Gurov and coauthors7. The main idea is that RBCs on a frame have maximal intensities that helps to detect their positions in the video frames and calculate the RBCs velocities. The approach is complex and requires video stabilization. The flow rate in a capillary is periodic, with the frequency equal to the heart beating frequency. As the result, each frame corresponds to the velocity value (see the velocity plot in the feature extraction section of the Figure 3). The obtained velocity data is used as targets in the following machine learning method.  The next few steps are related to the preliminary extraction of features before data fitting. The video frames are cropped from the size of 800×800 pixels to the size of 224×224 pixels. The cropped frames include 3 capillaries for the training, validation and test procedures (see Figure 4).  A part of the dataset of about 300 cropped frames was used as a training set for the semantic segmentation with UNet10. The trained UNet provides segmentation for 3 classes for capillaries 1, 2 and 3. The masked frames represent 3 independent subsets for the following feature extraction (see Figure 5). A deep convolutional network allows to transform an input image to a feature vector. In this paper the GoogLeNet and the ResNet18 are used (see Figure 6). The feature vectors are the output of the activations on the last pooling layer  11. 4. DATA FITTING USING THE LSTM NETWORK A sequence of N feature vectors represents a video fragment of the RBCs flow and an input for the LSTM network (see Fig. 1). The network architecture includes a sequence input layer, a BiLSTM layer with NH hidden units, a dropout layer, a fully connected layer and regression layer with one output that is the velocity value11. During training, the LSTM network receives sequences and velocities from the train and validation datasets. The test set obtained for the vessel 3 helped to test the trained network with the absolutely new data. The results are presented in Table 1.   Table 1. Accuracy of the mean velocity approximation in the time range of 10 s for the training, validation and test sets. Feature extractor / # features Sequence length LSTM depth Set Comments Train Validation Test GoogLeNet 15 200 98.7 96.4 49.7 Overfitting GoogLeNet 15 20 99.9 99.7 62.2 Overfitting ResNet18 30 20 95.4 96.0 61.3 Overfitting ResNet18 15 20 96.2 92.3 96.7  ResNet18 7 20 98.7 91.2 87.8   The first 3 lines in Table 1 demonstrate the overfitting. The best results were obtained with the ResNet18 feature extractor when the sequence length was of 15 and the BiLSTM layer was of 20 hidden units. It can be seen in the velocity plot of the data fitting section of the Figure 7 that the accuracy at each time step was small. But the mean velocity values were approximated with high accuracy (see Table 1).  Proc. of SPIE Vol. 12194  121940C-2Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 29 Jun 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use      The proposed algorithm is semi-automatic and needs velocity data labeling. The following research is supposed to be related to the application of an object detection method to the RBCs and the velocity approximation by tracing the bounding boxes10. This may help to exclude the velocity data labeling. The more important is that the motion detection and study the RBCs elastic deformation may help to solve problems of health monitoring and diagnostics. ACKNOWLEDGEMENTS The is work is funded by the Russian Science Foundation under the grant No 20-79-00332. The authors gratefully acknowledge this support. The authors also express graduate to Dmitry Stavtsev for the help with the experiment results preprocessing. REFERENCES [1]  Ulep, T. H., Zenhausern, R., Gonzales, A., Knoff, D. S., Lengerke Diaz, P. A., Castro, J. E. and Yoon, J. Y., “Smartphone based on-chip fluorescence imaging and capillary flow velocity measurement for detecting ROR1+ cancer cells from buffy coat blood samples on dual-layer paper microfluidic chip,” Biosensors and Bioelectronics 153, 112042 (2020). [2]  Maslianitsyna, A. I., Kadanova, I. M., Neznanov, A. I., Ermolinskiy, P. B., Gurfinkel, Yu. I., Pigurenko, А. A., Dyachuk, L. I., Lugovtsov, A. E. and Priezzhev, A. V., “Microrheologic properties of blood and capillary blood flow in case of arterial hypertension and type 2 diabetes mellitus: in vitro and in vivo optical assessment,” Complex Issues of Cardiovascular Diseases 9(2), 53–63 (2020). [3]  Plotnikov, M. B., Zibareva, L. N., Vasil’ev, A. S., Aliev, O. I., Anishchenko, A. M. and Maslov, M. Y., “Antihyperglycaemic, haemorheological and antioxidant activities of Lychnis chalcedonica L. extract in a streptozotocin-induced rat model of diabetes mellitus,” Journal of Complementary and Integrative Medicine 17(2) (2020). [4]  Yeow, N., Tabor, R. F. and Garnier, G., “Atomic force microscopy: From red blood cells to immunohaematology,” Advances in Colloid and Interface Science 249, 149–162 (2017). [5]  Harris, M. J., Wirtz, D. and Wu, P. H., “Dissecting cellular mechanics: Implications for aging, cancer, and immunity,” Seminars in Cell and Developmental Biology 93, 16–25 (2019). [6]  Machikhin, A. S., Volkov, M. v., Khokhlov, D. D., Lovchikova, E. D., Potemkin, A. v., Danilycheva, I. v., Dorofeeva, I. v. and Shulzhenko, A. E., “Exoscope-based videocapillaroscopy system for in vivo skin microcirculation imaging of various body areas,” Biomedical Optics Express 12(8), 4627 (2021). [7]  Gurov, I., Volkov, M., Margaryants, N., Pimenov, A. and Potemkin, A., “High-speed video capillaroscopy method for imaging and evaluation of moving red blood cells,” Optics and Lasers in Engineering 104, 244–251 (2018). [8]  Dremin, V., Kozlov, I., Volkov, M., Margaryants, N., Potemkin, A., Zherebtsov, E., Dunaev, A. and Gurov, I., “Dynamic evaluation of blood flow microcirculation by combined use of the laser Doppler flowmetry and high-speed videocapillaroscopy methods,” Journal of Biophotonics 12(6), e201800317 (2019). [9]  Stavtsev, D., Volkov, M., Margaryants, N., Potemkin, A., Dremin, V., Kozlov, I., Makovik, I., Zherebtsov, E. and Dunaev, A., “Investigation of blood microcirculation parameters in patients with rheumatic diseases by videocapillaroscopy and laser Doppler flowmetry during cold pressor test,” Proc. SPIE 11065, 110650T, 110651 (2019). [10]  “Deep Learning.”, <https://www.deeplearningbook.org/> (25 November 2021). [11]  “Classify Videos Using Deep Learning - MATLAB & Simulink.”, <https://www.mathworks.com/help/deeplearning/ug/classify-videos-using-deep-learning.html> (28 November 2021).   Proc. of SPIE Vol. 12194  121940C-3Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 29 Jun 2022Terms of Use: https://www.spiedigitallibrary.org/terms-of-use

Application of deep convolutional and long short-term memory neural networks to red blood cells motion detection and velocity approximation

https://publications.aston.ac.uk/id/eprint/43976/1/121940C.pdf

Application of deep convolutional and long short-term memory neural networks to red blood cells motion detection and velocity approximation

Abstract

Similar works

Full text

Available Versions

Aston Publications Explorer