Multi-object Tracking Based on a Novel Feature Image with Multi-modal Information

Multi-object tracking technology plays a crucial role in many applications, such as autonomous vehicles and security monitoring. This paper proposes a multi-object tracking framework based on the multi-modal information of 3D point clouds and color images. At each sampling instant, the 3D point cloud and image acquired by a LiDAR and a camera are fused into a color point cloud, where objects are detected by the Point-GNN method. And, a novel height-intensity-density (HID) image is constructed from the bird's eye view. The HID image truly reflects the shapes and materials of objects and effectively avoids the influence of object occlusion, which is helpful to object tracking. In two sequential HID images, a new rotation kernel correlation filter is proposed to predict the objects. Furthermore, an object retention module and an object re-recognition module are developed to overcome the object matching failure in the in-between frames. The proposed method takes full advantage of the multi-modal data and effectively achieves the information complementation to improve the accuracy of multi-object tracking. The experiments with the KITTI dataset show that the proposed method has the best performance among the existing traditional multi-object tracking methods

Multiscale Adaptive Edge Detector for Images Based on a Novel Standard Deviation Map

Edge detection plays an important role in many applications, such as industrial inspection and automatic driving. However, it is difficult to effectively distinguish between faint edges and noise, which may result in losing effective edges or generating spurious edges. This will reduce the accuracy of edge detection. In addition, some parameters need to be set artificially. In the case of the fixed parameters, the overall performance of edge detection on different images is not high. The adaptivity of edge detection needs to be improved further. To solve these problems, this article proposes a multiscale adaptive edge detector for images. First, multiscale pyramid images are constructed from an input image to provide multiscale features for edge detection. At each scale, a gradient map and a novel standard deviation map are calculated based on the gradients and the statistical characteristics of the local gradient differences, respectively, to accurately distinguish the edges from the background and noise. By using these two feature maps, candidate edges are adaptively identified from the image by using pixel-by-pixel detection. Then, candidate edges at different scales are thinned and fused together based on a novel voting mechanism. Finally, a binarized edge map is obtained by using adaptive hysteresis linking. These steps make the proposed edge detector accurate and adaptive. Experiments demonstrate that the proposed edge detector achieves good performance, which is beneficial to measurement applications

Lightweight Attention Module for Deep Learning on Classification and Segmentation of 3-D Point Clouds

Research on classification and segmentation of 3-D point clouds using deep learning methods has become a hot topic in emerging applications, such as autonomous driving, augmented reality, and indoor navigation. However, as the complexity of the network structures increases, the computational efficiency reduces, which affects the practical applications of these methods. In addition, prior researchers mostly seek to enhance the quality of spatial encodings, while the channel relationships are ignored. It makes the feature learning of point clouds insufficient, which will reduce the accuracy of classification and segmentation. In this article, a lightweight attention module (LAM) is proposed to improve the computational efficiency and accuracy at the same time by adopting a novel convolution mode and introducing a new attention mechanism based on channelwise statistical features. As the submodules of LAM, the lightweight module and the attention module can also be used independently to focus on improving the computational efficiency and accuracy, respectively, according to the actual applications. LAM and its submodules can be easily integrated into state-of-the-art deep learning methods on classification and segmentation of 3-D point clouds. The experimental results show that the proposed modules have a good performance on benchmark data sets

Novel Hybrid Physics-Informed Deep Neural Network for Dynamic Load Prediction of Electric Cable Shovel

Abstract Electric cable shovel (ECS) is a complex production equipment, which is widely utilized in open-pit mines. Rational valuations of load is the foundation for the development of intelligent or unmanned ECS, since it directly influences the planning of digging trajectories and energy consumption. Load prediction of ECS mainly consists of two types of methods: physics-based modeling and data-driven methods. The former approach is based on known physical laws, usually, it is necessarily approximations of reality due to incomplete knowledge of certain processes, which introduces bias. The latter captures features/patterns from data in an end-to-end manner without dwelling on domain expertise but requires a large amount of accurately labeled data to achieve generalization, which introduces variance. In addition, some parts of load are non-observable and latent, which cannot be measured from actual system sensing, so they can’t be predicted by data-driven methods. Herein, an innovative hybrid physics-informed deep neural network (HPINN) architecture, which combines physics-based models and data-driven methods to predict dynamic load of ECS, is presented. In the proposed framework, some parts of the theoretical model are incorporated, while capturing the difficult-to-model part by training a highly expressive approximator with data. Prior physics knowledge, such as Lagrangian mechanics and the conservation of energy, is considered extra constraints, and embedded in the overall loss function to enforce model training in a feasible solution space. The satisfactory performance of the proposed framework is verified through both synthetic and actual measurement dataset

Ore Rock Fragmentation Calculation Based on Multi-Modal Fusion of Point Clouds and Images

The accurate calculation of ore rock fragmentation is important for achieving the autonomous mining operation of mine excavators. However, a single mode cannot accurately calculate the ore rock fragmentation due to the low resolution of the point cloud mode and the lack of spatial position information of the image mode. To solve this problem, we propose an ore rock fragmentation calculation method (ORFCM) based on the multi-modal fusion of point clouds and images. The ORFCM makes full use of the advantages of multi-modal data, including the fine-grained object segmentation of images and spatial location information of point clouds. To solve the problem of image under-segmentation, we propose a multiscale adaptive edge-detection method based on an innovative standard deviation map to enhance the weak edges. Furthermore, an improved marked watershed segmentation algorithm is proposed to solve the problem of low segmentation accuracy caused by excessive noise of the gradient map and weak edges submerged. Experiments demonstrate that ORFCM can accurately calculate ore rock fragmentation in the local excavation area without relying on external markers for pixel calibration. The average error of the equivalent diameter of ore rock blocks is 0.66 cm, the average error of the elliptical long diameter is 1.42 cm, and the average error of the elliptical short diameter is 1.06 cm, which can effectively meet practical engineering needs

Semantic Segmentation of 3D Point Clouds in Outdoor Environments Based on Local Dual-Enhancement

Semantic segmentation of 3D point clouds in drivable areas is very important for unmanned vehicles. Due to the imbalance between the size of various outdoor scene objects and the sample size, the object boundaries are not clear, and small sample features cannot be extracted. As a result, the semantic segmentation accuracy of 3D point clouds in outdoor environment is not high. To solve these problems, we propose a local dual-enhancement network (LDE-Net) for semantic segmentation of 3D point clouds in outdoor environments for unmanned vehicles. The network is composed of local-global feature extraction modules, and a local feature aggregation classifier. The local-global feature extraction module captures both local and global features, which can improve the accuracy and robustness of semantic segmentation. The local feature aggregation classifier considers the feature information of neighboring points to ensure clarity of object boundaries and the high overall accuracy of semantic segmentation. Experimental results show that provides clearer boundaries between various objects, and has higher identification accuracy for small sample objects. The LDE-Net has good performance for semantic segmentation of 3D point clouds in outdoor environments

CT Image-Based Biopsy to Aid Prediction of HOPX Expression Status and Prognosis for Non-Small Cell Lung Cancer Patients

This study aimed to elucidate a computed tomography (CT) image-based biopsy with a radiogenomic signature to predict homeodomain-only protein homeobox (HOPX) gene expression status and prognosis in patients with non-small cell lung cancer (NSCLC). Patients were labeled as HOPX-negative or positive based on HOPX expression and were separated into training (n = 92) and testing (n = 24) datasets. In correlation analysis between genes and image features extracted by Pyradiomics for 116 patients, eight significant features associated with HOPX expression were selected as radiogenomic signature candidates from the 1218 image features. The final signature was constructed from eight candidates using the least absolute shrinkage and selection operator. An imaging biopsy model with radiogenomic signature was built by a stacking ensemble learning model to predict HOPX expression status and prognosis. The model exhibited predictive power for HOPX expression with an area under the receiver operating characteristic curve of 0.873 and prognostic power in Kaplan–Meier curves (p = 0.0066) in the test dataset. This study’s findings implied that the CT image-based biopsy with a radiogenomic signature could aid physicians in predicting HOPX expression status and prognosis in NSCLC

Fabrication of Poly Dopamine@poly (Lactic Acid-Co-Glycolic Acid) Nanohybrids for Cancer Therapy via a Triple Collaboration Strategy

Breast cancer is a common malignant tumor among women and has a higher risk of early recurrence, distant metastasis, and poor prognosis. Systemic chemotherapy is still the most widely used treatment for patients with breast cancer. However, unavoidable side effects and acquired resistance severely limit the efficacy of treatment. The multi-drug combination strategy has been identified as an effective tumor therapy pattern. In this investigation, we demonstrated a triple collaboration strategy of incorporating the chemotherapeutic drug doxorubicin (DOX) and anti-angiogenesis agent combretastatin A4 (CA4) into poly(lactic-co-glycolic acid) (PLGA)-based co-delivery nanohybrids (PLGA/DC NPs) via an improved double emulsion technology, and then a polydopamine (PDA) was modified on the PLGA/DC NPs’ surface through the self-assembly method for photothermal therapy. In the drug-loaded PDA co-delivery nanohybrids (PDA@PLGA/DC NPs), DOX and CA4 synergistically induced tumor cell apoptosis by interfering with DNA replication and inhibiting tumor angiogenesis, respectively. The controlled release of DOX and CA4-loaded PDA@PLGA NPs in the tumor region was pH dependent and triggered by the hyperthermia generated via laser irradiation. Both in vitro and in vivo studies demonstrated that PDA@PLGA/DC NPs enhanced cytotoxicity under laser irradiation, and combined therapeutic effects were obtained when DOX, CA4, and PDA were integrated into a single nanoplatform. Taken together, the present study demonstrates a nanoplatform for combined DOX, CA4, and photothermal therapy, providing a potentially promising strategy for the synergistic treatment of breast cancer

A high-precision multi-dimensional microspectroscopic technique for morphological and properties analysis of cancer cell

Abstract Raman and Brillouin scattering are sensitive approaches to detect chemical composition and mechanical elasticity pathology of cells in cancer development and their medical treatment researches. The application is, however, suffering from the lack of ability to synchronously acquire the scattering signals following three-dimensional (3D) cell morphology with reasonable spatial resolution and signal-to-noise ratio. Herein, we propose a divided-aperture laser differential confocal 3D Geometry-Raman-Brillouin microscopic detection technology, by which reflection, Raman, and Brillouin scattering signals are simultaneously in situ collected in real time with an axial focusing accuracy up to 1 nm, in the height range of 200 μm. The divided aperture improves the anti-noise capability of the system, and the noise influence depth of Raman detection reduces by 35.4%, and the Brillouin extinction ratio increases by 22 dB. A high-precision multichannel microspectroscopic system containing these functions is developed, which is utilized to study gastric cancer tissue. As a result, a 25% reduction of collagen concentration, 42% increase of DNA substances, 17% and 9% decrease in viscosity and elasticity are finely resolved from the 3D mappings. These findings indicate that our system can be a powerful tool to study cancer development new therapies at the sub-cell level