As space debris poses substantial risks to space-based assets, the need for efficient, high-resolution monitoring and prediction methods is pressing. This paper presents the findings from the project NEU4SST, exploring Neuromorphic Engineering, specifically event-based visual sensing coupled with Spiking Neural Networks (SNNs), as a solution for enhanced Space Domain Awareness (SDA). Our research concentrates on event-based visual sensors and SNNs, offering low power consumption and precise high-resolution data capture and processing. These technologies bolster the ability to detect and track objects in space, addressing key challenges in the Space domain. Our method exceeded previous models by 15% on the informedness metric, demonstrating its potential in improving SDA, and aiding safer, more efficient space operations. Continued research and development in this area are crucial for realising the full potential of Neuromorphic engineering for future space missions

Clemente, Carmine

Di Caterina, Gaetano

Kirkland, Paul

Macdonald, Malcolm

Meoni, Gabriele

University of Strathclyde Institutional Repository

NEUROMORPHIC SENSING AND PROCESSING FOR SPACE DOMAIN AWARENESSPaul Kirkland, Carmine Clemente,Malcolm Macdonald and Gaetano Di Caterina ∗Department of Electrical and Electronic EngineeringUniversity of Strathclyde, Glasgow, UKGabriele MeoniΦ-lab, ESRINEuropean Space AgencyFrascati, ItalyABSTRACTAs space debris poses substantial risks to space-based assets,the need for efficient, high-resolution monitoring and predic-tion methods is pressing. This paper presents the findingsfrom the project NEU4SST, exploring Neuromorphic Engi-neering, specifically event-based visual sensing coupled withSpiking Neural Networks (SNNs), as a solution for enhancedSpace Domain Awareness (SDA). Our research concentrateson event-based visual sensors and SNNs, offering low powerconsumption and precise high-resolution data capture andprocessing. These technologies bolster the ability to detectand track objects in space, addressing key challenges in theSpace domain. Our method exceeded previous models by15% on the informedness metric, demonstrating its potentialin improving SDA, and aiding safer, more efficient spaceoperations. Continued research and development in this areaare crucial for realising the full potential of Neuromorphicengineering for future space missions.Index Terms— Neuromorphic, Event-Based Camera,Spiking Neural Network.1. INTRODUCTIONThe rapid evolution of Space Situational Awareness (SSA)coupled with the vital importance of Space Domain Aware-ness (SDA) necessitates efficient tools for tracking the ever-increasing number of objects in near-Earth space [1, 2]. Thispaper explores a cutting-edge approach that harnesses thesynergy of neuromorphic event-based visual sensors [3] andSpiking Neural Networks (SNNs) [4], to provide a transfor-mative platform for enhancing SSA and SDA capabilities.Neuromorphic sensors, inspired by the workings of thebiological retina, asynchronously capture high temporal res-olution spatial data, consuming significantly less power, aparamount attribute in space applications [3]. Their uniqueability to exploit non-linear signal-noise relationships [5],capturing data at optimal signal-to-noise times, optimisesSSA precision and increases the efficiency of tracking objectsin space.∗This work was funded by the European Space Agency through grant IDI-2021-05556 as part of the Cognitive Cloud Computing in Space campaign.SNNs are the third generation of neural networks, mim-icking human neural processes to adeptly handle the com-plex data from these event-based sensors [4]. This equipsSNNs with enhanced learning and adaptation abilities, of-fering a novel solution to the SSA and SDA challenges. Byenabling simultaneous detection and tracking of moving ob-jects in low signal-to-noise environments [5], this synergyimproves space debris management and collision risk miti-gation, thereby safeguarding both space and terrestrial assets[6, 7].We identify several key areas where neuromorphic en-gineering and SNNs can potentially enhance SSA and SDAcapabilities: capturing space object motion paths with hightemporal resolution; developing a cataloguing system able tohandle increasing numbers of entries; ensuring accuracy suf-ficient for reliable conjunction warnings; and enabling objecttracking during daytime. These areas align with findings froma recent UK Space Agency report [1].In this paper, we present a novel approach to the detectionand tracking of objects in space, based on the combinationof event sensor data and SNN-based neuromorphic process-ing. The experimental results reported indicate that, on thesame dataset, our novel approach can achieve better perfor-mance than the state-of-the-art, which does not employ SNNor neuromorphic processing. At the same time, the approachproposed provides a step closer to realising a complete end-to-end neuromorphic pipeline, from sensing to processing. Thisapproach is data-efficient, meaning that it produces data onlywhen needed, and therefore also energy-efficient. These arekey elements for applications onboard spacecraft.The rest of the paper is organised as follows. Section 2describes the SNN developed in this work, and the data andmetrics used for comparison. Section 3 reports on the experi-mental results obtained, while section 4 concludes the paper.2. METHODOLOGY2.1. Event-based SSA datasetThe study employed the event-based space situational aware-ness (EBSSA) dataset by Afshar et al. [8] as a benchmarkfor testing the SNN developed in this research. This dataset,This is a peer reviewed, accepted author manuscript of the following research contribution: Kirkland, P., Clemente, C., Macdonald, M., Di Caterina, G., & Meoni, G. (Accepted/In press - 03 April 2023). Neuromorphic sensing and processing for space domain awareness. In 2023 International Geoscience and Remote Sensing Symposium, IGARSS2023 - Proceedings IEEE. 1Fig. 1. An accumulated image (3.5 seconds) and breakdownsof the slow- and fast-moving objects (100ms) that need to beextracted simultaneously. (SL8RB-21938)comprising over 8 hours of data and 377 million events, isunique in the literature as the only publicly available event-based space imaging dataset. The data was gathered from arobotic electro-optic telescope facility, using both ATIS [9]and DAVIS [10] event-based sensors, mounted alongside ex-isting astronomy equipment. Hand-labelled ground truth isprovided in the dataset, representing the expert human inter-pretation of the motion of objects in the field of view againstthe highly noisy event stream, offering a comprehensive andchallenging standard for algorithm testing.The dataset provided the opportunity to explore the dy-namics of the non-linear relationship between noise and sig-nal, and to investigate how the highest signal-to-noise ratiomight be achieved through careful information accumulation.The asynchronous event-driven nature of the sensing, with ahigh temporal resolution allowing for the capture of one orfew events, served to highlight the benefits of this approach.Further advantage could be taken of the knowledge that thesignal would display a high degree of spatial locality over ashort period, particularly in the case of very fast-moving ob-jects. This is a feature that can be successfully exploited by aconvolutional network.The advantage of a high sampling rate, combined witha selective accumulation process, is illustrated in Figure 1.This set of images reveals both the overall scene and the mo-tion within it, including two objects moving at very differentspeeds. In the instance of a slow-moving but bright object, ahigh amount of signal is present whenever the object exhibitsrelative motion. Capturing information at these moments, al-though minimal in contrast to the full accumulated images,allows this information to be accumulated, but without thenoise. The fast-moving object, despite appearing as a sparseand incoherent series of events, still produces a higher num-ber of activations in a local area compared to the noise in thebackground, when viewed on a small time scale. Therefore,the highest signal-to-noise opportunities within the SNN canbe found in a similar way. The challenge is in learning thespatial priors required to detect and understand this as onecontinuous object motion path. Example trajectories from theFig. 2. Example trajectories from the SL8RB-21938 [8] .dataset are shown in Fig. 2, illustrating the complexity andvolume of object tracking through space and time.2.2. Spiking Neural NetworkThis work leverages an SNN for processing event-based dataderived from visual sensors. In contrast to traditional neu-ral networks, SNNs employ binary spikes for encoding andtransmitting information, thereby enhancing their compatibil-ity with high-speed, event-based data. SNNs not only excel indata efficiency but also in energy conservation [11], therebybolstering their suitability for deployment in space.This project builds on the foundational work of a spik-ing instance segmentation network [12], employing the samearchitectural blueprint and methodology. To address the com-plex challenge of SSA and motion segmentation, which de-mands simultaneous recognition of various spatio-temporalfeatures, we have transitioned from the Integrate and Fire (IF)neuron model [5] to a set of Leak Integrate-and-Fire (LIF)neurons, with layer-wise leak variation.In this model, we utilise a simplified version of the LIFneuron, whose leak factor extends the range of temporal co-incidence detection. The equation of the LIF neuron modelis:Vi(t) = Vi(t− 1) +∑jwj,iSj(t− 1)− λi (1)where Vi, wj,i, and Sj represent neuron membrane poten-tial, synaptic weight, and spike train of the neuron, respec-tively, while λi is the leak factor and t is time. For this work,the SNN is simulated with discrete input steps of t =100ms,although the data is still fed to the SNN event by event, asspike-timing/sequence is important for learning. λ is also set2Fig. 3. Visual representation of the features being activatedthroughout the network, transforming input events into outputevents segmented with instance labels. (SL8RB-21938)to 90% and 10% of the neuron threshold in layers 1 and 2respectively, using a 5x5 convolution kernel and a 7x7 finalclassification kernel with no leakage. This output is then cou-pled with the instance segmentation method in Kirkland et al.[12] to better handle the segmentation task.The introduction of a layer-wise network leak parameterallows neurons to handle inputs with both short-term andlong-term temporal dependencies. When integrated with theconvolutional hierarchical spatial dependency, this modeleffectively encodes the output of an event-based sensor.Alongside conventional SNN methodologies like homeostasisand adaptive thresholding, the network undergoes unsuper-vised Spike-Timing Dependent Plasticity (STDP) training[13], a proven coincidence detection method for event-baseddata, particularly when paired with convolution kernels [14].Within this work, the learning parameters are α+ = 0.04,α− = 0.03 with a weight initialisation of 0.8±0.01. As thereare not many spatio-temporal features, they converge within1 epoch.2.3. Performance metricsIn accordance with prior studies utilising the EBSSA dataset,we assessed the effectiveness of our model using sensitivity,specificity, and informedness metrics [8]. For consistency, aone-pixel boundary, as outlined in [8], was implemented forspatial domain classification, forming an evaluative volumewhen extended to the temporal domain.Sensitivity and specificity are binary classification modelmeasures, signifying the proportions of true positives andnegatives respectively. Instead informedness combines sen-sitivity and specificity into a singular performance measure.Both the sensitivity and specificity are dependent on the hy-perparameter choices, which is a crucial element given thedynamic nature of the SNN parameters leakage and thresh-old. These show an inverse correlation with sensitivity, whichcould lead to an increased false positives count. However, dueto the hierarchical structure of the network, initial layers canTable 1. Numerical comparison of the proposed SNN ap-proach with previous results from Afshar et al. [8].Algorithm Informedness Sensitivity Specificitymean std mean std mean stdRaw Events 0.324 0.301 0.690 0.340 0.634 0.148Hough D 0.244 0.343 0.552 0.408 0.692 0.224Hough D+T 0.417 0.478 0.442 0.488 0.975 0.073GMD D 0.609 0.323 0.756 0.284 0.853 0.117GMD D+T 0.664 0.374 0.813 0.314 0.851 0.223max (GMD, Hough) D 0.617 0.309 0.754 0.286 0.863 0.103max (GMD, Hough) D+T 0.753 0.344 0.804 0.331 0.950 0.096Feat. D. 0.564 0.443 0.580 0.430 0.984 0.041Feat. D+T 0.775 0.348 0.782 0.349 0.992 0.019SNN (Prop.) 0.891 0.154 0.895 0.153 0.996 0.010exhibit high sensitivity while subsequent layers demonstratehigh specificity, leading to enhanced pixel-level sensitivityand feature-level specificity.3. RESULTSThrough the innovative addition of LIF neurons in the convo-lutional spiking process, the model can assign different leak-age time constants per layer and threshold values on a per-neuron basis, which significantly enhances both learning andinference. This allows the network to simultaneously detectand track short-time/fast-moving and long-time/slow-movingobjects.It is particularly noteworthy how efficiently the networkhandles noise filtering and the detection of spatio-temporalfeatures. The application of the initial 5x5 convolutionalkernel alongside dynamic thresholds and leakage provides arobust mechanism to capture features, even in situations ofsparse event occurrence, fast-moving objects and inconsistenttrack paths (tumbling objects).This study also introduces a novel approach to illustratehow the learned features map onto the input signal to rep-resent the outputs of the network. Figure 3 offers a visualdemonstration of this process: from the input stage wherea time-accumulated image is presented with fast and slow-moving objects (along with the associated noise), through theinitial noise filtering and feature mapping stages, to the finalinstance segmentation output.This work demonstrates the superior performance of anSNN applied to the SSA problem, and its marked improve-ment over the state-of-the-art reported in [8]. The LIF neu-rons and convolution kernels within the network enable ef-ficient noise filtering, resulting in high levels of sensitivityand specificity for instance segmentation. Table 1 summarisesthe performance of the proposed SNN, indicating notable im-provements in specificity (99.6%), sensitivity (89.5%), andinformedness (89.1%). The superior specificity of the SNNis due to its direct mapping to the input spiking activation,3Fig. 4. Example from a difficult scene with fast and slowmoving objects present. This figure shows the ground truthlines (blue) and detected objects (red). (SL8RB-21938)ensuring precise segmentation outputs.Furthermore, the high sensitivity achieved by the net-work underpins its ability to accumulate information withinthe spiking neurons. This, along with the inherent abilityof leaky integrate and fire neurons and convolution kernelsto filter noise allow for signal detection even amidst highlevels of noise. These features of the SNN are advantageousin the second layer, where the network is designed to cap-ture both long and short-term signals, again exploiting theaccumulative nature of the neurons.Figure 4 provides qualitative insights, illustrating a 3Dplot of the ground truth (blue dots) versus network output(red stars). Some instances of the ground truth have no corre-sponding network output (highlighted in orange circles), con-firmed by manual inspection to lack visible features in theevent streams. Smaller ground truth motion paths with lim-ited network output are due to restricted visual data whencompared to the ground truth (highlighted in green circles).4. CONCLUSIONThe event-camera technology shows promising potential inthe field of Space Situational Awareness and Space DomainAwareness, providing valuable event-based data for analy-sis. Our study extends the capabilities of the event-cameraby leveraging the SNN model, demonstrating its effectivenessand efficiency in processing spatio-temporal context data.The SNN model exhibits superior performance in instancesegmentation tasks, surpassing previous models by 15% andfurther enhancing the capabilities of the event-based sensor.By exploiting the input spiking activation and effectively dis-cerning patterns amidst noise, the SNN model achieves im-proved specificity, sensitivity, and informedness.The integration of the event-camera with the SNN modelpresents a compelling approach for efficient and effectiveSSA and SDA work. This combination leverages the uniquedata acquisition properties of the event-based camera, andthe robust noise reduction and pattern recognition capabilitiesof the SNN model, to enhance the analysis of space-relatedphenomena.Future research should focus on refining this integratedapproach, exploring the full potential of event camera data,and optimising the dynamics and parameters of the SNNmodel for different SSA and SDA scenarios. Continuedimprovements in this field hold the promise of advancingour understanding of space dynamics and supporting timelydecision-making in space-related operations.5. REFERENCES[1] CGI, “Space domain awareness study 2022,”Tech. Rep.,UK Space Agency, 2022 [Online].[2] European Space Agency, “Space debris by the num-bers,” Dec. 22, 2022 [Online].[3] G. Gallego et al., “Event-based vision: A survey,” IEEETransactions on Pattern Analysis and Machine Intelli-gence, 2020.[4] Wolfgang Maass, “Networks of spiking neurons: thethird generation of neural network models,” Neural net-works, vol. 10, no. 9, pp. 1659–1671, 1997.[5] Paul Kirkland and Gaetano Di Caterina, “Movementclassification and segmentation using event-based sens-ing and spiking neural networks,” in 2022 Sensor Sig-nal Processing for Defence Conference (SSPD). IEEE,2022, pp. 1–5.[6] Luigi Scatteia, “Space situational awareness,” PwC, Ac-cessed Jan. 10, 2023 [Online].[7] European Space Agency, “When debris disaster strikes,”Nov. 18 2021 [Online].[8] Saeed Afshar et al., “Event-based object detection andtracking for space situational awareness,” IEEE SensorsJournal, vol. 20, no. 24, pp. 15117–15132, 2020.[9] Christoph Posch et al., “A QVGA 143 dB DynamicRange Frame-Free PWM Image Sensor With Loss-less Pixel-Level Video Compression and Time-DomainCDS,” IEEE J. Solid-State Circuits, vol. 46, no. 1, pp.259, 2011.[10] Christian Brandli et al., “A 240 x 180 130 dB 3 µsLatency Global Shutter Spatiotemporal Vision Sensor,”IEEE Journal of Solid-State Circuits, vol. 49, no. 10, pp.2333–2341, oct 2014.[11] Mike Davies et al., “Loihi: A Neuromorphic ManycoreProcessor with On-Chip Learning,”IEEE Micro, vol. 38,no. 1, pp. 82–99, jan 2018.[12] Paul Kirkland et al., “Unsupervised spiking instancesegmentation on event data using stdp features,” IEEETransactions on Computers, pp. 1–12, 2022.[13] Wulfram Gerstner et al., “A neuronal learning rule forsub-millisecond temporal coding,” Nature, vol. 383, no.6595, pp. 76–78, 1996.[14] Timothée Masquelier and Simon J Thorpe, “Unsuper-vised learning of visual features through spike timingdependent plasticity,” PLoS computational biology, vol.3, no. 2, 2007.4

Neuromorphic sensing and processing for space domain awareness

https://strathprints.strath.ac.uk/85675/1/Kirkland_etal_IGARSS_2023_Neuromorphic_sensing_and_processing_for_space_domain_awareness.pdf

Neuromorphic sensing and processing for space domain awareness

Abstract

Similar works

Full text

Available Versions

University of Strathclyde Institutional Repository