Solution Path Clustering with Adaptive Concave Penalty

Fast accumulation of large amounts of complex data has created a need for more sophisticated statistical methodologies to discover interesting patterns and better extract information from these data. The large scale of the data often results in challenging high-dimensional estimation problems where only a minority of the data shows specific grouping patterns. To address these emerging challenges, we develop a new clustering methodology that introduces the idea of a regularization path into unsupervised learning. A regularization path for a clustering problem is created by varying the degree of sparsity constraint that is imposed on the differences between objects via the minimax concave penalty with adaptive tuning parameters. Instead of providing a single solution represented by a cluster assignment for each object, the method produces a short sequence of solutions that determines not only the cluster assignment but also a corresponding number of clusters for each solution. The optimization of the penalized loss function is carried out through an MM algorithm with block coordinate descent. The advantages of this clustering algorithm compared to other existing methods are as follows: it does not require the input of the number of clusters; it is capable of simultaneously separating irrelevant or noisy observations that show no grouping pattern, which can greatly improve data interpretation; it is a general methodology that can be applied to many clustering problems. We test this method on various simulated datasets and on gene expression data, where it shows better or competitive performance compared against several clustering methods.Comment: 36 page

OCO-3 early mission operations and initial (vEarly) XCO₂ and SIF retrievals

NASA's Orbiting Carbon Observatory-3 (OCO-3) was installed on the International Space Station (ISS) on 10 May 2019. OCO-3 combines the flight spare spectrometer from the Orbiting Carbon Observatory-2 (OCO-2) mission, which has been in operation since 2014, with a new Pointing Mirror Assembly (PMA) that facilitates observations of non-nadir targets from the nadir-oriented ISS platform. The PMA is a new feature of OCO-3, which is being used to collect data in all science modes, including nadir (ND), sun-glint (GL), target (TG), and the new snapshot area mapping (SAM) mode. This work provides an initial assessment of the OCO-3 instrument and algorithm performance, highlighting results from the first 8 months of operations spanning August 2019 through March 2020. During the In-Orbit Checkout (IOC) phase, critical systems such as power and cooling were verified, after which the OCO-3 spectrometer and PMA were subjected to a series of rigorous tests. First light of the OCO-3 spectrometer was on 26 June 2019, with full science operations beginning on 6 August 2019. The OCO-3 spectrometer on-orbit performance is consistent with that seen during preflight testing. Signal to noise ratios are in the expected range needed for high quality retrievals of the column-averaged carbon dioxide (CO₂) dry-air mole fraction (XCO₂) and solar-induced chlorophyll fluorescence (SIF), which will be used to help quantify and constrain the global carbon cycle. The first public release of OCO-3 Level 2 (L2) data products, called “vEarly”, is being distributed by NASA's Goddard Earth Sciences Data and Information Services Center (GES DISC). The intent of the vEarly product is to evaluate early mission performance, facilitate comparisons with OCO-2 products, and identify key areas to improve for the next data release. The vEarly XCO2 exhibits a root-mean-squared-error (RMSE) of ≃ 1, 1, 2 ppm versus a truth proxy for nadir-land, TG&SAM, and glint-water observations, respectively. The vEarly SIF shows a correlation with OCO-2 measurements of >0.9 for highly coincident soundings. Overall, the Level 2 SIF and XCO₂ products look very promising, with performance comparable to OCO-2. A follow-on version of the OCO-3 L2 product containing a number of refinements, e.g., instrument calibration, pointing accuracy, and retrieval algorithm tuning, is anticipated by early in 2021

Solution Path Clustering with Minimax Concave Penalty and Its Applications to Noisy Big Data

Fast accumulation of large amounts of complex data has created a needfor more sophisticated statistical methodologies to discoverinteresting patterns and better extract information from these data.The large scale of the data often results in challenginghigh-dimensional estimation problems where only a minority of the datashows specific grouping patterns. To address these emerging challenges,we develop a new clustering methodology that introduces the idea of aregularization path into unsupervised learning. A regularization pathfor a clustering problem is created by varying the degree of sparsityconstraint that is imposed on the differences between objects via theminimax concave penalty with adaptive tuning parameters. Instead ofproviding a single solution represented by a cluster assignment foreach object, the method produces a short sequence of solutions thatdetermines not only the cluster assignment but also a correspondingnumber of clusters for each solution. The optimization of the penalizedloss function is carried out through an MM algorithm with blockcoordinate descent. The advantages of this clustering algorithmcompared to other existing methods are as follows: it does not requirethe input of the number of clusters; it is capable of simultaneouslyseparating irrelevant or noisy observations that show no groupingpattern, which can greatly improve data interpretation; and it is a generalmethodology that can be applied to many clustering problems. We then develop an iterative subsampling approach to improve the computational efficiencyof this clustering methodology. The proposed approach iterates between clustering a small subsample of the full data and sequentially assigning the other data points to attain orders of magnitude of computational savings. It preserves the ability to isolate noise, includes a solution selection mechanism that ultimately provides one clustering solution with an estimated number of clusters, and is shown to be able to extract small tight clusters from noisy data. The iterative subsampling approach's relatively minor losses in accuracy are demonstrated through simulation studies, and its ability to handle large datasets is illustrated through applications to gene expression datasets

Classification of Anomalous Pixels in the Focal Plane Arrays of Orbiting Carbon Observatory-2 and -3 via Machine Learning

A machine learning approach was developed to improve the bad pixel maps that mask damaged or unusable pixels in the imaging spectrometers of National Aeronautics and Space Administration (NASA) Orbiting Carbon Observatory-2 (OCO-2) and Orbiting Carbon Observatory-3 (OCO-3). The OCO-2 and OCO-3 instruments use nearly 500,000 pixels to record high resolution spectra in three infrared wavelength ranges. These spectra are analyzed to retrieve estimates of the column-average carbon dioxide (XCO 2) concentration in Earth’s atmosphere. To meet mission requirements, these XCO 2 estimates must have accuracies exceeding 0.25%, and small uncertainties in the bias or gain of even one detector pixel can add significant error to the retrieved XCO 2 estimates. Thus, anomalous pixels are identified and removed from the data stream by applying a bad pixel map prior to further processing. To develop these maps, we first characterize each pixel’s behavior through a collection of interpretable and statistically well-defined metrics. These features and a prior map are then used as inputs in a Random Forest classifier to assign a likelihood that a given pixel is bad. Consequently, the likelihoods are analyzed and thresholds are chosen to produce a new bad pixel map. The machine learning approach adopted here has improved data quality by identifying hundreds of new bad pixels in each detector. Such an approach can be generalized to other instruments that require independent calibration of many individual elements

Solution Path Clustering with Minimax Concave Penalty and Its Applications to Noisy Big Data

Fast accumulation of large amounts of complex data has created a needfor more sophisticated statistical methodologies to discoverinteresting patterns and better extract information from these data.The large scale of the data often results in challenginghigh-dimensional estimation problems where only a minority of the datashows specific grouping patterns. To address these emerging challenges,we develop a new clustering methodology that introduces the idea of aregularization path into unsupervised learning. A regularization pathfor a clustering problem is created by varying the degree of sparsityconstraint that is imposed on the differences between objects via theminimax concave penalty with adaptive tuning parameters. Instead ofproviding a single solution represented by a cluster assignment foreach object, the method produces a short sequence of solutions thatdetermines not only the cluster assignment but also a correspondingnumber of clusters for each solution. The optimization of the penalizedloss function is carried out through an MM algorithm with blockcoordinate descent. The advantages of this clustering algorithmcompared to other existing methods are as follows: it does not requirethe input of the number of clusters; it is capable of simultaneouslyseparating irrelevant or noisy observations that show no groupingpattern, which can greatly improve data interpretation; and it is a generalmethodology that can be applied to many clustering problems. We then develop an iterative subsampling approach to improve the computational efficiencyof this clustering methodology. The proposed approach iterates between clustering a small subsample of the full data and sequentially assigning the other data points to attain orders of magnitude of computational savings. It preserves the ability to isolate noise, includes a solution selection mechanism that ultimately provides one clustering solution with an estimated number of clusters, and is shown to be able to extract small tight clusters from noisy data. The iterative subsampling approach's relatively minor losses in accuracy are demonstrated through simulation studies, and its ability to handle large datasets is illustrated through applications to gene expression datasets

Spatial data compression via adaptive dispersion clustering

Spatial Dispersion Clustering (ASDC), a new method of spatial data compression, is specifically designed to reduce the size of a spatial dataset in order to facilitate subsequent spatial prediction. Unlike traditional data and image compression methods, the goal of ASDC is to create a new dataset that will be used as input into spatial-prediction methods, such as traditional kriging or Fixed Rank Kriging, where using the full dataset may be computationally infeasible. ASDC can be classified as a lossy compression method and is based on spectral clustering. It aims to produce contiguous spatial clusters and to preserve the spatial-correlation structure of the data so that the loss of predictive information is minimal. An extensive simulation study demonstrates the predictive performance of these adaptively compressed datasets for several scenarios. ASDC is compared to two other data-reduction schemes, one using local neighborhoods and one using simple binning. An application to remotely sensed sea-surface-temperature data is also presented, and computational costs are discussed

Deepening the Sense of Touch in Planetary Exploration with Geometric and Topological Deep Learning

Tactile and embedded sensing is a new concept that has recently appeared in the context of rovers and planetary exploration missions. Various sensors such as those measuring pressure and integrated directly on wheels have the potential to add a "sense of touch" to exploratory vehicles. We investigate the utility of deep learning (DL), from conventional Convolutional Neural Networks (CNN) to emerging geometric and topological DL, to terrain classification for planetary exploration based on a novel dataset from an experimental tactile wheel concept. The dataset includes 2D conductivity images from a pressure sensor array, which is wrapped around a rover wheel and is able to read pressure signatures of the ground beneath the wheel. Neither newer nor traditional DL tools have been previously applied to tactile sensing data. We discuss insights into advantages and limitations of these methods for the analysis of non-traditional pressure images and their potential use in planetary surface science

TCN: Pioneering Topological-Based Convolutional Networks for Planetary Terrain Learning

Implementations of artificial intelligence (AI) based on deep learning (DL) have proven to be highly successful in many domains, from biomedical imaging to natural language processing, but are still rarely applied in the space industry, particularly for onboard learning of planetary surfaces. In this project, we discuss the utility and limitations of DL, enhanced with topological footprints of the sensed objects, for multi-class classification of planetary surface patterns, in conjunction with tactile and embedded sensing in rover exploratory missions. We consider a Topological Convolutional Network (TCN) model with a persistence-based attention mechanism for supervised classification of various landforms. We study TCN's performance on the Barefoot surface pattern dataset, a novel surface pressure dataset from a prototype tactile rover wheel, known as the Barefoot Rover tactile wheel. Multi-class pattern recognition in the Barefoot data has neither been ever tackled before with DL nor assessed with topological methods. We provide insights into advantages and restrictions of topological DL as the early-stage concept for onboard learning and planetary exploration

OCO-3 Version 11: Better Data and More Data

The Orbiting Carbon Observatory 3 payload was installed on the International Space Station (ISS) on May 10, 2019 and completed in-orbit checkout on August 6, 2019. To enable precise retrievals of the column-average carbon dioxide dry air mole fraction, high resolution spectra of reflected sunlight are collected in three infrared channels. Band 1 (“ABO2”) from 757.6 to 772.7 nm is used to measure molecular oxygen and solar-induced chlorophyll fluorescence, and Bands 2 and 3 measure weak and strong carbon dioxide absorption features from 1591.2 to 1622.7 nm (“WCO2”) and 2042.0 to 2082.8 nm (“SCO2”). The three individual grating spectrometers share a common entrance telescope with a 1.8˚ field of regard, divided into eight along-slit footprints. Spectra are acquired at 3 Hz from an altitude of 400-420 km, with each footprint covering roughly 5 km2. OCO-3 employs an agile two-axis Pointing Mirror Assembly to view Earth in nadir over land, near the glint spot over water, and observe in target and area map modes over locations of interest. The PMA also points into an onboard calibrator to view one of three tungsten halogen lamps or to collect dark measurements. On rare occasions OCO-3 has viewed the Moon but cannot safely view the Sun. The current Level 1B product (Version 10.3) contains a number of imperfections, especially from June 2020 to January 2021 when the instrument was most contaminated. For Version 11, due for public release in 2024, changes were made to ABO2 radiometric calibration, both to the spectrally flat absolute level and the in-band spectral shape. A drift of roughly 1 % per year in Lamp 1, which is assumed to have constant output in Version 10, was corrected. The ratios between preflight and in-orbit checkout were also revised and made footprint-dependent. For spectral shape, data shortly after decontamination is given the spectral shape of Lamp 2 instead of Lamp 1. For later orbits, the spectral shape is based on ratios of ocean spectra to earlier in the cycle. Additional refinements were made to the instrument line shape, spatial response function, and signal to noise coefficients. OCO-3 was scheduled for a three-year prime mission, which concluded successfully in September 2022. Originally, the payload was to be uninstalled to make space for another mission. More recently, an extension was approved for continued operations through the lifetime of ISS in 2029. When the next payload arrives, currently expected in December 2023, OCO-3 will be temporarily stowed for approximately 6 months instead of being disposed. After reinstallation and another 90-day in-orbit checkout, nominal science operations will resume with no further planned interruptions