Improved techniques for atmospheric ozone retrievals from lidar measurements using the Optimal Estimation Method and Machine Learning

A new first-principle Optimal Estimation Method (OEM) to retrieve ozone number density profiles in both the troposphere and stratosphere using Differential Absorption Lidar (DIAL) measurements obtained at the Observatoire de Haute Provence (OHP) in France is described. The method is robust and applicable to any DIAL ozone lidar. The ozone retrievals are compared to ozonesonde measurements, and these comparisons show the profiles match within the measurement uncertainties. The OEM retrieval also successfully catches much of the structure seen by the ozonesondes. The OEM retrievals are compared with the traditional analysis, and for most heights the difference between the two methods is small. One main advantage of the OEM is that all available measurements from multiple channels as well as lidars are used in the retrieval, eliminating the need to merge or perform corrections on the raw measurement. Thus, the tropospheric and stratospheric lidar measurements can be used together to generate an ozone profile which extends from 2.5\,km to about 42\,km. The upper troposphere and the lower stratosphere (UTLS) coincides with the measurements overlapping region. In the UTLS, even small changes in the distribution of the greenhouse gases can result in large changes in the atmospheric radiative forcing. The OEM can significantly improve the our understanding of the UTLS by providing an ozone density profile with a well-defined statistical and systematic uncertainty budget in this region. A new state-of-the-art machine learning technique was developed to automatically classify raw (level 0) lidar measurements to remove bad scans, and to distinguish between clear sky measurements and measurements with traces of either clouds or aerosols. We have examined different supervised learning methods and found the random forest classifier, the support vector machine (SVM), and the gradient boosting trees could successfully classify our lidar data with more than 90\% accuracy score with the random forest classifier recommended because of its greater computational speed

Classification of lidar measurements using supervised and unsupervised machine learning methods

While it is relatively straightforward to automate the processing of lidar signals, it is more difficult to choose periods of good measurements to process. Groups use various ad hoc procedures involving either very simple (e.g. signal-to-noise ratio) or more complex procedures (e.g. Wing et al. 2018) to perform a task that is easy to train humans to perform but is time-consuming. Here, we use machine learning techniques to train the machine to sort the measurements before processing. The presented method is generic and can be applied to most lidars. We test the techniques using measurements from the Purple Crow Lidar (PCL) system located in London, Canada. The PCL has over 200 000 raw profiles in Rayleigh and Raman channels available for classification. We classify raw (level-0) lidar measurements as clear sky profiles with strong lidar returns, bad profiles, and profiles which are significantly influenced by clouds or aerosol loads. We examined different supervised machine learning algorithms including the random forest, the support vector machine, and the gradient boosting trees, all of which can successfully classify profiles. The algorithms were trained using about 1500 profiles for each PCL channel, selected randomly from different nights of measurements in different years. The success rate of identification for all the channels is above 95 %. We also used the t-SNE) method, which is an unsupervised algorithm, to cluster our lidar profiles. Because the t-SNE is a data-driven method in which no labelling of the training set is needed, it is an attractive algorithm to find anomalies in lidar profiles. The method has been tested on several nights of measurements from the PCL measurements. The t-SNE can successfully cluster the PCL data profiles into meaningful categories. To demonstrate the use of the technique, we have used the algorithm to identify stratospheric aerosol layers due to wildfires

A Bayesian Neural Network Approach for Tropospheric Temperature Retrievals from a Lidar Instrument

We have constructed a Bayesian neural network able of retrieving tropospheric temperature profiles from rotational Raman-scatter measurements of nitrogen and oxygen and applied it to measurements taken by the RAman Lidar for Meteorological Observations (RALMO) in Payerne, Switzerland. We give a detailed description of using a Bayesian method to retrieve temperature profiles including estimates of the uncertainty due to the network weights and the statistical uncertainty of the measurements. We trained our model using lidar measurements under different atmospheric conditions, and we tested our model using measurements not used for training the network. The computed temperature profiles extend over the altitude range of 0.7 km to 6 km. The mean bias estimate of our temperatures relative to the MeteoSwiss standard processing algorithm does not exceed 0.05 K at altitudes below 4.5 km, and does not exceed 0.08 K in an altitude range of 4.5 km to 6 km. This agreement shows that the neural network estimated temperature profiles are in excellent agreement with the standard algorithm. The method is robust and is able to estimate the temperature profiles with high accuracy for both clear and cloudy conditions. Moreover, the trained model can provide the statistical and model uncertainties of the estimated temperature profiles. Thus, the present study is a proof of concept that the trained NNs are able to generate temperature profiles along with a full-budget uncertainty. We present case studies showcasing the Bayesian neural network estimations for day and night measurements, as well as in clear and cloudy conditions. We have concluded that the proposed Bayesian neural network is an appropriate method for the statistical retrieval of temperature profiles

Stratospheric Ozone Density Retrieval Using the Optimal Estimation Method (OEM)

International audienceWe use an Optimal Estimation Method (OEM) to retrieve ozone profiles from the CANDAC Stratospheric Ozone Differential Absorption Lidar in Eureka, Canada. The OEM is a well known inverse method in which a forward model (FM) is used to describe the instrument and geophysical situation. We have developed a FM and are testing its validity using synthetic measurements. We will present the advantages of using OEM retrievals over the traditional method, including a full uncertainty budget

Using the Optimal Estimation Method (OEM) for Retrieval of Stratospheric Ozone Profiles from DIAL lidar measurements

International audienceThe Optimal Estimation Method (OEM) is an inverse method that allows the retrieval of parameters based on measurements and a forward model of the measurement. A complete uncertainty budget on a profile to profile basis, plus the vertical resolution of the measurements as a function of height can be found by this method. We use OEM for the first time to retrieval ozone profiles from a DIAL ozone lidar. The retrievals will be used on measurements from the CANDAC Stratospheric Ozone Differential Absorption Lidar located in Eureka, Canada. We will show results for simulated measurements using one Rayleigh channel. The synthetic pro?les are similar to 3 hours of the real measurements. The ozone is retrieved at 900 m vertical resolution from 7 km to 55 km altitude. The averaging kernel shows essentially no contribution from the a priori below 40 km; above this altitude the response of the averaging kernel drops to 0.8 around 45 km; above which height the retrieval becomes less sensitive to the measurements. The percentage error between the true and retrieved profiles varies between 0.5% to 2% in the region where the retrieval is valid and is less than the statistical uncertainties. In this pilot study background counts are also retrieved. A constant background was used to make synthetic measurements, however, due to the Signal Induced Noise (SIN), the background counts are not constant. We will include the effect of SIN offset on the background counts in the near future. The retrieval is currently being extended to use both of the lidar data channels. Using the two channels, we are planning to retrieve the ozone density profile, the aerosol extinction coefficient, deadtime of the detectors, and the lidar constants. We will then validate the method using measurements of ozone from other instruments, as well as against the traditional DIAL ozone analysis

Implementing Machine Learning Algorithms to Classify Postures and Forecast Motions When Using a Dynamic Chair

Many modern jobs require long periods of sitting on a chair that may result in serious health complications. Dynamic chairs are proposed as alternatives to the traditional sitting chairs; however, previous studies have suggested that most users are not aware of their postures and do not take advantage of the increased range of motion offered by the dynamic chairs. Building a system that identifies users’ postures in real time, as well as forecasts the next few postures, can bring awareness to the sitting behavior of each user. In this study, machine learning algorithms have been implemented to automatically classify users’ postures and forecast their next motions. The random forest, gradient decision tree, and support vector machine algorithms were used to classify postures. The evaluation of the trained classifiers indicated that they could successfully identify users’ postures with an accuracy above 90%. The algorithm can provide users with an accurate report of their sitting habits. A 1D-convolutional-LSTM network has also been implemented to forecast users’ future postures based on their previous motions, the model can forecast a user’s motions with high accuracy (97%). The ability of the algorithm to forecast future postures could be used to suggest alternative postures as needed

Optimal estimation method retrievals of stratospheric ozone profiles from a DIAL

International audienceThis paper provides a detailed description of a first-principle optimal estimation method (OEM) applied to ozone retrieval analysis using differential absorption lidar (DIAL) measurements. The air density, detector dead times, background coefficients, and lidar constants are simultaneously retrieved along with ozone density profiles. Using an averaging kernel, the OEM provides the vertical resolution of the retrieval as a function of altitude. A maximum acceptable height at which the a priori has a small contribution to the retrieval is calculated for each profile as well. Moreover, a complete uncertainty budget including both systematic and statistical uncertainties is given for each individual retrieved profile. Long-term stratospheric DIAL ozone measurements have been carried out at the Observatoire de Haute-Provence (OHP) since 1985. The OEM is applied to three nights of measurements at OHP during an intensive ozone campaign in July 2017 for which coincident lidar-ozonesonde measurements are available. The retrieved ozone density profiles are in good agreement with both traditional analysis and the ozonesonde measurements. For the three nights of measurements , below 15 km the difference between the OEM and the sonde profiles is less than 25 %, and at altitudes between 15 and 25 km the difference is less than 10 %; the OEM can successfully catch many variations in ozone, which are detected in the sonde profiles due to its ability to adjust its vertical resolution as the signal varies. Above 25 km the difference between the OEM and the sonde profiles does not exceed 20 %

Stratospheric Ozone Density Retrieval Using the Optimal Estimation Method (OEM)

We use an Optimal Estimation Method (OEM) to retrieve ozone profiles from the CANDAC Stratospheric Ozone Differential Absorption Lidar in Eureka, Canada. The OEM is a well known inverse method in which a forward model (FM) is used to describe the instrument and geophysical situation. We have developed a FM and are testing its validity using synthetic measurements. We will present the advantages of using OEM retrievals over the traditional method, including a full uncertainty budget

Improved ozone DIAL retrievals in the upper troposphere and lower stratosphere using an optimal estimation method

International audienceWe have implemented a first-principle optimal estimation method to retrieve ozone density profiles using simultaneously tropospheric and stratospheric differential absorption lidar (DIAL) measurements. Our retrieval extends from 2.5 km to about 42 km in altitude, and in the upper troposphere and the lower stratosphere (UTLS) it shows a significant improvement in the overlapping region, where the optimal estimation method (OEM) can retrieve a single ozone profile consistent with the measurements from both lidars. Here stratospheric and tropospheric measurements from the Observatoire de Haute Provence are used, and the OEM retrievals in the UTLS region compared with coincident ozonesonde measurements. The retrieved ozone profiles have a small statistical uncertainty in the UTLS region relative to individual determinations of ozone from each lidar, and the maximum statistical uncertainty does not exceed a maximum of 7%

Updated Climatology of Mesospheric Temperature Inversions Detected by Rayleigh Lidar above Observatoire de Haute Provence, France, Using a K-Mean Clustering Technique

International audienceA climatology of Mesospheric Inversion Layers (MIL) has been created using the Rayleigh lidar located in the south of France at L’Observatoire de Haute Provence (OHP). Using criteria based on lidar measurement uncertainties and climatological mean gravity wave amplitudes, we have selected significant large temperature anomalies that can be associated with MILs. We have tested a novel approach for classifying MILs based on a k-mean clustering technique. We supplied different parameters such as the MIL amplitudes, altitudes, vertical extension, and lapse rate and allowed the computer to classify each individual MIL into one of three clusters or classes. For this first proof of concept study, we selected k = 3 and arrived at three distinct MIL clusters, each of which can be associated with different processes generating MILs in different regimes. All clusters of MIL exhibit a strong seasonal cycle with the largest occurrence in winter. The four decades of measurements do not reveal any long-term changes that can be associated with climate changes and only show an inter-annual variability with a quasi-decadal oscillation