The STONE Curve: A ROC‐Derived Model Performance Assessment Tool

A new model validation and performance assessment tool is introduced, the sliding threshold of observation for numeric evaluation (STONE) curve. It is based on the relative operating characteristic (ROC) curve technique, but instead of sorting all observations in a categorical classification, the STONE tool uses the continuous nature of the observations. Rather than defining events in the observations and then sliding the threshold only in the classifier/model data set, the threshold is changed simultaneously for both the observational and model values, with the same threshold value for both data and model. This is only possible if the observations are continuous and the model output is in the same units and scale as the observations, that is, the model is trying to exactly reproduce the data. The STONE curve has several similarities with the ROC curve—plotting probability of detection against probability of false detection, ranging from the (1,1) corner for low thresholds to the (0,0) corner for high thresholds, and values above the zero‐intercept unity‐slope line indicating better than random predictive ability. The main difference is that the STONE curve can be nonmonotonic, doubling back in both the x and y directions. These ripples reveal asymmetries in the data‐model value pairs. This new technique is applied to modeling output of a common geomagnetic activity index as well as energetic electron fluxes in the Earth’s inner magnetosphere. It is not limited to space physics applications but can be used for any scientific or engineering field where numerical models are used to reproduce observations.Plain Language SummaryScientists often try to reproduce observations with a model, helping them explain the observations by adjusting known and controllable features within the model. They then use a large variety of metrics for assessing the ability of a model to reproduce the observations. One such metric is called the relative operating characteristic (ROC) curve, a tool that assesses a model’s ability to predict events within the data. The ROC curve is made by sliding the event‐definition threshold in the model output, calculating certain metrics and making a graph of the results. Here, a new model assessment tool is introduced, called the sliding threshold of observation for numeric evaluation (STONE) curve. The STONE curve is created by sliding the event definition threshold not only for the model output but also simultaneously for the data values. This is applicable when the model output is trying to reproduce the exact values of a particular data set. While the ROC curve is still a highly valuable tool for optimizing the prediction of known and preclassified events, it is argued here that the STONE curve is better for assessing model prediction of a continuous‐valued data set.Key PointsA new event‐detection‐based metric for model performance appraisal is given with sliding thresholds in both observational and model valuesThe new metric is like the relative operating characteristic curve but uses continuous observational values, not just categorical statusThe new metric is used on real‐time model predictions of common geomagnetic activity parameters, demonstrating its features and strengthsPeer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/156486/2/ess2610.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/156486/1/ess2610_am.pd

Testing the necessity of transient spikes in the storm time ring current drivers

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/95070/1/jgra20908.pd

Contrasting dynamics of electrons and protons in the near-Earth plasma sheet during dipolarization

The fortunate location of Cluster and the THEMIS P3 probe in the near-Earth plasma sheet (PS) (at X similar to -7-9- R-E) allowed for the multipoint analysis of properties and spectra of electron and proton injections. The injections were observed during dipolarization and substorm current wedge formation associated with braking of multiple bursty bulk flows (BBFs). In the course of dipolarization, a gradual growth of the B-Z magnetic field lasted similar to 13 min and it was comprised of several B-Z pulses or dipolarization fronts (DFs) with duration 50 keV) electron fluxes - the injection boundary - was observed in the PS simultaneously with the dipolarization onset and it propagated dawnward along with the onset-related DF. The subsequent dynamics of the energetic electron flux was similar to the dynamics of the magnetic field during the dipolarization. Namely, a gradual linear growth of the electron flux occurred simultaneously with the gradual growth of the B-Z field, and it was comprised of multiple short (similar to few minutes) electron injections associated with the B-Z pulses. This behavior can be explained by the combined action of local betatron acceleration at the B-Z pulses and subsequent gradient drifts of electrons in the flux pile up region through the numerous braking and diverting DFs. The nonadiabatic features occasionally observed in the electron spectra during the injections can be due to the electron interactions with high-frequency electromagnetic or electrostatic fluctuations transiently observed in the course of dipolarization. On the contrary, proton injections were detected only in the vicinity of the strongest B-Z pulses. The front thickness of these pulses was less than a gyroradius of thermal protons that ensured the nonadiabatic acceleration of protons. Indeed, during the injections in the energy spectra of protons the pronounced bulge was clearly observed in a finite energy range similar to 70-90 keV. This feature can be explained by the nonadiabatic resonant acceleration of protons by the bursts of the dawn-dusk electric field associated with the B-Z pulses

Deciphering magnetospheric cross‐field currents

Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/94998/1/grl28658.pd

Model Evaluation Guidelines for Geomagnetic Index Predictions

Geomagnetic indices are convenient quantities that distill the complicated physics of some region or aspect of near‐Earth space into a single parameter. Most of the best‐known indices are calculated from ground‐based magnetometer data sets, such as Dst, SYM‐H, Kp, AE, AL, and PC. Many models have been created that predict the values of these indices, often using solar wind measurements upstream from Earth as the input variables to the calculation. This document reviews the current state of models that predict geomagnetic indices and the methods used to assess their ability to reproduce the target index time series. These existing methods are synthesized into a baseline collection of metrics for benchmarking a new or updated geomagnetic index prediction model. These methods fall into two categories: (1) fit performance metrics such as root‐mean‐square error and mean absolute error that are applied to a time series comparison of model output and observations and (2) event detection performance metrics such as Heidke Skill Score and probability of detection that are derived from a contingency table that compares model and observation values exceeding (or not) a threshold value. A few examples of codes being used with this set of metrics are presented, and other aspects of metrics assessment best practices, limitations, and uncertainties are discussed, including several caveats to consider when using geomagnetic indices.Plain Language SummaryOne aspect of space weather is a magnetic signature across the surface of the Earth. The creation of this signal involves nonlinear interactions of electromagnetic forces on charged particles and can therefore be difficult to predict. The perturbations that space storms and other activity causes in some observation sets, however, are fairly regular in their pattern. Some of these measurements have been compiled together into a single value, a geomagnetic index. Several such indices exist, providing a global estimate of the activity in different parts of geospace. Models have been developed to predict the time series of these indices, and various statistical methods are used to assess their performance at reproducing the original index. Existing studies of geomagnetic indices, however, use different approaches to quantify the performance of the model. This document defines a standardized set of statistical analyses as a baseline set of comparison tools that are recommended to assess geomagnetic index prediction models. It also discusses best practices, limitations, uncertainties, and caveats to consider when conducting a model assessment.Key PointsWe review existing practices for assessing geomagnetic index prediction models and recommend a “standard set” of metricsAlong with fit performance metrics that use all data‐model pairs in their formulas, event detection performance metrics are recommendedOther aspects of metrics assessment best practices, limitations, uncertainties, and geomagnetic index caveats are also discussedPeer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/147764/1/swe20790_am.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/147764/2/swe20790.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/147764/3/swe20790-sup-0001-2018SW002067-SI.pd

SWMF Global Magnetosphere Simulations of January 2005: Geomagnetic Indices and Cross‐Polar Cap Potential

We simulated the entire month of January 2005 using the Space Weather Modeling Framework (SWMF) with observed solar wind data as input. We conducted this simulation with and without an inner magnetosphere model and tested two different grid resolutions. We evaluated the model’s accuracy in predicting Kp, SYM‐H, AL, and cross‐polar cap potential (CPCP). We find that the model does an excellent job of predicting the SYM‐H index, with a root‐mean‐square error (RMSE) of 17–18 nT. Kp is predicted well during storm time conditions but overpredicted during quiet times by a margin of 1 to 1.7 Kp units. AL is predicted reasonably well on average, with an RMSE of 230–270 nT. However, the model reaches the largest negative AL values significantly less often than the observations. The model tended to overpredict CPCP, with RMSE values on the order of 46–48 kV. We found the results to be insensitive to grid resolution, with the exception of the rate of occurrence for strongly negative AL values. The use of the inner magnetosphere component, however, affected results significantly, with all quantities except CPCP improved notably when the inner magnetosphere model was on.Key PointsIncreasing grid resolution from that used by SWPC improves AL prediction during disturbances but has little effect on Kp, SYM‐H, or CPCPThe model does an excellent job at predicting SYM‐H but less well in predicting ALSWMF tends to overpredict Kp and CPCP during quiet times but predicts those quantities better during active timesPeer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/141399/1/swe20534_am.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/141399/2/swe20534.pdfhttps://deepblue.lib.umich.edu/bitstream/2027.42/141399/3/swe20534-sup-0001-supplementary.pd

Challenges associated with near‐Earth nightside current

Every magnetic field of solar and planetary space environments is associated with a current of differentially flowing charged particles. Electric potential patterns in geospace and near other planets are also closely linked with currents. Close to the Earth, particularly in the near‐Earth nightside magnetosphere, several current systems wax and wane during periods of space weather activity. The velocity‐dependent drift, energization, and loss processes in this region complicate current system evolution. There is a discrepancy about the magnitude, timing, and location of these currents, however, and this Commentary pitches the case for a concerted community effort to resolve this issue.Key PointsCurrent densities in the near‐Earth nightside magnetosphere are not well quantifiedCurrent density partitioning between current systems is dynamic and not well understoodResolution of current system issues is important for better space weather forecastingPeer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/134227/1/jgra52747_am.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/134227/2/jgra52747.pd

Forecasting the Earth"s radiation belts and modeling solar energetic particle events: Recent results from SPACECAST

High-energy charged particles in the van Allen radiation belts and in solar energetic particle events can damage satellites on orbit leading to malfunctions and loss of satellite service. Here we describe some recent results from the SPACECAST project on modelling and forecasting the radiation belts, and modelling solar energetic particle events. We describe the SPACECAST forecasting system that uses physical models that include wave-particle interactions to forecast the electron radiation belts up to 3 h ahead. We show that the forecasts were able to reproduce the >2 MeV electron flux at GOES 13 during the moderate storm of 7-8 October 2012, and the period following a fast solar wind stream on 25-26 October 2012 to within a factor of 5 or so. At lower energies of 10- a few 100 keV we show that the electron flux at geostationary orbit depends sensitively on the high-energy tail of the source distribution near 10 RE on the nightside of the Earth, and that the source is best represented by a kappa distribution. We present a new model of whistler mode chorus determined from multiple satellite measurements which shows that the effects of wave-particle interactions beyond geostationary orbit are likely to be very significant. We also present radial diffusion coefficients calculated from satellite data at geostationary orbit which vary with Kp by over four orders of magnitude. We describe a new automated method to determine the position at the shock that is magnetically connected to the Earth for modelling solar energetic particle events and which takes into account entropy, and predict the form of the mean free path in the foreshock, and particle injection efficiency at the shock from analytical theory which can be tested in simulations