202 research outputs found
Attributes of Big Data Analytics for Data-Driven Decision Making in Cyber-Physical Power Systems
Big data analytics is a virtually new term in power system terminology. This concept delves into the way a massive volume of data is acquired, processed, analyzed to extract insight from available data. In particular, big data analytics alludes to applications of artificial intelligence, machine learning techniques, data mining techniques, time-series forecasting methods. Decision-makers in power systems have been long plagued by incapability and weakness of classical methods in dealing with large-scale real practical cases due to the existence of thousands or millions of variables, being time-consuming, the requirement of a high computation burden, divergence of results, unjustifiable errors, and poor accuracy of the model. Big data analytics is an ongoing topic, which pinpoints how to extract insights from these large data sets. The extant article has enumerated the applications of big data analytics in future power systems through several layers from grid-scale to local-scale. Big data analytics has many applications in the areas of smart grid implementation, electricity markets, execution of collaborative operation schemes, enhancement of microgrid operation autonomy, management of electric vehicle operations in smart grids, active distribution network control, district hub system management, multi-agent energy systems, electricity theft detection, stability and security assessment by PMUs, and better exploitation of renewable energy sources. The employment of big data analytics entails some prerequisites, such as the proliferation of IoT-enabled devices, easily-accessible cloud space, blockchain, etc. This paper has comprehensively conducted an extensive review of the applications of big data analytics along with the prevailing challenges and solutions
User-Friendly Covariance Estimation for Heavy-Tailed Distributions
We offer a survey of recent results on covariance estimation for heavy-tailed
distributions. By unifying ideas scattered in the literature, we propose
user-friendly methods that facilitate practical implementation. Specifically,
we introduce element-wise and spectrum-wise truncation operators, as well as
their -estimator counterparts, to robustify the sample covariance matrix.
Different from the classical notion of robustness that is characterized by the
breakdown property, we focus on the tail robustness which is evidenced by the
connection between nonasymptotic deviation and confidence level. The key
observation is that the estimators needs to adapt to the sample size,
dimensionality of the data and the noise level to achieve optimal tradeoff
between bias and robustness. Furthermore, to facilitate their practical use, we
propose data-driven procedures that automatically calibrate the tuning
parameters. We demonstrate their applications to a series of structured models
in high dimensions, including the bandable and low-rank covariance matrices and
sparse precision matrices. Numerical studies lend strong support to the
proposed methods.Comment: 56 pages, 2 figure
FarmTest: Factor-Adjusted Robust Multiple Testing with Approximate False Discovery Control
Large-scale multiple testing with correlated and heavy-tailed data arises in
a wide range of research areas from genomics, medical imaging to finance.
Conventional methods for estimating the false discovery proportion (FDP) often
ignore the effect of heavy-tailedness and the dependence structure among test
statistics, and thus may lead to inefficient or even inconsistent estimation.
Also, the commonly imposed joint normality assumption is arguably too stringent
for many applications. To address these challenges, in this paper we propose a
Factor-Adjusted Robust Multiple Testing (FarmTest) procedure for large-scale
simultaneous inference with control of the false discovery proportion. We
demonstrate that robust factor adjustments are extremely important in both
controlling the FDP and improving the power. We identify general conditions
under which the proposed method produces consistent estimate of the FDP. As a
byproduct that is of independent interest, we establish an exponential-type
deviation inequality for a robust -type covariance estimator under the
spectral norm. Extensive numerical experiments demonstrate the advantage of the
proposed method over several state-of-the-art methods especially when the data
are generated from heavy-tailed distributions. The proposed procedures are
implemented in the R-package FarmTest.Comment: 52 pages, 9 figure
- …