PACE-LM: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud Incident Root Cause Analysis

Major cloud providers have employed advanced AI-based solutions like large language models to aid humans in identifying the root causes of cloud incidents. Despite the growing prevalence of AI-driven assistants in the root cause analysis process, their effectiveness in assisting on-call engineers is constrained by low accuracy due to the intrinsic difficulty of the task, a propensity for LLM-based approaches to hallucinate, and difficulties in distinguishing these well-disguised hallucinations. To address this challenge, we propose to perform confidence estimation for the predictions to help on-call engineers make decisions on whether to adopt the model prediction. Considering the black-box nature of many LLM-based root cause predictors, fine-tuning or temperature-scaling-based approaches are inapplicable. We therefore design an innovative confidence estimation framework based on prompting retrieval-augmented large language models (LLMs) that demand a minimal amount of information from the root cause predictor. This approach consists of two scoring phases: the LLM-based confidence estimator first evaluates its confidence in making judgments in the face of the current incident that reflects its ``grounded-ness" level in reference data, then rates the root cause prediction based on historical references. An optimization step combines these two scores for a final confidence assignment. We show that our method is able to produce calibrated confidence estimates for predicted root causes, validate the usefulness of retrieved historical data and the prompting strategy as well as the generalizability across different root cause prediction models. Our study takes an important move towards reliably and effectively embedding LLMs into cloud incident management systems

Analyses of the Spring Dust Storm Frequency of Northern China in Relation to Antecedent and Concurrent Wind, Precipitation, Vegetation, and Soil Moisture Conditions

The relationships between the spring (March–May) dust storm frequency (DSF) of northern China, gridded precipitation based on gauge observations, wind velocity at different geopotential heights, satellite-measured land vegetation index, and grid box soil moisture data during 1982–2001 are examined using correlation analysis and singular value decomposition methods. The results show that the spring DSF time series has strong positive correlations with the upwind wind velocity but strong negative correlations with the antecedent summer (June–August) and annual (June of the prior year to May of the current year) precipitation and soil moisture anomalies, as well as with the spring vegetation condition across a region running northeast-southwest from the northeast China and China-Mongolia border to the Taklimakan Desert. This region has been identified as the major source of dust emission in northern China. The results suggest that the summer rainfall anomaly over an extensive area close to the China-Mongolia border is the primary factor that determines the local soil moisture condition in the summer and then the vegetation condition in the following spring through persistence of the soil moisture, eventually determining the variation pattern of the spring DSF in northern China