Data analysis is challenging as it requires synthesizing domain knowledge,
statistical expertise, and programming skills. Assistants powered by large
language models (LLMs), such as ChatGPT, can assist analysts by translating
natural language instructions into code. However, AI-assistant responses and
analysis code can be misaligned with the analyst's intent or be seemingly
correct but lead to incorrect conclusions Therefore, validating AI assistance
is crucial and challenging. Here, we explore how analysts across a range of
backgrounds and expertise understand and verify the correctness of AI-generated
analyses. We develop a design probe that allows analysts to pursue diverse
verification workflows using natural language explanations, code,
visualizations, inspecting data tables, and performing common data operations.
Through a qualitative user study (n=22) using this probe, we uncover common
patterns of verification workflows influenced by analysts' programming,
analysis, and AI backgrounds. Additionally, we highlight open challenges and
opportunities for improving future AI analysis assistant experiences