Intelligent agents perform key tasks in several application domains by processing sensor data and taking actions that maximize reward functions based on internal models of the environment and the agent itself. In this paper we present eXplainable Modeling (XM), a Python software which supports data analysis for intelligent agents. XM enables to analyze state-models, namely models of the agent states, discovered from sensor traces by data-driven methods, and to interpret them for improved situation awareness. The main features of the tool are described through the analysis of a real case study concerning aquatic drones for water monitoring
