Heuristics Miners for Streaming Event Data

More and more business activities are performed using information systems. These systems produce such huge amounts of event data that existing systems are unable to store and process them. Moreover, few processes are in steady-state and due to changing circumstances processes evolve and systems need to adapt continuously. Since conventional process discovery algorithms have been defined for batch processing, it is difficult to apply them in such evolving environments. Existing algorithms cannot cope with streaming event data and tend to generate unreliable and obsolete results. In this paper, we discuss the peculiarities of dealing with streaming event data in the context of process mining. Subsequently, we present a general framework for defining process mining algorithms in settings where it is impossible to store all events over an extended period or where processes evolve while being analyzed. We show how the Heuristics Miner, one of the most effective process discovery algorithms for practical applications, can be modified using this framework. Different stream-aware versions of the Heuristics Miner are defined and implemented in ProM. Moreover, experimental results on artificial and real logs are reported

Mining Transaction Data for Process Instance Monitoring in Legacy Systems

End-to-End business processes in organizations are implemented across multiple applications, legacy systems, ERP systemsand products. In such scenarios where applications are developed over a period of time and with varying technologies,monitoring end-to-end business processes is a challenge. Typical methods for providing process monitoring capabilities areintrusive methods like changing code and introducing probes; or introducing new software tools like EAI and BAM. Wepropose a non-intrusive process instance monitoring (PIM) method that uses the persistent data generated by the businesstransactions to monitor the process instances in Legacy Information Systems. We propose a slightly unconventional datamining method where the transaction data is parsed from the application data stores, loaded into custom schema and thenassociated to the process flow for monitoring the state of individual process instances. The approach further provides foralerting when business events like an SLA violation occur

Fast Data in the Era of Big Data: Twitter's Real-Time Related Query Suggestion Architecture

We present the architecture behind Twitter's real-time related query suggestion and spelling correction service. Although these tasks have received much attention in the web search literature, the Twitter context introduces a real-time "twist": after significant breaking news events, we aim to provide relevant results within minutes. This paper provides a case study illustrating the challenges of real-time data processing in the era of "big data". We tell the story of how our system was built twice: our first implementation was built on a typical Hadoop-based analytics stack, but was later replaced because it did not meet the latency requirements necessary to generate meaningful real-time results. The second implementation, which is the system deployed in production, is a custom in-memory processing engine specifically designed for the task. This experience taught us that the current typical usage of Hadoop as a "big data" platform, while great for experimentation, is not well suited to low-latency processing, and points the way to future work on data analytics platforms that can handle "big" as well as "fast" data