Neural Probabilistic Methods for Event Sequence Modeling

This thesis focuses on modeling event sequences, namely, sequences of discrete events in continuous time. We build a family of generative probabilistic models that is able to reason about what events will happen in the future and when, given the history of previous events. Under our models, each event—as it happens—is allowed to update the future intensities of multiple event types, and the intensity of each event type—as nothing happens—is allowed to evolve with time along a trajectory. We use neural networks to allow the “updates” and “trajectories” to be complex and realistic. In the purely neural version of our model, all future event intensities are conditioned on the hidden state of a continuous-time LSTM, which has consumed every past event as it happened. To exploit domain-specific knowledge of how an event might only affect a few—but not all—future event intensities, we propose to introduce domain-specific structure into the model. We design a modeling language, by which a domain expert can write down the rules of a temporal deductive database. The database tracks facts over time; the rules deduce facts from other facts and from past events. Each fact has a time-varying state, computed by a neural network whose topology is determined by the fact’s provenance, including its experience of the past events that have contributed to deducing it. The possible event types at any time are given by special facts, whose intensities are neurally modeled alongside their states. We develop efficient methods for training our models, and doing inference with them. Applying the general principle of noise-contrastive estimation, we work out a stochastic training objective that is less expensive to optimize than the log-likelihood, which people typically maximize for parameter estimation. As in the discrete-time case that inspired us, the parameters that maximize our objective will provably maximize the log-likelihood as well. For the scenarios where we are given incomplete sequences, we propose particle smoothing—a form of sequential importance sampling—to impute the missing events. This thesis includes extensive experiments, demonstrating the effectiveness of our models and algorithms. On many synthetic and real-world datasets, on held-out sequences, we show empirically: (1) our purely neural model achieves competitive likelihood and predictive accuracy; (2) our neural-symbolic model improves prediction by encoding appropriate domain knowledge in the architecture; (3) for models to achieve the same level of log-likelihood, our noise-contrastive estimation needs considerably fewer function evaluations and less wall-clock time than maximum likelihood estimation; (4) our particle smoothing method is effective at inferring the ground-truth unobserved events. In this thesis, I will also discuss a few future research directions, including embedding our models within a reinforcement learner to discover causal structure and learn an intervention policy

Probability-Based Algorithm for Bearing Diagnosis with Untrained Spall Sizes

Bearing spall detection and predicting its size are great challenges. Model-based simulation is a well-known traditional approach to physically model the influence of the spall on the bearing. Building a physical model is challenging due to the bearing complexity and the expert knowledge required to build such a model. Obviously, building a partial physical model for some of the spall sizes is easier. In this paper, we propose a machine-learning algorithm, called Probability-Based Forest, that uses a partial physical model. First, the behavior of some of the spall sizes is physically modeled and a simulator based on this model generates scenarios for these spall sizes in different conditions. Then, the machine-learning algorithm trains these scenarios to generate a prediction model of spall sizes even for those that have not been modeled by the physical model. Feature extraction is a key factor in the success of this approach. We extract features using two traditional approaches: statistical and physical, and an additional new approach: Time Series FeatuRe Extraction based on Scalable Hypothesis tests (TSFRESH). Experimental evaluation with well-known physical model shows that our approach achieves high accuracy, even in cases that have not been modeled by the physical model. Also, we show that the TSFRESH feature-extraction approach achieves the highest accuracy

Design, organization and implementation of a methods pool and an application systematics for condition based maintenance

Zunehmender Wettbewerb in der Industrie erfordert immer kürzere Amortisationszeiten von kapitalintensiven Produktionsanlangen. Wesentliche Voraussetzungen für die Realisierung kurzer Amortisationszeiträume sind eine hohe Verfügbarkeit der Anlagen und das Erreichen einer gleichmäßig hohen und konstanten Produktqualität. Eine effiziente Instandhaltungsstrategie unterstützt diese Anforderungen an die Verfügbarkeit und an die Produktqualität, vor allem durch eine geringe Bedarfswartung und zunehmend vorbeugende Instandhaltungsbemühungen. In der Industrie wird hierzu häufig die zustandsbasierte Instandhaltung (Condition Based Maintenance - CBM) angewendet. Die CBM Methode versucht aus Zustandseinschätzung der Maschinen, abgeleitet von verschiedenen Zustandsüberwachungs-Verfahren (Condition Monitoring Technique - CMT) und zerstörungsfreien Prüfungen (Nondestructive Test - NDT), erste Mängel zu identifizieren, bevor sie sich kritisch auf die Produktion auswirken. Ein effektives CBM Programm verlangt eine frühe Fehlererkennung und eine genaue Identifikation der Fehlerattribute. Diese Anforderungen werden in der Industrie heute noch unzureichend erfüllt. Die Ursache liegt vor allem in den hohen Kosten, die sich aufgrund unzureichender Information über die potenziellen Fehler ergeben, sowie in der unzulänglichen Kenntnis oder ungeeigneten Anwendung von verschiedenem CMTs und NDTs begründet. Daher werden im Rahmen dieser Arbeit eine neuartige Toolbox und ein Anwendungskonzept entwickelt, um die Umsetzung eines effektiven CBM Programms in der Automobil-Zulieferindustrie zu unterstützen. Hierbei ist der Ansatz so allgemein gewählt, dass er nicht nur auf das Anwendungsgebiet der Automobilindustrie beschränkt ist, sondern auch auf die allgemeine Herstellungs- oder Produktionsindustrie angewendet werden kann. Die CBM-Toolbox setzt sich aus drei Hauptwerkzeugen zusammen. Das erste Werkzeug fasst statistische Fehler-Analysen zusammen, die die in einem Informationssystem des Betriebes vorhandenen Fehlerdaten auswertet, um die relevanten Informationen tabellarisch bzw. grafisch darzustellen. Das zweite Werkzeug ist eine Wissensdatenbank in der das Expertenwissen über verschiedene CMTs und NDTs verwaltet wird. Dieses Expertenwissen ist so strukturiert, dass zusätzlich zu jeder Methode, ihre Anwendbarkeit, Nachweisbarkeit und Vorteile bzw. Nachteile dargestellt werden. Das dritte Werkzeug ist eine objektbasierte Problem-und-Ursache-Analyse, deren Ergebnis eine tabellarisch dargestellte Problem-Ursache Beziehung von besonderen Maschinenanlagen ist. Diese Hauptwerkzeuge werden durch zwei weitere Werkzeuge, ein Finanzanalyse-Werkzeug und eine Auswahlmatrix ergänzt, die die verschiedenen Entscheidungsmöglichkeiten hinsichtlich der Umsetzbarkeit bewertet.The everyday increasing competition in industry and the compulsion of faster investment paybacks for complex and expensive machinery, in addition to operational safety, health and environmental requirements, take for granted high availability of the production machinery and high and stable quality of products. These targets are reached only if the machinery is kept in proper working condition by utilizing an appropriate maintenance tactic. In this frame of thought, monitoring of machinery systems has become progressively more important in meeting the rapidly changing maintenance requirements of today’s manufacturing systems. Besides, as the pressure to reduce manning in plants increases, so does the need for additional automation and reduced organizational level maintenance. Augmented automation in manufacturing plants has led to rapid growth in the number of machinery sensors installed. Along with reduced manning, increased operating tempos are requiring maintenance providers to make repairs faster and ensure that equipment operates reliably for longer periods. To deal with these challenges, condition based maintenance (CBM) has been widely employed within industry. CBM, as a preventive and predictive action, strives to identify incipient faults before they become critical through structural condition assessment derived from Different condition monitoring techniques (CMT) and nondestructive tests (NDT). An effective CBM program requires early recognition of failures and accurate identification of the associated attributes in a feasible manner. The achievement of this proficiency in industry is still intricate and relatively expensive due to deficient information about the potential failures as well as inadequate knowledge or improper application of different CMTs and NDTs. Accordingly, a new toolbox has been developed to facilitate and sustain effective CBM programs in the automotive supply industry. The CBM toolbox is consisted of three major tools. The first tool is a series of statistical failure analyses which uses the failure history data available in a plant’s information system to generate valuable information in tabulated and graphical postures. The second tool is a repository filled with expert knowledge about different CMTs and NDTs formatted in a way that in addition to the concept of each technique, its applicability, detectability, and its pros and cons are expressed. The third tool is an object based problem and cause analysis whose outcome is tabulated problem-cause relationships associated with particular machinery objects. These major tools are also accompanied by two supplementary tools, a financial analysis tool and a selection matrix, to ensure feasibility of all undertaken decisions while using the toolbox