A Comparison of Nuggets and Clusters for Evaluating Timeline Summaries

There is growing interest in systems that generate timeline summaries by filtering high-volume streams of documents to retain only those that are relevant to a particular event or topic. Continued advances in algorithms and techniques for this task depend on standardized and reproducible evaluation methodologies for comparing systems. However, timeline summary evaluation is still in its infancy, with competing methodologies currently being explored in international evaluation forums such as TREC. One area of active exploration is how to explicitly represent the units of information that should appear in a 'good' summary. Currently, there are two main approaches, one based on identifying nuggets in an external 'ground truth', and the other based on clustering system outputs. In this paper, by building test collections that have both nugget and cluster annotations, we are able to compare these two approaches. Specifically, we address questions related to evaluation effort, differences in the final evaluation products, and correlations between scores and rankings generated by both approaches. We summarize advantages and disadvantages of nuggets and clusters to offer recommendations for future system evaluation

Visualizing Business Process Deviance With Timeline Diagrams

Lõputöös püstitatakse kaks peamist küsimust mõiste "hälvete kaevandamine" kohta ja tehakse ettepanek uue meetodi jaoks, mis näitab kahe sündmuse logide erinevusi ajalise dünaamika mõttes, et täita hälvete kaevandamist. Hälvete kaevandamise eesmärk on täpsustada probleemide päritolu ja kõrvalekaldeid. Kogu sellega seotud töö uurimisel on täheldatud, et enamus olemasolevatest meetoditest keskenduvad protsessi põhistruktuurile, mis on ülesannete täitmise järjekord. Uus tehnika näitab tavapäraste ja hälbivate jälgede tegevuste suhtelisi kestusi, st ajalist dünaamikat, joonistades variantide ajakava. Lisaks pakutakse välja tehnika, mis näitab diagrammide kohandamiseks erinevaid seadeid, nagu tulemuslikkuse mõõdik ja protsessi üksikasjalikkus. Lõpuks onvälja arendatud kontseptsioonivahend, mis järgib välja pakutud lähenemisviisi ja on veebis saadaval.The thesis poses two main questions regarding to the notion of “deviance mining” andproposes a new technique to visualise the differences of two event logs in terms oftemporal dynamics in order to perform deviance mining. The objective of deviancemining is to pinpoint the origin of the problems and the deviance. Throughout theresearch of the related work it’s observed that most of the existing methods focus on themain structure of the process which is the order of the tasks being executed. The newtechnique brings out the relative durations i.e temporal dynamics of the activities in thenormal and deviant traces by drawing a timeline diagram of the variants. Additionally theproposed technique puts forward set of different settings such as the performancemeasure and the granularity level of the process to customize the diagram. Lastly, aproof-of-concept tool abiding by the proposed approach is implemented which is servedon the web

Analyzing collaborative learning processes automatically

In this article we describe the emerging area of text classification research focused on the problem of collaborative learning process analysis both from a broad perspective and more specifically in terms of a publicly available tool set called TagHelper tools. Analyzing the variety of pedagogically valuable facets of learners’ interactions is a time consuming and effortful process. Improving automated analyses of such highly valued processes of collaborative learning by adapting and applying recent text classification technologies would make it a less arduous task to obtain insights from corpus data. This endeavor also holds the potential for enabling substantially improved on-line instruction both by providing teachers and facilitators with reports about the groups they are moderating and by triggering context sensitive collaborative learning support on an as-needed basis. In this article, we report on an interdisciplinary research project, which has been investigating the effectiveness of applying text classification technology to a large CSCL corpus that has been analyzed by human coders using a theory-based multidimensional coding scheme. We report promising results and include an in-depth discussion of important issues such as reliability, validity, and efficiency that should be considered when deciding on the appropriateness of adopting a new technology such as TagHelper tools. One major technical contribution of this work is a demonstration that an important piece of the work towards making text classification technology effective for this purpose is designing and building linguistic pattern detectors, otherwise known as features, that can be extracted reliably from texts and that have high predictive power for the categories of discourse actions that the CSCL community is interested in

Mix Multiple Features to Evaluate the Content and the Linguistic Quality of Text Summaries

In this article, we propose a method of text summary\u27s content and linguistic quality evaluation that is based on a machine learning approach. This method operates by combining multiple features to build predictive models that evaluate the content and the linguistic quality of new summaries (unseen) constructed from the same source documents as the summaries used in the training and the validation of models. To obtain the best model, many single and ensemble learning classifiers are tested. Using the constructed models, we have achieved a good performance in predicting the content and the linguistic quality scores. In order to evaluate the summarization systems, we calculated the system score as the average of the score of summaries that are built from the same system. Then, we evaluated the correlation of the system score with the manual system score. The obtained correlation indicates that the system score outperforms the baseline scores

AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges

Artificial Intelligence for IT operations (AIOps) aims to combine the power of AI with the big data generated by IT Operations processes, particularly in cloud infrastructures, to provide actionable insights with the primary goal of maximizing availability. There are a wide variety of problems to address, and multiple use-cases, where AI capabilities can be leveraged to enhance operational efficiency. Here we provide a review of the AIOps vision, trends challenges and opportunities, specifically focusing on the underlying AI techniques. We discuss in depth the key types of data emitted by IT Operations activities, the scale and challenges in analyzing them, and where they can be helpful. We categorize the key AIOps tasks as - incident detection, failure prediction, root cause analysis and automated actions. We discuss the problem formulation for each task, and then present a taxonomy of techniques to solve these problems. We also identify relatively under explored topics, especially those that could significantly benefit from advances in AI literature. We also provide insights into the trends in this field, and what are the key investment opportunities