Predictive modelling : flight delays and associated factors hartsfield–Jackson Atlanta international airport

Project Work presented as the partial requirement for obtaining a Master's degree in Information Management, specialization in Knowledge Management and Business IntelligenceAtualmente, um ponto negativo nas viagens de avião são os atrasos que, constantemente, são anunciados aos passageiros resultando numa diminuição da sua satisfação enquanto clientes. Este e outros fatores fazem com que elevados custos, tanto quantitativos como qualitativos sejam imputados às companhias. Consequentemente, existe a necessidade de prever e mitigar a existência de atrasos aéreos que pode ajudar as companhias aéreas bem como aeroportos a melhorar a sua performance e a aplicar algumas medidas, dirigidas ao consumidor, que permitiam atenuar ou até anular o efeito que estes atrasos provoca nos seus passageiros. Deste modo, este estudo tem como principal objetivo prever a ocorrência de atrasos nas chegadas ao aeroporto internacional de Hartsfield-Jackson. Esta estimativa será possível através da elaboração de um modelo preditivo, recorrendo a diversas técnicas de Data Mining. Com a aplicação destas técnicas, foi possível identificar as variáveis que mais contribuíram para a existência do atraso. No desenvolvimento deste trabalho, foi seguida a metodologia da descoberta de conhecimento em base de dados (conhecida em inglês por Knowledge Discovery Database, KDD). Fases como a recolha dos dados, a aplicação de técnicas de amostragem (SMOTE e Undersampling), a partição dos dados em treino e teste, o pré-processamento (dados omissos e outliers) e transformação dos dados (normalização dos dados e seleção de atributos), a definição de modelos a treinar (Decision Trees, Random Forest e Multilayer Perceptron) bem como a avaliação da performance dos modelos através de métricas variadas foram aplicadas. Depois de testar diferentes abordagens, concluiu-se que o melhor modelo é alcançado com as variáveis relacionadas com a partida, usando o algoritmo Multilayer Perceptron e aplicando a técnica de SMOTE para lidar com dados não balanceados, removendo outliers e selecionando dez variáveis usando GainRatio. Por outro lado, quando as variáveis com informação da partida são excluídas, o algoritmo que melhor se destaca é o Multilayer Perceptron usando a técnica SMOTE, mas desta vez, incluindo os outliers e com quinze variáveis selecionadas novamente pelo GainRatio. Em ambas as hipóteses, as variáveis explicativas que mais contribuem para a existência do atraso na chegada são relacionadas com o clima, com as características do avião e com a propagação do atraso. Os resultados do algoritmo de Random Forests mostraram melhor desempenho, em relação à precisão, em comparação com outros autores (Belcastro, Marozzo, Talia, & Trunfio, 2016; Choi, Kim, Briceno, & Mavris, 2016). Contrariamente, o algoritmo Multilayer Perceptron, apresentou menor precisão em comparação com outro estudo equivalente (Y. J. Kim, Choi, Briceno, & Mavris, 2016).Nowadays, a downside to traveling is the delays that are constantly advertised to passengers resulting in a decrease in customer satisfaction. These delays associated with other factors can cause costs, both quantitative and qualitative. Consequently, there is a need to anticipate and mitigate the existence of airborne delays that can help airlines and airports improving their performance or even take some consumer-oriented measures that can undo or attenuate the effect that these delays have on their passengers. This study has as primary objective to predict the occurrence of arrival delays of the international airport of Hartsfield-Jackson. It was possible by building a predictive model, applying several Data Mining techniques. With these applications, it was possible to show the variables, among the proposals, that most contributed to the existence of the delay. In this work, the Knowledge Discovery Database (KDD) methodology was followed. Phases such as data collection; sampling techniques (SMOTE and Undersampling); Data partitioning in training and testing; Pre-processing (missing data and outliers) and data transformation (data normalization and attribute selection); And, finally the definition of models to be trained (Decision Trees, Random Forests, and Multilayer Perceptron), as well as the evaluation of the performance of the models through varied metrics, were used. After testing different approaches, it was concluded that the best model is achieved with the variables related to departure, using the Multilayer Perceptron algorithm and applying SMOTE to deal with unbalanced data, removing outliers and selecting ten variables using GainRatio. On the other hand, when the variables with information of the departure are excluded, the algorithm that performs best is also the Multilayer Perceptron using the SMOTE technique but, this time, including the outliers and with fifteen variables selected again by the GainRatio. On both hypotheses, the explanatory variables that most contributed to the existence of the delay in arrivals were related to the weather, the airplane characteristics and the propagation of the delay. Our results for the Random Forests algorithm shown better performance, regarding accuracy, compared to other authors (Belcastro et al., 2016; Choi et al., 2016). Contrary, for the Multilayer Perceptron algorithm, was presented a lower accuracy compared to another equivalent study (Y. J. Kim et al., 2016)

Transforming user data into user value by novel mining techniques for extraction of web content, structure and usage patterns. The Development and Evaluation of New Web Mining Methods that enhance Information Retrieval and improve the Understanding of User¿s Web Behavior in Websites and Social Blogs.

The rapid growth of the World Wide Web in the last decade makes it the largest publicly accessible data source in the world, which has become one of the most significant and influential information revolution of modern times. The influence of the Web has impacted almost every aspect of humans' life, activities and fields, causing paradigm shifts and transformational changes in business, governance, and education. Moreover, the rapid evolution of Web 2.0 and the Social Web in the past few years, such as social blogs and friendship networking sites, has dramatically transformed the Web from a raw environment for information consumption to a dynamic and rich platform for information production and sharing worldwide. However, this growth and transformation of the Web has resulted in an uncontrollable explosion and abundance of the textual contents, creating a serious challenge for any user to find and retrieve the relevant information that he truly seeks to find on the Web. The process of finding a relevant Web page in a website easily and efficiently has become very difficult to achieve. This has created many challenges for researchers to develop new mining techniques in order to improve the user experience on the Web, as well as for organizations to understand the true informational interests and needs of their customers in order to improve their targeted services accordingly by providing the products, services and information that truly match the requirements of every online customer. With these challenges in mind, Web mining aims to extract hidden patterns and discover useful knowledge from Web page contents, Web hyperlinks, and Web usage logs. Based on the primary kinds of Web data used in the mining process, Web mining tasks can be categorized into three main types: Web content mining, which extracts knowledge from Web page contents using text mining techniques, Web structure mining, which extracts patterns from the hyperlinks that represent the structure of the website, and Web usage mining, which mines user's Web navigational patterns from Web server logs that record the Web page access made by every user, representing the interactional activities between the users and the Web pages in a website. The main goal of this thesis is to contribute toward addressing the challenges that have been resulted from the information explosion and overload on the Web, by proposing and developing novel Web mining-based approaches. Toward achieving this goal, the thesis presents, analyzes, and evaluates three major contributions. First, the development of an integrated Web structure and usage mining approach that recommends a collection of hyperlinks for the surfers of a website to be placed at the homepage of that website. Second, the development of an integrated Web content and usage mining approach to improve the understanding of the user's Web behavior and discover the user group interests in a website. Third, the development of a supervised classification model based on recent Social Web concepts, such as Tag Clouds, in order to improve the retrieval of relevant articles and posts from Web social blogs