The challenges of extract, transform and load (ETL) for data integration in near real-time environment

Organization with considerable investment into data warehousing, the influx of various data types and forms require certain ways of prepping data and staging platform that support fast, efficient and volatile data to reach its targeted audiences or users of different business needs. Extract, Transform and Load (ETL) system proved to be a choice standard for managing and sustaining the movement and transactional process of the valued big data assets. However, traditional ETL system can no longer accommodate and effectively handle streaming or near real-time data and stimulating environment which demands high availability, low latency and horizontal scalability features for functionality. This paper identifies the challenges of implementing ETL system for streaming or near real-time data which needs to evolve and streamline itself with the different requirements. Current efforts and solution approaches to address the challenges are presented. The classification of ETL system challenges are prepared based on near real-time environment features and ETL stages to encourage different perspectives for future research

Extract, Transform, and Load data from Legacy Systems to Azure Cloud

Internship report presented as partial requirement for obtaining the Master’s degree in Information Management, with a specialization in Knowledge Management and Business IntelligenceIn a world with continuously evolving technologies and hardened competitive markets, organisations need to continually be on guard to grasp cutting edge technology and tools that will help them to surpass any competition that arises. Modern data platforms that incorporate cloud technologies, support organisations to strive and get ahead of their competitors by providing solutions that help them capture and optimally use untapped data, and scalable storages to adapt to ever-growing data quantities. Also, adopt data processing and visualisation tools that help to improve the decision-making process. With many cloud providers available in the market, from small players to major technology corporations, this offers much flexibility to organisations to choose the best cloud technology that will align with their use cases and overall products and services strategy. This internship came up at the time when one of Accenture’s significant client in the financial industry decided to migrate from legacy systems to a cloud-based data infrastructure that is Microsoft Azure cloud. During this internship, development of the data lake, which is a core part of the MDP, was done to understand better the type of challenges that can be faced when migrating data from on-premise legacy systems to a cloud-based infrastructure. Also, provided in this work, are the main recommendations and guidelines when it comes to performing a large scale data migration

EasyBDI: integração automática de big data e consultas analíticas de alto nível

Abstract The emergence of new areas, such as the internet of things, which require access to the latest data for data analytics and decision-making environments, created constraints for the execution of analytical queries on traditional data warehouse architectures. In addition, the increase of semi-structure and unstructured data led to the creation of new databases to deal with these types of data, namely, NoSQL databases. This led to the information being stored in several different systems, each with more suitable characteristics for different use cases, which created difficulties in accessing data that are now spread across various systems with different models and characteristics. In this work, a system capable of performing analytical queries in real time on distributed and heterogeneous data sources is proposed: EasyBDI. The system is capable of integrating data logically, without materializing data, creating an overview of the data, thus offering an abstraction over the distribution and heterogeneity of data sources. Queries are executed interactively on data sources, which means that the most recent data will always be used in queries. This system presents a user interface that helps in the configuration of data sources, and automatically proposes a global schema that presents a generic and simplified view of the data, which can be modified by the user. The system allows the creation of multiple star schemas from the global schema. Finally, analytical queries are also made through a user interface that uses drag-and-drop elements. EasyBDI is able to solve recent problems by using recent solutions, hiding the details of several data sources, at the same time that allows users with less knowledge of databases to also be able to perform real-time analytical queries over distributed and heterogeneous data sources.O aparecimento de novas áreas, como a Internet das Coisas, que requerem o acesso aos dados mais recentes para ambientes de tomada de decisão, criou constrangimentos na execução de consultas analíticas usando as arquiteturas tradicionais de data warehouses. Adicionalmente, o aumento de dados semi-estruturados e não estruturados levou a que outras bases de dados fossem criadas para lidar com esse tipo de dados, nomeadamente bases NoSQL. Isto levou a que a informação seja armazenada em sistemas com características distintas e especializados em diferentes casos de uso, criando dificuldades no acesso aos dados que estão agora espalhados por vários sistemas com modelos e características distintas. Neste trabalho, propõe-se um sistema capaz de efetuar consultas analíticas em tempo real sobre fontes de dados distribuídas e heterogéneas: o EasyBDI. O sistema é capaz de integrar dados logicamente, sem materializar os dados, criando uma vista geral dos dados que oferece uma abstração sobre a distribuição e heterogeneidade das fontes de dados. As consultas são executadas interativamente nas fontes de dados, o que significa que os dados mais recentes serão sempre usados nas consultas. Este sistema apresenta uma interface de utilizador que ajuda na configuração de fontes de dados, e propõe automaticamente um esquema global que apresenta a vista genérica e simplificada dos dados, podendo ser modificado pelo utilizador. O sistema permite a criação de múltiplos esquema em estrela a partir do esquema global. Por fim, a realização de consultas analíticas é feita também através de uma interface de utilizador que recorre ao drag-and-drop de elementos. O EasyBDI é capaz de resolver problemas recentes, utilizando também soluções recentes, escondendo os detalhes de diversas fontes de dados, ao mesmo tempo que permite que utilizadores com menos conhecimentos em bases de dados possam também realizar consultas analíticas em tempo-real sobre fontes de dados distribuídas e heterogéneas.Mestrado em Engenharia Informátic

Introduction to Supply Chain and Operations Management – A Real World Perspective

This project was funded by KU Libraries’ Parent’s Campaign with support from the David Shulenburger Office of Scholarly Communication & Copyright and the Open Educational Resources Working Group in the University of Kansas Libraries.This textbook looks at operations management and supply chain management from a real world perspective. The foundations of operations and supply chain are presented in a format that builds upon the theories found in most operations management texts but looks at the applications of the principles through the lens of experience and practice using examples from over 40 years as a practitioner, strategic planner and consultant. The goal of the textbook is to supplement lectures and discussions about operations and supply chain management while linking the topics to other disciplines within a business environment. The topics are grouped based on the Supply Chain Council’s Supply Chain Operations Reference Model core functions.University of Kansas Librarie

Reliability Analysis of On-Demand High-Speed Machining

Current trends in high-speed machining aim to increase manufacturing efficiency by maximizing material removal rates and minimizing part cycle times. This project explores three related technologies and presents a system design for rapid production of custom machined parts. First a reliability analysis in high-speed machining of thin wall features is put forth with experimental results. Second an implementation of on-demand manufacturing is presented with emphasis on flexibility and automation. Finally innovative manufacturing cell design is used to drive costs down by optimizing material and information flow. The resulting high-speed on-demand machining cell design employs effective techniques to reduce production time, meet changing customer needs, and drive down costs

Gender and the Green Economy

The Green Economy is supposed to be sustainable but is it? Being sustainable would entail being equitable. Feminist scholarship shows that the mainstream economy is thoroughly organized by gender, is inequitable, and facilitated by the marginalization of reproductive labor or care work. Ecofeminist theory broadens feminist analysis by situating human social relations in the broader context of our relationship with the environment. In this dissertation I begin from the standpoint of women to explore the degree to which gender inequality is organizing the green economy in the U.S. I argue that a key mechanism reproducing gender inequality is the privileging of green jobs in industries dominated by men and the marginalization and devaluation of environmental care work. I do this by analyzing the organization of the green labor market in the US and through observing the organization and implementation of a program to foster green economic development in an urban area in the Midwest. Understanding the gendered nature of the green economy is important for advancing knowledge about gender segregation and integration of labor markets, gender equality in employment, and gendered opportunities in growing green sector of the economy. This research contributes to scholarship on gender and work, the green economy, ecofeminism, and care work

The use of distance learning in maritime education and training for Malaysia

This dissertation considers the application of distance education (DE) methodology in the field of maritime education and training (MET) for the first time in Malaysia. To date, this technique has only been scarcely used universally, within the context of MET. The research examines the sequential structure and origin of MET within the framework of the country\u27s national policies and background, with emphasis laid on maritime issues. The outcome of this examination provides the rationale for the investigation into the need for the employment of DE as an innovative and novel concept in MET while underlining the benefits for the parties concerned. A general approach introduces this vast DE field. The fundamental concepts of DE are first studied, advantages and drawbacks underlined, and its current application in various areas illustrated, with emphasis on its use in Malaysia. Modern technology is discussed with a view to its possible uses. Various possibilities with regard to DE\u27s untapped potential areas are surveyed. Collaboration with local and foreign DE institutions are compared. The evaluation of the research leads to the design of a proposed course through DE\u27s integration with a conventionally-taught course, that is a hybrid, course, for its possible use at the Maritime Academy of Malaysia (ALAM). Special attention is paid to DE\u27s limitations. An analysis of the research concludes that a powerful policy and a dynamic supportive administration needs to be introduced to back the academic faculty. Various suggestions are offered, citing staff organization structures from a renowned DE institution. The concluding chapter discusses policies in general terms and an appraisal of their feasibility at ALAM, subject to a number of criteria. Various recommendations are made to support the success of the project

Academic Ableism

Academic Ableism brings together disability studies and institutional critique to recognize the ways that disability is composed in and by higher education, and rewrites the spaces, times, and economies of disability in higher education to place disability front and center. For too long, argues Jay Timothy Dolmage, disability has been constructed as the antithesis of higher education, often positioned as a distraction, a drain, a problem to be solved. The ethic of higher education encourages students and teachers alike to accentuate ability, valorize perfection, and stigmatize anything that hints at intellectual, mental, or physical weakness, even as we gesture toward the value of diversity and innovation. Examining everything from campus accommodation processes, to architecture, to popular films about college life, Dolmage argues that disability is central to higher education, and that building more inclusive schools allows better education for all

Creativity in Urban Placemaking: Horizontal Networks and Social Equity in Three Cultural Districts

Many authors point to expanding disparities related to wealth and social benefits brought by globalization and the creative city movement while culture and creativity emerge as growing forces in urban placemaking and economic development. The phenomenon of cultural district formation in cities around the globe presents challenges and opportunities for leaders, planners, and managers. Emerging theory related to cultural districts suggests culture can serve to build horizontal relationships that bridge people and networks from different sectors and professions as well as across ethnicities, class, and interests. Research for this dissertation examined the formation of three urban cultural districts social and their respective organizational networks in different contexts. I employed a multiple case study approach to ask: How do horizontal networks form in the process of planning, organizing and/or ongoing management of cultural districts, and what kinds of benefits do those networks generate within their communities? Field research focused on districts in Los Angeles, Minneapolis, and Miami. This dissertation is positioned within ongoing discourse around the tension between form and function in the production of space (Lefebvre, 1974/1991) and within the dialectic of centralization and decentralization in urban planning and governance (Friedmann, 1971) characterized by the push for broad social equity and the pull of local control. Research found that strong horizontal networks characterized by dense and active grassroots leadership were present at the same time as relative community stability and higher levels of social and economic equity. Where horizontal networks were weak, social and economic tensions were higher. The research did not examine other potential factors and thus cannot ascertain whether strong networks resulted in greater stability and equity or whether stability and more equitable conditions brought on by other factors fostered the formation of stronger networks. This dissertation is available in open access at AURA, http://aura.antioch.edu/etds/ and OhioLink ETD Center, http://www.ohiolink.edu/et