Extraction transformation load (ETL) solution for data integration: a case study of rubber import and export information

Data integration is important in consolidating all the data in the organization or outside the organization to provide a unified view of the organization's information. Extraction Transformation Load (ETL) solution is the back-end process of data integration which involves collecting data from various data sources, preparing and transforming the data according to business requirements and loading them into a Data Warehouse (DW). This paper explains the integration of the rubber import and export data between Malaysian Rubber Board (MRB) and Royal Malaysian Customs Department (Customs) using the ETL solution. Microsoft SQL Server Integration Services (SSIS) and Microsoft SQL Server Agent Jobs have been used as the ETL tool and ETL scheduling

Framework for Interoperable and Distributed Extraction-Transformation-Loading (ETL) Based on Service Oriented Architecture

Extraction. Transformation and Loading (ETL) are the major functionalities in data warehouse (DW) solutions. Lack of component distribution and interoperability is a gap that leads to many problems in the ETL domain, which is due to tightly-coupled components in the current ETL framework. This research discusses how to distribute the Extraction, Transformation and Loading components so as to achieve distribution and interoperability of these ETL components. In addition, it shows how the ETL framework can be extended. To achieve that, Service Oriented Architecture (SOA) is adopted to address the mentioned missing features of distribution and interoperability by restructuring the current ETL framework. This research contributes towards the field of ETL by adding the distribution and inter- operability concepts to the ETL framework. This Ieads to contributions towards the area of data warehousing and business intelligence, because ETL is a core concept in this area. The Design Science Approach (DSA) and Scrum methodologies were adopted for achieving the research goals. The integration of DSA and Scrum provides the suitable methods for achieving the research objectives. The new ETL framework is realized by developing and testing a prototype that is based on the new ETL framework. This prototype is successfully evaluated using three case studies that are conducted using the data and tools of three different organizations. These organizations use data warehouse solutions for the purpose of generating statistical reports that help their top management to take decisions. Results of the case studies show that distribution and interoperability can be achieved by using the new ETL framework

Business intelligence in the electrical power industry

Nowadays, the electrical power industry has gained tremendous interest from both entrepreneurs and researchers due to its essential roles in everyday life. However, the current sources for generating electricity are astonishing decreasing, which leads to more challenges for the power industry. Based on the viewpoint of sustainable development, the solution should maintain three layers of economically, ecologically, and society; simultaneously, support business decision-making, increases organizational productivity and operational energy efficiency. In the smart and innovative technology context, business intelligence solution is considered as a potential option in the data-rich environment, which is still witnessed disjointed theoretical progress. Therefore, this study aimed to conduct a systematic literature review and build a body of knowledge related to business intelligence in the electrical power sector. The author also built an integrative framework displaying linkages between antecedents and outcomes of business intelligence in the electrical power industry. Finally, the paper depicted the underexplored areas of the literature and shed light on the research objectives in terms of theoretical and practical implications

Data and Artificial Intelligence Strategy: A Conceptual Enterprise Big Data Cloud Architecture to Enable Market-Oriented Organisations

Market-Oriented companies are committed to understanding both the needs of their customers, and the capabilities and plans of their competitors through the processes of acquiring and evaluating market information in a systematic and anticipatory manner. On the other hand, most companies in the last years have defined that one of their main strategic objectives for the next years is to become a truly data-driven organisation in the current Big Data context. They are willing to invest heavily in Data and Artificial Intelligence Strategy and build enterprise data platforms that will enable this Market-Oriented vision. In this paper, it is presented an Artificial Intelligence Cloud Architecture capable to help global companies to move from the use of data from descriptive to prescriptive and leveraging existing cloud services to deliver true Market-Oriented in a much shorter time (compared with traditional approaches).This paper has been elaborated with the financing of FEDER funds in the Spanish National research project TIN2016-75850-R from Spanish Department for Economy and Competitiveness

Data and Artificial Intelligence Strategy: A Conceptual Enterprise Big Data Cloud Architecture to Enable Market-Oriented Organisations

Market-Oriented companies are committed to understanding both the needs of their customers, and the capabilities and plans of their competitors through the processes of acquiring and evaluating market information in a systematic and anticipatory manner. On the other hand, most companies in the last years have defined that one of their main strategic objectives for the next years is to become a truly data-driven organisation in the current Big Data context. They are willing to invest heavily in Data and Artificial Intelligence Strategy and build enterprise data platforms that will enable this Market-Oriented vision. In this paper, it is presented an Artificial Intelligence Cloud Architecture capable to help global companies to move from the use of data from descriptive to prescriptive and leveraging existing cloud services to deliver true Market-Oriented in a much shorter time (compared with traditional approaches)

Data aggregation for multi-instance security management tools in telecommunication network

Communication Service Providers employ multiple instances of network monitoring tools within extensive networks that span large geographical regions, encompassing entire countries. By collecting monitoring data from various nodes and consolidating it in a central location, a comprehensive control dashboard is established, presenting an overall network status categorized under different perspectives. In order to achieve this centralized view, we evaluated three architectural options: polling data from individual nodes to a central node, asynchronous push of data from individual nodes to a central node, and a cloud-based Extract, Transform, Load (ETL) approach. Our analysis leads us to the conclusion that the third option is most suitable for the telecommunication system use case. Remarkably, we observed that the quantity of monitoring results is approximately 30 times greater than the total number of devices monitored within the network. Implementing the ETL-based approach, we achieved favorable performance times of 2.23 seconds, 7.16 seconds, and 27.96 seconds for small, medium, and large networks, respectively. Notably, the extraction operation required the most significant amount of time, followed by the load and processing phases. Furthermore, in terms of average memory consumption, the small, medium, and large networks necessitated 323.59 MB, 497.34 MB, and 1668.59 MB, respectively. It is worth noting that the relationship between the total number of devices in the system and both performance and memory consumption is linear in nature

Monitoring spread of epidemic diseases by using clinical data from multiple hospitals: a data warehouse approach

A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of Master’s in Information and Communication Science and Engineering of the Nelson Mandela African Institution of Science and TechnologyMany countries apply data science techniques to enhance their health sectors and the surveillance of diseases. The success of the innovations lies on the availability and quality of datasets to be analyzed. In Tanzania, while different Hospital Management Information Systems (HoMIS) like the Government of Tanzania Hospital Management Information System (GoTHoMIS) are installed in various hospitals, the data stored in the systems are not integrated. This causes unavailability of high quality, timely, anonymous, harmonized, and integrated datasets that can be shared and exhaustively analyzed for epidemic diseases surveillance. This study intended to develop a data warehouse to host patients’ demographic and clinical particulars essential for epidemic diseases surveillance from a multi-node GoT-HoMIS, and yield an integrated dataset that can be used for epidemic diseases surveillance. Interviews were conducted in three strategic health facilities and the Ministry responsible for Health in Tanzania. Documents were reviewed, and observation done on the patient’s registration process in the GoT-HoMIS. Thereafter, a data warehouse was developed to run under MariaDB database server, and using Hypertext Preprocessor an Extract, Transform, and Load (ETL) module was developed. The ETL module was deployed at six health facilities, and the resulting integrated dataset of 152 104 facts was visualized by using FusionCharts libraries. The study demonstrates a novel means to extract data straight from the GoT-HoMIS nodes, which has the potential to make available and provide timely data and integrated reports for decision-making on epidemics. By scaling the innovation to other health facilities, epidemics surveillance can be significantly enhanced