Optimising HYBRIDJOIN to Process Semi-Stream Data in Near-real-time Data Warehousing

Near-real-time data warehousing plays an essential role for decision making in organizations where latest data is to be fed from various data sources on near-real-time basis. The stream of sales data coming from data sources needs to be transformed to the data warehouse format using disk-based master data. This transformation process is a challenging task due to slow disk access rate as compare to the fast stream data. For this purpose, an adaptive semi-stream join algorithm called HYBRIDJOIN (Hybrid Join) is presented in the literature. The algorithm uses a single buffer to load partitions from the master data. Therefore, the algorithm has to wait until the next disk partition overwrites the existing partition in the buffer. As the cost of loading the disk partition into the buffer is a major cost in the total algorithm’s processing cost, this leaves the performance of the algorithm sub-optimal. This paper presents optimisation of existing HYBRIDJOIN by introducing another buffer. This enables the algorithm to load the second buffer while the first one is under join execution. This reduces the time that the algorithm wait for loading of master data partition and consequently, this improves the performance of the algorithm significantly

A data cube model for analysis of high volumes of ambient data

Ambient systems generate large volumes of data for many of their application areas with XML often the format for data exchange. As a result, large scale ambient systems such as smart cities require some form of optimization before different components can merge their data streams. In data warehousing, the cube structure is often used for optimizing the analytics process with more recent structures such as dwarf, providing new orders of magnitude in terms of optimizing data extraction. However, these systems were developed for relational data and as a result, we now present the development of an XML dwarf to manage ambient systems generating XML data

Enhanced Stream Processing in a DBMS Kernel

Continuous query processing has emerged as a promising query processing paradigm with numerous applications. A recent development is the need to handle both streaming queries and typical one-time queries in the same application. For example, data warehousing can greatly benefit from the integration of stream semantics, i.e., online analysis of incoming data and combination with existing data. This is especially useful to provide low latency in data-intensive analysis in big data warehouses that are augmented with new data on a daily basis. However, state-of-the-art database technology cannot handle streams efficiently due to their "continuous" nature. At the same time, state-of-the-art stream technology is purely focused on stream applications. The research efforts are mostly geared towards the creation of specialized stream management systems built with a different philosophy than a DBMS. The drawback of this approach is the limited opportunities to exploit successful past data processing technology, e.g., query optimization techniques. For this new problem we need to combine the best of both worlds. Here we take a completely different route by designing a stream engine on top of an existing relational database kernel. This includes reuse of both its storage/execution engine and its optimizer infrastructure. The major challenge then becomes the efficient support for specialized stream features. This paper focuses on incremental window-based processing, arguably the most crucial stream-specific requirement. In order to maintain and reuse the generic storage and execution model of the DBMS, we elevate the problem at the query plan level. Proper op

A Flexible Inventory Model for Municipal Solid Waste Recycling

Most of the United States have laws mandating the recycling of municipal solid waste (MSW). In order to comply, municipalities recycle quotas of materials, without regard to fluctuating prices. An inventory system is proposed that allows municipalities to be sensitive to materials prices as they recycle in accordance with state mandates. A dynamic model is developed; it uses historical secondary material prices as exogenous inputs to minimize the net present value of MSW recycling system cost. The model provides a cost-effective method for municipalities to achieve their MSW recycling targets. The savings is approximately $1.43 per ton of MSW generated based on total MSW management costs of$13.5 per ton. The model also allows one to investigate the effectiveness of various strategies for increasing the recycling rate. These strategies include: reducing the transportation cost for recyclables, supporting the market price of selected secondary materials, and landfill bans on selected materials. This model may also be used to investigate the effect of market price changes on the portfolio of materials held in inventory for recycling.Municipal Solid Waste, Recycling, Inventory, Optimization