Database Streaming Compression on Memory-Limited Machines

Dynamic Huffman compression algorithms operate on data-streams with a bounded symbol list. With these algorithms, the complete list of symbols must be contained in main memory or secondary storage. A horizontal format transaction database that is streaming can have a very large item list. Many nodes tax both the processing hardware primary memory size, and the processing time to dynamically maintain the tree. This research investigated Huffman compression of a transaction-streaming database with a very large symbol list, where each item in the transaction database schema’s item list is a symbol to compress. The constraint of a large symbol list is, in this research, equivalent to the constraint of a memory-limited machine. A large symbol set will result if each item in a large database item list is a symbol to compress in a database stream. In addition, database streams may have some temporal component spanning months or years. Finally, the horizontal format is the format most suited to a streaming transaction database because the transaction IDs are not known beforehand This research prototypes an algorithm that will compresses a transaction database stream. There are several advantages to the memory limited dynamic Huffman algorithm. Dynamic Huffman algorithms are single pass algorithms. In many instances a second pass over the data is not possible, such as with streaming databases. Previous dynamic Huffman algorithms are not memory limited, they are asymptotic to O(n), where n is the number of distinct item IDs. Memory is required to grow to fit the n items. The improvement of the new memory limited Dynamic Huffman algorithm is that it would have an O(k) asymptotic memory requirement; where k is the maximum number of nodes in the Huffman tree, k \u3c n, and k is a user chosen constant. The new memory limited Dynamic Huffman algorithm compresses horizontally encoded transaction databases that do not contain long runs of 0’s or 1’s

Analysis of runtime re-configuration systems

In recent years Programmable Logic Devices (PLD) and in particular Field Programmable Gate Arrays (FPGAs) have seen a tremendous increase in sales and applications in the area of embedded systems. The main advantage of FPGAs is the flexibility that they offer a designer in reconfiguring the hardware. The flexibility achieved through re-configuration of FPGAs usually incurs an overhead of extra execution time, data memory and also power dissipation; FPGAs provide an ideal template for run-time reconfigurable (RTR) designs. Only recently have RTR enabling design tools that bypass the traditional synthesis and bitstream generation process for FPGAs become available, JBits is one of them. With run-time reconfiguration of FPGAs, we can perform partial reconfiguration, which allows reconfiguration of a part of an FPGA while the other part is executing some functional computation. The partial reconfiguration of a function can be performed earlier than the time when the function is really needed. Such configuration pre-fetch can hide the reconfiguration overhead more effectively; This thesis will implement a reconfigurable system and study the effect of runtime reconfiguration using VERILOG and a new Java based tool JBITS. This work will provide pointers to high level synthesis tools targeting runtime re-configuration

Lane Compression: A Lightweight Lossless Compression Method for Machine Learning on Embedded Systems

This article presents Lane Compression, a lightweight lossless compression technique for machine learning that is based on a detailed study of the statistical properties of machine learning data. The proposed technique profiles machine learning data gathered ahead of run-time and partitions values bit-wise into different lanes with more distinctive statistical characteristics. Then the most appropriate compression technique is chosen for each lane out of a small number of low-cost compression techniques. Lane Compression’s compute and memory requirements are very low and yet it achieves a compression rate comparable to or better than Huffman coding. We evaluate and analyse Lane Compression on a wide range of machine learning networks for both inference and re-training. We also demonstrate the profiling prior to run-time and the ability to configure the hardware based on the profiling guarantee robust performance across different models and datasets. Hardware implementations are described and the scheme’s simplicity makes it suitable for compressing both on-chip and off-chip traffic. Samsung Advanced Institute of Technology (SAIT

A Modular Approach to Adaptive Reactive Streaming Systems

The latest generations of FPGA devices offer large resource counts that provide the headroom to implement large-scale and complex systems. However, there are increasing challenges for the designer, not just because of pure size and complexity, but also in harnessing effectively the flexibility and programmability of the FPGA. A central issue is the need to integrate modules from diverse sources to promote modular design and reuse. Further, the capability to perform dynamic partial reconfiguration (DPR) of FPGA devices means that implemented systems can be made reconfigurable, allowing components to be changed during operation. However, use of DPR typically requires low-level planning of the system implementation, adding to the design challenge. This dissertation presents ReShape: a high-level approach for designing systems by interconnecting modules, which gives a ‘plug and play’ look and feel to the designer, is supported by tools that carry out implementation and verification functions, and is carried through to support system reconfiguration during operation. The emphasis is on the inter-module connections and abstracting the communication patterns that are typical between modules – for example, the streaming of data that is common in many FPGA-based systems, or the reading and writing of data to and from memory modules. ShapeUp is also presented as the static precursor to ReShape. In both, the details of wiring and signaling are hidden from view, via metadata associated with individual modules. ReShape allows system reconfiguration at the module level, by supporting type checking of replacement modules and by managing the overall system implementation, via metadata associated with its FPGA floorplan. The methodology and tools have been implemented in a prototype for a broad domain-specific setting – networking systems – and have been validated on real telecommunications design projects

JPEG decoder implementation on FPGA using dynamic partial reconfiguration

Trabalho Final de Mestrado para obtenção do grau de Mestre em Engenharia de Electrónica e telecomunicaçõesEsta tese descreve o estudo realizado sobre o tema de Sistemas Computacionais Reconfiguráveis utilizando Field-Programmable Gate Array (FPGA). Sistemas Computacionais Reconfiguráveis é um conceito tão antigo como a computação utilizando circuitos electrónicos. Para explorar os aspetos práticos do conceito, foi implementado um descodificador de imagens codificadas em sistema Baseline JPEGsobre uma FPGA da família Zynq™-7000. Realizado todo o trabalho de desenho, implementação e depuração do descodificador utilizando métodos tradicionais de implementação estática da lógica na FPGA, foi posteriormente realizado o trabalho de adaptação do descodificador desenvolvido para implementação na mesma FPGA utilizando métodos de implementação com reconfiguração parcialdinâmica. Este novo método tem como objetivo principal a realização de um descodificador funcional utilizando apenas uma parte dos recursos lógicos da FPGA quando comparado com a implementação estática do descodificador. A utilização de reconfiguração dinâmica tem como consequência um incremento da complexidade do sistema, originando, numa perspetiva macro, diferenças entre ambos os descodificadores, mas globalmente baseados nos mesmos critérios de desenho e partilhando grande parte dos módulos internos. São ainda descritos os passos para atingir o objetivo, de forma a clarificar o processo de reconfiguração parcial dinâmica para uma aplicação em eventuais novos critérios de projeto e diferentes cenários de aplicação. Esta tese explora ainda o desenvolvimento de sistemas auxiliares que permitem a descodificação direta de ficheiros .jpg e a sua apresentação num monitor VGA.Abstract: This thesis describes a study conducted in Reconfigurable Computing using a Field-Programmable Gate Array (FPGA). Reconfigurable Computing is a concept almost as old as high-speed electronic computing itself. To explore the practical aspects of the concept, a Baseline JPEG image decoder was implemented over a Zynq™-7000 family FPGA. After using traditional methods for the design, implementation and debugging of static decoder logic, the work path was set to adapt the decoder to be implemented on the same FPGA using methods based on Dynamic Partial Reconfiguration. Using this approach the main objective was to develop a working decoder with only a subset of the used resources ofthe FPGA when compared to static implementation of the similar decoder. The dynamic partial reconfiguration brings some additional complexity to the system resulting on two different decoders from a macro perspective view but globally relying on the same design considerations and that share the majority of the internal modules. The steps to achieve the objective are described in order to clarify the dynamic partial reconfiguration process and to eventually open new design possibilities that can be exploited in different application scenarios. The thesis also explores the development of auxiliary systems to enable the ability to decode direct .jpg files and present them on a VGA monitor

GIF image hardware compressors

Increasing requirements for data transfer and storage is one of the crucial questions now. There are several ways of high-speed data transmission, but they meet limited requirements applied to their narrowly focused specific target. The data compression approach gives the solution to the problems of high-speed transfer and low-volume data storage. This paper is devoted to the compression of GIF images, using a modified LZW algorithm with a tree-based dictionary. It has led to a decrease in lookup time and an increase in the speed of data compression, and in turn, allows developing the method of constructing a hardware compression accelerator during the future research

Design and demonstration of digital pre-distortion using software defined radio

Abstract. High data rates for large number of users set tight requirements for signal quality measured in terms of error vector magnitude (EVM). In radio transmitters, nonlinear distortion dominated by power amplifiers (PAs) often limits the achievable EVM. However, the linearity can be improved by linearization techniques. Digital pre-distortion (DPD) is one of these widely used linearization techniques for an effective distortion reduction over a wide bandwidth. In DPD, the nonlinearity of the transmitter is pre-compensated in the digital domain to achieve linear output. Moreover, DPD is used to enable PAs to operate in the power-efficient region with a decent linearity. As we are moving towards millimetre-wave frequencies to enable the wideband communications, the design of the DPD algorithm must be optimized in terms of performance and power consumption. Moreover, continuous development of wireless infrastructure motivates to make research on programmable and reconfigurable platforms in order to decrease the demonstration cost and time, especially for the demonstration purposes. This thesis illustrates and presents how software defined radio (SDR) platforms can be used to demonstrate DPD. Universal software defined peripheral (USRP) X300 is a commercial software defined radio (SDR) platform. The chosen model, X300, has two independent channels equipped with individual transceiver cards. SIMULINK is used to communicate with the device and the two channels of X300 are used as transmitter and receiver simultaneously in full-duplex mode. Hence, a single USRP device is acting as an operational transmitter and feedback receiver, simultaneously. The implemented USRP design consists of SIMULINK based transceiver design and lookup table based DPD in which the coefficients are calculated in MATLAB offline. An external PA, i.e. ZFL-2000+ together with a directional coupler and attenuator are connected between the TX/RX port and RX2 port to measure the nonlinearity. The nonlinearity transceiver is measured with and without the external PA. The experimental results show decent performance for linearization by using the USRP platform. However, the results differ widely due to the used USRP transceiver parameterization and PA operational point. The 16 QAM test signal with 500 kHz bandwidth is fed to the USRP transmit chain. As an example, the DPD algorithm improves the EVM from 7.6% to 2.1% and also the ACPR is reduced around 10 dB with the 16 QAM input signal where approximately + 2.2 dBm input power applied to the external PA