4,160 research outputs found
Recommended from our members
Data compressions on machines with limited memory
We consider two problems in which machines with limited internal memory are used to compress and decompress data. In the first application, a powerful encoder transmits a coded file to a decoder that has severely constrained memory. A data structure that achieves minimum storage is presented, and alternative methods that sacrifice a small amount of storage to attain faster decoding are described. The second problem we address is that of encoding and decoding in limited memory. Methods for representing context models succinctly are described. These methods provide compression performance that is superior to state-of-the-art techniques, and competitive with newer approaches that use five times as much internal memory
New Algorithms and Lower Bounds for Sequential-Access Data Compression
This thesis concerns sequential-access data compression, i.e., by algorithms
that read the input one or more times from beginning to end. In one chapter we
consider adaptive prefix coding, for which we must read the input character by
character, outputting each character's self-delimiting codeword before reading
the next one. We show how to encode and decode each character in constant
worst-case time while producing an encoding whose length is worst-case optimal.
In another chapter we consider one-pass compression with memory bounded in
terms of the alphabet size and context length, and prove a nearly tight
tradeoff between the amount of memory we can use and the quality of the
compression we can achieve. In a third chapter we consider compression in the
read/write streams model, which allows us passes and memory both
polylogarithmic in the size of the input. We first show how to achieve
universal compression using only one pass over one stream. We then show that
one stream is not sufficient for achieving good grammar-based compression.
Finally, we show that two streams are necessary and sufficient for achieving
entropy-only bounds.Comment: draft of PhD thesi
Data structures
We discuss data structures and their methods of analysis. In particular, we treat the unweighted and weighted dictionary problem, self-organizing data structures, persistent data structures, the union-find-split problem, priority queues, the nearest common ancestor problem, the selection and merging problem, and dynamization techniques. The methods of analysis are worst, average and amortized case
Learning, Categorization, Rule Formation, and Prediction by Fuzzy Neural Networks
National Science Foundation (IRI 94-01659); Office of Naval Research (N00014-91-J-4100, N00014-92-J-4015) Air Force Office of Scientific Research (90-0083, N00014-92-J-4015
An overview of decision table literature 1982-1995.
This report gives an overview of the literature on decision tables over the past 15 years. As much as possible, for each reference, an author supplied abstract, a number of keywords and a classification are provided. In some cases own comments are added. The purpose of these comments is to show where, how and why decision tables are used. The literature is classified according to application area, theoretical versus practical character, year of publication, country or origin (not necessarily country of publication) and the language of the document. After a description of the scope of the interview, classification results and the classification by topic are presented. The main body of the paper is the ordered list of publications with abstract, classification and comments.
A review of clustering techniques and developments
Β© 2017 Elsevier B.V. This paper presents a comprehensive study on clustering: exiting methods and developments made at various times. Clustering is defined as an unsupervised learning where the objects are grouped on the basis of some similarity inherent among them. There are different methods for clustering the objects such as hierarchical, partitional, grid, density based and model based. The approaches used in these methods are discussed with their respective states of art and applicability. The measures of similarity as well as the evaluation criteria, which are the central components of clustering, are also presented in the paper. The applications of clustering in some fields like image segmentation, object and character recognition and data mining are highlighted
DCT Implementation on GPU
There has been a great progress in the field of graphics processors. Since, there is no rise in the speed of the normal CPU processors; Designers are coming up with multi-core, parallel processors. Because of their popularity in parallel processing, GPUs are becoming more and more attractive for many applications. With the increasing demand in utilizing GPUs, there is a great need to develop operating systems that handle the GPU to full capacity. GPUs offer a very efficient environment for many image processing applications. This thesis explores the processing power of GPUs for digital image compression using Discrete cosine transform
κ°κ²°ν μλ£κ΅¬μ‘°λ₯Ό νμ©ν λ°κ΅¬μ‘°νλ λ¬Έμ νμλ€μ κ³΅κ° ν¨μ¨μ ννλ²
νμλ
Όλ¬Έ (λ°μ¬) -- μμΈλνκ΅ λνμ : 곡과λν μ κΈ°Β·μ»΄ν¨ν°κ³΅νλΆ, 2021. 2. Srinivasa Rao Satti.Numerous big data are generated from a plethora of sources. Most of the data stored as files contain a non-fixed type of schema, so that the files are suitable to be maintained as semi-structured document formats. A number of those formats, such as XML (eXtensible Markup Language), JSON (JavaScript Object Notation), and YAML (YAML Ain't Markup Language) are suggested to sustain hierarchy in the original corpora of data. Several data models structuring the gathered data - including RDF (Resource Description Framework) - depend on the semi-structured document formats to be serialized and transferred for future processing.
Since the semi-structured document formats focus on readability and verbosity, redundant space is required to organize and maintain the document. Even though general-purpose compression schemes are widely used to compact the documents, applying those algorithms hinder future handling of the corpora, owing to loss of internal structures.
The area of succinct data structures is widely investigated and researched in theory, to provide answers to the queries while the encoded data occupy space close to the information-theoretic lower bound. Bit vectors and trees are the notable succinct data structures. Nevertheless, there were few attempts to apply the idea of succinct data structures to represent the semi-structured documents in space-efficient manner.
In this dissertation we propose a unified, space-efficient representation of various semi-structured document formats. The core functionality of this representation is its compactness and query-ability derived from enriched functions of succinct data structures. Incorporation of (a) bit indexed arrays, (b) succinct ordinal trees, and (c) compression techniques engineers the compact representation. We implement this representation in practice, and show by experiments that construction of this representation decreases the disk usage by up to 60% while occupying 90% less RAM. We also allow processing a document in partial manner, to allow processing of larger corpus of big data even in the constrained environment.
In parallel to establishing the aforementioned compact semi-structured document representation, we provide and reinforce some of the existing compression schemes in this dissertation. We first suggest an idea to encode an array of integers that is not necessarily sorted. This compaction scheme improves upon the existing universal code systems, by assistance of succinct bit vector structure. We show that our suggested algorithm reduces space usage by up to 44% while consuming 15% less time than the original code system, while the algorithm additionally supports random access of elements upon the encoded array.
We also reinforce the SBH bitmap index compression algorithm. The main strength of this scheme is the use of intermediate super-bucket during operations, giving better performance on querying through a combination of compressed bitmap indexes. Inspired from splits done during the intermediate process of the SBH algorithm, we give an improved compression mechanism supporting parallelism that could be utilized in both CPUs and GPUs. We show by experiments that this CPU parallel processing optimization diminishes compression and decompression times by up to 38% in a 4-core machine without modifying the bitmap compressed form. For GPUs, the new algorithm gives 48% faster query processing time in the experiments, compared to the previous existing bitmap index compression schemes.μ
μ μλ λΉ
λ°μ΄ν°κ° λ€μν μλ³Έλ‘λΆν° μμ±λκ³ μλ€. μ΄λ€ λ°μ΄ν°μ λλΆλΆμ κ³ μ λμ§ μμ μ’
λ₯μ μ€ν€λ§λ₯Ό ν¬ν¨ν νμΌ ννλ‘ μ μ₯λλλ°, μ΄λ‘ μΈνμ¬ λ°κ΅¬μ‘°νλ λ¬Έμ νμμ μ΄μ©νμ¬ νμΌμ μ μ§νλ κ²μ΄ μ ν©νλ€. XML, JSON λ° YAMLκ³Ό κ°μ μ’
λ₯μ λ°κ΅¬μ‘°νλ λ¬Έμ νμμ΄ λ°μ΄ν°μ λ΄μ¬νλ ꡬ쑰λ₯Ό μ μ§νκΈ° μνμ¬ μ μλμλ€. μμ§λ λ°μ΄ν°λ₯Ό ꡬ쑰ννλ RDFμ κ°μ μ¬λ¬ λ°μ΄ν° λͺ¨λΈλ€μ μ¬ν μ²λ¦¬λ₯Ό μν μ μ₯ λ° μ μ‘μ μνμ¬ λ°κ΅¬μ‘°νλ λ¬Έμ νμμ μμ‘΄νλ€.
λ°κ΅¬μ‘°νλ λ¬Έμ νμμ κ°λ
μ±κ³Ό λ€λ³μ±μ μ§μ€νκΈ° λλ¬Έμ, λ¬Έμλ₯Ό ꡬ쑰ννκ³ μ μ§νκΈ° μνμ¬ μΆκ°μ μΈ κ³΅κ°μ νμλ‘ νλ€. λ¬Έμλ₯Ό μμΆμν€κΈ° μνμ¬ μΌλ°μ μΈ μμΆ κΈ°λ²λ€μ΄ λ리 μ¬μ©λκ³ μμΌλ, μ΄λ€ κΈ°λ²λ€μ μ μ©νκ² λλ©΄ λ¬Έμμ λ΄λΆ ꡬ쑰μ μμ€λ‘ μΈνμ¬ λ°μ΄ν°μ μ¬ν μ²λ¦¬κ° μ΄λ ΅κ² λλ€.
λ°μ΄ν°λ₯Ό μ 보μ΄λ‘ μ ννμ κ°κΉμ΄ 곡κ°λ§μ μ¬μ©νμ¬ μ μ₯μ κ°λ₯νκ² νλ©΄μ μ§μμ λν μλ΅μ μ 곡νλ κ°κ²°ν μλ£κ΅¬μ‘°λ μ΄λ‘ μ μΌλ‘ λ리 μ°κ΅¬λκ³ μλ λΆμΌμ΄λ€. λΉνΈμ΄κ³Ό νΈλ¦¬κ° λ리 μλ €μ§ κ°κ²°ν μλ£κ΅¬μ‘°λ€μ΄λ€. κ·Έλ¬λ λ°κ΅¬μ‘°νλ λ¬Έμλ€μ μ μ₯νλ λ° κ°κ²°ν μλ£κ΅¬μ‘°μ μμ΄λμ΄λ₯Ό μ μ©ν μ°κ΅¬λ κ±°μ μ§νλμ§ μμλ€.
λ³Έ νμλ
Όλ¬Έμ ν΅ν΄ μ°λ¦¬λ λ€μν μ’
λ₯μ λ°κ΅¬μ‘°νλ λ¬Έμ νμμ ν΅μΌλκ² νννλ κ³΅κ° ν¨μ¨μ ννλ²μ μ μνλ€. μ΄ κΈ°λ²μ μ£Όμν κΈ°λ₯μ κ°κ²°ν μλ£κ΅¬μ‘°κ° κ°μ μΌλ‘ κ°μ§λ νΉμ±μ κΈ°λ°ν κ°κ²°μ±κ³Ό μ§μ κ°λ₯μ±μ΄λ€. λΉνΈμ΄λ‘ μΈλ±μ±λ λ°°μ΄, κ°κ²°ν μμ μλ νΈλ¦¬ λ° λ€μν μμΆ κΈ°λ²μ ν΅ν©νμ¬ ν΄λΉ ννλ²μ κ³ μνμλ€. μ΄ κΈ°λ²μ μ€μ¬μ μΌλ‘ ꡬνλμκ³ , μ€νμ ν΅νμ¬ μ΄ κΈ°λ²μ μ μ©ν λ°κ΅¬μ‘°νλ λ¬Έμλ€μ μ΅λ 60% μ μ λμ€ν¬ 곡κ°κ³Ό 90% μ μ λ©λͺ¨λ¦¬ 곡κ°μ ν΅ν΄ ννλ μ μλ€λ κ²μ 보μΈλ€. λλΆμ΄ λ³Έ νμλ
Όλ¬Έμμ λ°κ΅¬μ‘°νλ λ¬Έμλ€μ λΆν μ μΌλ‘ ννμ΄ κ°λ₯ν¨μ 보μ΄κ³ , μ΄λ₯Ό ν΅νμ¬ μ νλ νκ²½μμλ λΉ
λ°μ΄ν°λ₯Ό ννν λ¬Έμλ€μ μ²λ¦¬ν μ μλ€λ κ²μ 보μΈλ€.
μμ μΈκΈν κ³΅κ° ν¨μ¨μ λ°κ΅¬μ‘°νλ λ¬Έμ ννλ²μ ꡬμΆν¨κ³Ό λμμ, λ³Έ νμλ
Όλ¬Έμμ μ΄λ―Έ μ‘΄μ¬νλ μμΆ κΈ°λ² μ€ μΌλΆλ₯Ό μΆκ°μ μΌλ‘ κ°μ νλ€. 첫째λ‘, λ³Έ νμλ
Όλ¬Έμμλ μ λ ¬ μ¬λΆμ κ΄κ³μλ μ μ λ°°μ΄μ λΆνΈννλ μμ΄λμ΄λ₯Ό μ μνλ€. μ΄ κΈ°λ²μ μ΄λ―Έ μ‘΄μ¬νλ λ²μ© μ½λ μμ€ν
μ κ°μ ν ννλ‘, κ°κ²°ν λΉνΈμ΄ μλ£κ΅¬μ‘°λ₯Ό μ΄μ©νλ€. μ μλ μκ³ λ¦¬μ¦μ κΈ°μ‘΄ λ²μ© μ½λ μμ€ν
μ λΉν΄ μ΅λ 44\% μ μ 곡κ°μ μ¬μ©ν λΏλ§ μλλΌ 15\% μ μ λΆνΈν μκ°μ νμλ‘ νλ©°, κΈ°μ‘΄ μμ€ν
μμ μ 곡νμ§ μλ λΆνΈνλ λ°°μ΄μμμ μμ μ κ·Όμ μ§μνλ€.
λν λ³Έ νμλ
Όλ¬Έμμλ λΉνΈλ§΅ μΈλ±μ€ μμΆμ μ¬μ©λλ SBH μκ³ λ¦¬μ¦μ κ°μ μν¨λ€. ν΄λΉ κΈ°λ²μ μ£Όλ κ°μ μ λΆνΈνμ 볡νΈν μ§ν μ μ€κ° 맀κ°μΈ μνΌλ²μΌμ μ¬μ©ν¨μΌλ‘μ¨ μ¬λ¬ μμΆλ λΉνΈλ§΅ μΈλ±μ€μ λν μ§μ μ±λ₯μ κ°μ μν€λ κ²μ΄λ€. μ μμΆ μκ³ λ¦¬μ¦μ μ€κ° κ³Όμ μμ μ§νλλ λΆν μμ μκ°μ μ»μ΄, λ³Έ νμλ
Όλ¬Έμμ CPU λ° GPUμ μ μ© κ°λ₯ν κ°μ λ λ³λ ¬ν μμΆ λ§€μ»€λμ¦μ μ μνλ€. μ€νμ ν΅ν΄ CPU λ³λ ¬ μ΅μ νκ° μ΄λ£¨μ΄μ§ μκ³ λ¦¬μ¦μ μμΆλ ννμ λ³ν μμ΄ 4μ½μ΄ μ»΄ν¨ν°μμ μ΅λ 38\%μ μμΆ λ° ν΄μ μκ°μ κ°μμν¨λ€λ κ²μ 보μΈλ€. GPU λ³λ ¬ μ΅μ νλ κΈ°μ‘΄μ μ‘΄μ¬νλ GPU λΉνΈλ§΅ μμΆ κΈ°λ²μ λΉν΄ 48\% λΉ λ₯Έ μ§μ μ²λ¦¬ μκ°μ νμλ‘ ν¨μ νμΈνλ€.Chapter 1 Introduction 1
1.1 Contribution 3
1.2 Organization 5
Chapter 2 Background 6
2.1 Model of Computation 6
2.2 Succinct Data Structures 7
Chapter 3 Space-efficient Representation of Integer Arrays 9
3.1 Introduction 9
3.2 Preliminaries 10
3.2.1 Universal Code System 10
3.2.2 Bit Vector 13
3.3 Algorithm Description 13
3.3.1 Main Principle 14
3.3.2 Optimization in the Implementation 16
3.4 Experimental Results 16
Chapter 4 Space-efficient Parallel Compressed Bitmap Index Processing 19
4.1 Introduction 19
4.2 Related Work 23
4.2.1 Byte-aligned Bitmap Code (BBC) 24
4.2.2 Word-Aligned Hybrid (WAH) 27
4.2.3 WAH-derived Algorithms 28
4.2.4 GPU-based WAH Algorithms 31
4.2.5 Super Byte-aligned Hybrid (SBH) 33
4.3 Parallelizing SBH 38
4.3.1 CPU Parallelism 38
4.3.2 GPU Parallelism 39
4.4 Experimental Results 40
4.4.1 Plain Version 41
4.4.2 Parallelized Version 46
4.4.3 Summary 49
Chapter 5 Space-efficient Representation of Semi-structured Document Formats 50
5.1 Preliminaries 50
5.1.1 Semi-structured Document Formats 50
5.1.2 Resource Description Framework 57
5.1.3 Succinct Ordinal Tree Representations 60
5.1.4 String Compression Schemes 64
5.2 Representation 66
5.2.1 Bit String Indexed Array 67
5.2.2 Main Structure 68
5.2.3 Single Document as a Collection of Chunks 72
5.2.4 Supporting Queries 73
5.3 Experimental Results 75
5.3.1 Datasets 76
5.3.2 Construction Time 78
5.3.3 RAM Usage during Construction 80
5.3.4 Disk Usage and Serialization Time 83
5.3.5 Chunk Division 83
5.3.6 String Compression 88
5.3.7 Query Time 89
Chapter 6 Conclusion 94
Bibliography 96
μμ½ 109
Acknowledgements 111Docto
Intelligent Sensor Networks
In the last decade, wireless or wired sensor networks have attracted much attention. However, most designs target general sensor network issues including protocol stack (routing, MAC, etc.) and security issues. This book focuses on the close integration of sensing, networking, and smart signal processing via machine learning. Based on their world-class research, the authors present the fundamentals of intelligent sensor networks. They cover sensing and sampling, distributed signal processing, and intelligent signal learning. In addition, they present cutting-edge research results from leading experts
- β¦