1 research outputs found
A Survey on Sampling and Profiling over Big Data (Technical Report)
Due to the development of internet technology and computer science, data is
exploding at an exponential rate. Big data brings us new opportunities and
challenges. On the one hand, we can analyze and mine big data to discover
hidden information and get more potential value. On the other hand, the 5V
characteristic of big data, especially Volume which means large amount of data,
brings challenges to storage and processing. For some traditional data mining
algorithms, machine learning algorithms and data profiling tasks, it is very
difficult to handle such a large amount of data. The large amount of data is
highly demanding hardware resources and time consuming. Sampling methods can
effectively reduce the amount of data and help speed up data processing. Hence,
sampling technology has been widely studied and used in big data context, e.g.,
methods for determining sample size, combining sampling with big data
processing frameworks. Data profiling is the activity that finds metadata of
data set and has many use cases, e.g., performing data profiling tasks on
relational data, graph data, and time series data for anomaly detection and
data repair. However, data profiling is computationally expensive, especially
for large data sets. Therefore, this paper focuses on researching sampling and
profiling in big data context and investigates the application of sampling in
different categories of data profiling tasks. From the experimental results of
these studies, the results got from the sampled data are close to or even
exceed the results of the full amount of data. Therefore, sampling technology
plays an important role in the era of big data, and we also have reason to
believe that sampling technology will become an indispensable step in big data
processing in the future