Wang, Liqiang

English

Clemson University: TigerPrints

A Semantics-Aware Optimization Framework for Data-Intensive Applications Using Hybrid Program AnalysisLiqiang WangDepartment of Computer ScienceUniversity of Central FloridaApps Development & Challenges Challenge 1: Multiple execution  plans with potential different performance due to the freedom of programming modelChallenge 2: Limited knowledge about system status (e.g. container configuration)Challenge 4: Performance is sensitive to many runtime factors and data placement.Challenge 3: What information is useful for further performance optimization?DesignCode  GenrationDeploymentOperationLog Data CollectionConfigurationSelectionAdaptationIDEData-IntensiveApplicationBig Data ServiceRuntimeAnalysisToolsData-IntensivePlatformStatic Code AnalysisDynamic Info ProfilingOur approach◼ A two-stage framework, offline and online stages, to assist programmers to design and optimize data-intensive applications semi-automatically.Offline PhaseCodeAnalyzerDOGConsole WarningsrcUserLogAnalyzerOptimizerElement PruningOperationMigrationOnline PhaseConfigGeneratorPerformanceLogProfiling GuidanceInput DataOutputRuntimeProfilerApplicationSystem Event LogCacheManagementMetrics System ReviseCodeThe full life cycle of semantics-aware optimization approach for data-intensive applicationsSemantics-Aware Data Model for Spark◼ An abstract data model associated with semantic context regarding code, data and system to represent skeleton of an application and track evolution of dataset(s) Data Representation: Attribute-Based Data Abstraction Data Manipulation: Predefined Primitive Operations Application Representation: Data Operational Graph (DOG)◼ Operation Strategies Element Pruning (remove redundant attributes in an element) Operation Reordering (Filter pushdown) Cache Management (persist a data block in memory)Primitive Operations◼ Define six primitive operations to abstract behaviors of a general data-intensive system, as shown in following TableData Operational Graph (DOG)◼ A directed graph G = (V, E)  V: Data manipulated operations and the corresponding generated datasets E: Data flows between operations Semantics knowledge regarding code, data and system is attached to verticesand edgesElement Pruning◼ It is a static optimization to eliminate unused attributes in an element by analyzing data dependency in the attribute level among operations. Analyzing attribute dependency between the input and output dataset of an operation and its UDF Building a directed data dependency graph (DDG) to represent the whole data flow of the application Removing nodes that does not makecontributions to output of the applicationSourceattr_0 attr_1 attr_2 attr_3 attr_nattr_0 attr_2 attr_3attr_0 [attr_2] [attr_3]attr_0 [attr_2].sumreviewRDDmapgroupByKeymapSinkOperation Reordering◼ Improve applications' performance by reordering operations along with data path, e.g. Filter Pushdown.  Statically ensuring identical semantics Evaluating performance improvement using dynamic/profiling information◼ Using performance models to predict and evaluate performance behavior after reordering. Cache Management◼ Spark RDD cache/release◼ Cache management determines a policy on the stage level to balance the gain  and overhead of caching each RDD. A very complex problem We designed an approach based on convex optimization◼ Dataset size affects memory capacity and perf behaviors. ◼ Executing order affects the performanceExperiment and Conclusion◼ Our semantic-aware optimization approach including two stages: Static stage (Offline): code operation Dynamic stage (online): customized profiling

A Semantics-Aware Optimization Framework for Data-Intensive Applications Using Hybrid Program Analysis (Presentation)

https://tigerprints.clemson.edu/cgi/viewcontent.cgi?article=1052&amp;context=hugedata

A Semantics-Aware Optimization Framework for Data-Intensive Applications Using Hybrid Program Analysis (Presentation)

Abstract

Similar works

Full text

Available Versions

Clemson University: TigerPrints