Badia Sala, Rosa Maria

Ejarque, Jorge

Ramon-Cortes, Cristian

UPCommons

Orchestration of software packages in data science workflows

UPCommons. Portal del coneixement obert de la UPC

Orchestration of Software Packages in Data ScienceWorkflowsCristian Ramon-Cortes∗, Jorge Ejarque∗, Rosa M. Badia∗∗Barcelona Supercomputing Center (BSC)E-mail: {cristian.ramoncortes, jorge.ejarque, rosa.m.badia}@bsc.esKeywords—Distributed Computing, Workflow Managers, Pro-gramming Models, Data Science pipelines, Dust ForecastI. EXTENDED ABSTRACTNowadays, Data Science applications are complex work-flows composed of binary executions, MPI simulations, multi-threaded applications, and user-defined analysis (possibly writ-ten in Java, Python, C/C++, or R). Our proposal integratesthis heterogeneity into a single task-based programming modelcapable of orchestrating the execution of the different frame-works in a transparent way and without modifying nor itsbehaviour, nor its syntax. Thus, our prototype is designed fornon-expert users that want to build complex workflows wheresome steps require a highly optimised state of the art softwarepackages.A. Integration with Software PackagesTo integrate the execution of other frameworks transpar-ently into a single programming model, our prototype extendsthe COMPSs framework [1], [2] to act as an orchestratorrather than a regular application executor. Next, we detailthe modifications of the programming model annotations,although the Runtime master, and worker executors have alsobeen modified to schedule multi-node tasks and executing thedifferent software packages.The COMPSs Programming Model [3], [4] defines anno-tations that must be added to the sequential code in orderto run the applications in parallel. These annotations can besplit into Method Annotations and Parameter Annotations. Ourprototype extends the programming model by providing anew set of Method Annotations and Parameter annotations tosupport the execution of binaries, multi-threaded applications(OmpSs [5]), MPI simulations, nested COMPSs applications,and multi-node tasks inside a workflow. From now on, tasksthat must execute software packages are called Non-NativeTasks.On the one hand, a new Method Annotation is definedfor each supported non-native task. Notice that the MethodAnnotation must contain framework related parameters and,thus, its content varies depending on the target framework.Next, we list the currently supported frameworks and theirspecific parameters.• Binaries: Execution of regular binaries (e.g., BASH,SH, C, C++, FORTRAN)◦ Binary: Binary name or path to an executable◦ Working Directory: Working directory for thefinal binary execution• OmpSs: Execution of OmpSs binaries◦ Binary: Path to the execution binary◦ Working Directory: Working directory for thefinal binary execution• MPI: Execution of MPI binaries◦ Binary: Path to the execution binary◦ MPI Runner: Path to the MPI command to run◦ Computing Nodes: Number of required com-puting nodes◦ Working Directory: Working directory for thefinal binary execution• COMPSs: Execution of nested COMPSs workflows◦ Application Name◦ Runcompss: Path to the runcompss command◦ Flags: Extra flags for the nested runcompsscommand◦ Computing Nodes: Number of required com-puting nodes◦ Working Directory: Working directory for thenested COMPSs application• Multi-node: Execution of native Java/Python tasksthat require more than one node◦ Computing Nodes: Number of required com-puting nodesOn the other hand, in order to input and output data fromthe execution of non-native tasks, the Parameter Annotationalso needs to be enhanced. When using multi-node tasks theparameters and the return value of the task are the same thanwhen using a regular method task. However, when execut-ing standalone binaries, OmpSs processes, MPI processes, orCOMPSs applications the exit value of the processes is used asthe return value. Thus, we have decided that the COMPSs non-native tasks must use the exit value of their internal binary asthe return value of the task. In this sense, our prototype allowsthe users to capture this value by defining the return type ofthe non-native task as an int (for implicit synchronisation),as an Integer (for post-access synchronisation) or to forget it(declaring the function as void).However, the users do not only need the process exitvalue to work with this kind of applications but need to setthe Standard Input (stdIn) and capture the Standard Output(stdOut) and Error (stdErr). For this purpose, our prototypeincludes a new Parameter Annotation, stream, that allows theusers to set some parameters as I/O streams for the non-native6th BSC Severo Ochoa Doctoral Symposium64tasks. Stream parameters are not passed directly to the binarycommand but rather they are set as stdIn, stdOut or stdErr ofthe binary process. Since this kind of redirection is restricted tofiles in LINUX Operating Systems, we have decided to keep thesame restrictions to the annotation. Consequently, all streamparameters must be files.B. Evaluation1) NMMB-MONARCH: The NMMB-MONARCH is afully online multiscale chemical weather prediction system forregional and global-scale applications. The system is based onthe meteorological Nonhydrostatic Multiscale Model on the B-grid [6], developed and widely verified at the National Centersfor Environmental Prediction (NCEP). The model couplesonline the NMMB with the gas-phase and aerosol continuityequations to solve the atmospheric chemistry processes indetail. It is also designed to account for the feedbacks amonggases, aerosol particles, and meteorology.The NMMB-MONARCH workflow is composed by fivemain steps, namely Initialization, Fixed, Variable, UMOModel, and Postprocess. Although all the phases spawn severalbinaries, the NEMS binary soars above the rest. It is calledduring UMO Model step and its a Fortran 90 applicationparallelised with MPI. Therefore, NEMS can be executedin multiple cores and multiple nodes, relying on the MPIparadigm. The model uses the Earth System Modelling Frame-work library as the main framework [7].2) Parallelisation design: The original NMMB-MONARCH application consists in a BASH script definingthe main workflow and a set of Fortran binaries. Since ourprototype supports Java and Python workflows, we haveported two different versions of the NMMB-MONARCH.Both of them parallelise the main workflow while keepingthe Fortran binaries.Regarding the parallelisation, all the binaries have beenconsidered as tasks. For instance, Figure 1 shows the annota-tion of the deeptemperature binary in Python. We do notshow the equivalent Java version due to space constraints.@binary(binary='/path/to/deeptemperature.x')@task(returns=int,seamask=FILE_IN,deep_temperature=FILE_OUT)def deeptemperature():passFig. 1. Annotation of the deeptemperature binary in PythonAs previously explained, the NEMS simulator invokedwithin the UMO Model step is implemented using MPI. Thus,in contrast to the rest of binaries, we have annotated it asan MPI task. Figure 2 shows the annotation of the NEMSbinary in Python. Notice that, considering that the NEMSexecution can be performed with a different number of nodes,the constraint decorator attached to the MPI task may varybetween executions. Hence, to ease this management to users,the constraint has been defined using an environment variable.To conclude, the final workflow structure for both the Javaand the Python versions is depicted in Figure 3. Readers canidentify the Fixed, Variable, UMO Model, Postprocess andFigures and animations creation steps, where the Variable,UMO Model and Postprocess are executed thrice.@constraint(computingUnits='$NEMS_CUS_PER_NODE')@mpi(runner='mpirun',binary='/path/to/NEMS.x',workingDir='/path/to/nems/out',computingNodes='$NEMS_NODES')@task(returns=int,stdOutFile=FILE_OUT_STDOUT,stdErrFile=FILE_OUT_STDERR)def nems():passFig. 2. Annotation of the nems MPI binary in PythonFig. 3. Dependency graph of three days simulation3) Computing infrastructure: The infrastructure used inthis comparison is the Nord 3 cluster (a subset of the MareNos-trum III Supercomputer [8]), located at the Barcelona Super-computing Center (BSC). This supercomputer is composed of84 nodes, each with two Intel SandyBridge-EP E5-2670 (8cores at 2.6 GHz with 20 MB cache each), a main memoryof 128 GB, FDR-10 Infiniband and Gigabit Ethernet networkinterconnections, and 1.9 PB of disk storage.The Java version used is 1.8.0 u112 64 bits, the Pythonversion used is 2.7.13, and the MPI used is OpenMPI 1.8.1.4) Performance: Table I compares each step of the previousversion (BASH) against the Java and Python ones using 4worker processes (64 cores).Step Execution Time (s) Speed-up (u)BASH Java Python Java PythonFixed 290 117 119 2.48 2.43Variable 26 19 22 1.37 1.18Model Simulation 244 242 233 1.01 1.04Post process 38 34 33 1.12 1.15Total 601 413 415 1.45 1.45TABLE I. PERFORMANCE PER STEP WITH 4 WORKERS (64 CORES)Notice that both Java and Python versions improve the per-formance in the Fixed and Variable steps due to the possibilityof performing multiple tasks at the same time during thesesteps. In opposition, the Model Simulation and Post Processdo not improve because they are composed, respectively, bya single task, and two tasks with a sequential dependency.However, the performance of these steps does not degradeeither.6th BSC Severo Ochoa Doctoral Symposium65Fig. 4. Paraver trace of the Python version using 4 workers (64 cores)Figure 4 shows Paraver [9] traces of a three-day simulationwith 4 workers (64 cores) of the Python workflow. There arethree timelines where each row corresponds to a thread inthe worker. The top view shows a task view, where differentcoloured segments represent tasks, and each colour identifiesa different task type. The middle and bottom views are theinternals of the MPI NEMS task, where the yellow linesrepresent communications between MPI ranks.The trace detail in the middle and bottom views show,respectively, the MPI events and the MPI communicationsof the Model simulation task. Notice that, during this phase,both frameworks, COMPSs and MPI, are working togetherand sharing the computing nodes. Although in this case MPIis using the four nodes, other applications may reserve somenodes for MPI and use the others to compute remainingsequential PyCOMPSs tasks.C. Conclusions and Future WorkOur prototype enables non-expert users to develop complexData Science workflows where some steps require a highlyoptimised state of the art software package. It provides a sin-gle task-based framework with a homogeneous annotation toorchestrate the execution of native Java and Python tasks alongwith binaries, MPI simulations, multi-threaded applications,and nested applications. From the users’ point of view, theannotation is transparent and does not require to modify nor thebehaviour, nor the syntax of the underlying software packages.Moreover, the evaluation demonstrates that our prototypeeases the development of complex workflows such as NMMB-MONARCH. We have developed two new implementationsin Java and Python that provide better configuration manage-ment and object-oriented structure and improve the debugging,maintenance, and extension of the code. Also, the performanceanalysis demonstrates that our prototype is capable of extract-ing the parallelism of the NMMB-MONARCH applicationachieving 1.45 overall speed-up and sharing resources withmulti-node frameworks such as MPI.II. ACKNOWLEDGMENTThis work has been published in proceedings of the2018 IEEE 14th International Conference on e-Science (e-Science) [10]. Cristian Ramon-Cortes predoctoral contract isfinanced by the Ministry of Economy and Competitivenessunder the contract BES-2016-076791.REFERENCES[1] R. M. Badia and et al., “COMP superscalar, an interoperableprogramming framework,” SoftwareX, vol. 3, pp. 32–36, 12 2015.[Online]. Available: http://dx.doi.org/10.1016/j.softx.2015.10.004[2] Workflows and Distributed Computing - Barcelona SupercomputingCenter (BSC). (2018) COMP Superscalar. [Online]. Available:http://compss.bsc.es[3] F. Lordan et al., “ServiceSs: an interoperable programming frameworkfor the Cloud,” Journal of Grid Computing, vol. 12, no. 1, pp. 67–91,3 2014. [Online]. Available: https://digital.csic.es/handle/10261/132141[4] E. Tejedor et al., “PyCOMPSs: Parallel computational workflows inPython,” The International Journal of High Performance ComputingApplications (IJHPCA), vol. 31, pp. 66–82, 2017. [Online]. Available:http://dx.doi.org/10.1177/1094342015594678[5] A. Duran et al., “Ompss: a proposal for programming heterogeneousmulti-core architectures,” Parallel Processing Letters, vol. 21, no. 02,pp. 173–193, 2011.[6] Z. Janjic and R. Gall, “Scientific Documentation of the NCEP Non-hydrostatic Multiscale Model on the B Grid (NMMB). Part 1 Dynam-ics, Technical Report,” NCEP, BOULDER, COLORADO, Tech. Rep.80307-3000, 4 2012.[7] C. Hill et al., “The architecture of the Earth System Modeling Frame-work,” Computing in Science Engineering, vol. 6, no. 1, pp. 18–28, 12004.[8] Barcelona Supercomputing Center (BSC). (2017) MareNostrum 3.[Online]. Available: https://www.bsc.es/marenostrum/marenostrum/mn3[9] ——. (2017) Paraver Tool. [Online]. Available:https://tools.bsc.es/paraver[10] J. Conejero et al., “Boosting Atmospheric Dust Forecast with Py-COMPSs,” in 2018 IEEE 14th International Conference on e-Science(e-Science), Oct 2018, pp. 464–474.Cristian Ramon-Cortes is a PhD Student forthe Computer Architecture Department (DAC -UPC) working in collaboration with the Workflowsand Distributed Computing group (WDC) at theBarcelona Supercomputing Center (BSC). He holdsa MSc. on Research and Innovation in Informatics -High Performance Computing (MIRI - HPC, 2017),an Engineering Degree on Computer Science (FIB,2014), an Engineering Degree on Industrial Engi-neering (ETSEIB, 2014), and a Dual BSc. Diploma(CFIS, 2014); all of them from the Technical Uni-versity of Catalonia (UPC). During his career at the BSC he has contributed inthe design and development of COMPSs, PyCOMPSs, and PMES. His areasof interest include Programming Models for Distributed Platform, WorkflowManagers, Task Flows, Data Flows, and Streaming Technologies.6th BSC Severo Ochoa Doctoral Symposium66

https://upcommons.upc.edu/bitstream/2117/167736/1/27_Orchestration_of_software_packages.pdf

Orchestration of software packages in data science workflows

Abstract

Similar works

Full text

Available Versions

UPCommons

UPCommons. Portal del coneixement obert de la UPC