Some processes are not easy to be programmed from scratch for parallel machines (clusters), but can be easily split on simple steps. Makefile::Parallel is a tool which lets users to specify how processes depend on each other. The language syntax resembles the well known Makefile makefiles format, but instead of specifying files or targets dependencies, Makefile::Parallel specifies processes (or jobs) dependencies. The scheduler submits jobs to the cluster scheduler (in our case, Rocks PBS) waiting them to end. When each process finishes, dependencies are calculated and direct dependent jobs are submitted. Makefile::Parallel language includes features to specify parametric rules, used to split and join processes dependencies. Some tasks can be split into n smaller jobs working on different portions of files. At the end, another process can be used to join the results.Partially supported by grant POSI/PLP/43931/2001 from Fundacao para a Cienciae Tecnologia (Portugal), co-financed by POSI

Almeida, J. J.

Fonseca, Rúben

Simões, Alberto

Universidade do Minho: RepositoriUM

Makefile::ParallelDependency Specification LanguageAlberto Simões??, Rúben Fonseca, and José João AlmeidaDepartamento de InformáticaUniversidade do Minho, Braga, Portugal,{ambs|rubenfonseca|jj}@di.uminho.ptAbstract. Some processes are not easy to be programmed from scratchfor parallel machines (clusters), but can be easily split on simple steps.Makefile::Parallel is a tool which lets users specify how processes dependon each other.The language syntax resembles the well known Makefile[1] format, butinstead of specifying files or targets dependencies, Makefile::Parallel spec-ifies processes (or jobs) dependencies.The scheduler reads the specification and submits jobs to the clusterscheduler (in our case, Rocks PBS) waiting them to end. When eachprocess finishes, dependencies are calculated and direct dependent jobsare submitted.Makefile::Parallel language includes features to specify parametric rules,used to split and join processes dependencies: some tasks can be splitinto smaller jobs working on different portions of files, and at the end,another process can be used to join results.1 IntroductionMore and more, researchers have access to multi-processors machines and, aswell, clusters. The problem is that most researchers do not have time to learnhow to program for parallel machines. Thus, they use their usual programs onclusters taking advantage just on the processor speeds and the big amount ofavailable memory.We propose a tool, Makefile::Parallel, to specify how small processes (or pro-grams) depend on each other, to create parallelism at the level of the program,instead of the usual parallelism at the instruction level. Makefile::Parallel maingoals are:– use a compact language to specify dependencies: in our main case study weare dealing with more than one hundred jobs. To specify their dependenciesmanually is time consuming and error prone;– reuse a well known language syntax that is being used in related tasks foryears: the Makefile syntax.?? Partially supported by grant POSI/PLP/43931/2001 from Fundação para a Ciênciae Tecnologia (Portugal), co-financed by POSI.– embed other languages to reuse their expressiveness. In our pmakefiles —the name we gave to the text specification files — we can define actions bothin Bash and Perl.– support parametric rules: in some situations we want to instantiate ruleswith different values, accordingly with results from a previous job. Thus,these rules need to be instantiated in runtime.– create information for profiling pmakefiles and create reports.Next section will present the Makefile::Parallel Domain Specific Language,showing the language grammar and explaining its versatility. Follows a sectionon the scheduler implementation and how it interacts with the main clusterscheduler. Before the final remarks, there is a section detailing our case study.2 Makefile::Parallel LanguageMakefile::Parallel language syntax is heavily inspired on Makefile’s syntax. Themain difference is that instead of defining dependencies between files or tar-gets, we define dependencies between jobs. The main idea can be seen as theformalization of a PERT[2]1 network.Figure 1 shows a simplified BNF version of the Makefile::Parallel grammar.The grammar was implemented in YAPP[3], a Perl version of the well knownyacc parser generator.To explain the language we will use the example shown on figure 2: supposewe have a program we want to test with a different parameter, an integer rangingfrom 3 to 10. We need to create a directory to save the data, and at the end wewant to remove some temporary files.To solve the problem presented above we defined three rules:– There is a job to prepare the output directory. This same rule defines avariable to be used on the next rule, a parametric one, named run (seesection 2.2 for detailed explanation on these rules). The variable is definedas a set of values that the variable p can have.– The parametric rule run$p will instantiate with values from 3 to 10, and runthe program being tested with different parameters.– Finally, the last rule cleanup depends on the run set of rules, and cleanupssome temporary files.Values between parenthesis indicate the expected walltime. This time is usedby ROCKS[4] to schedule the job in an adequate queue, and to kill the job incase it is taking too long.Following the walltime (a required parameter) there is an optional parameterin square brackets: the number of CPUs needed for the job. Because we are par-allelizing processes, does not mean we can not have processes that, themselves,are parallel, thus needing more than one processor.1 PERT (Program Evaluation and Review Technique) charts were first developed inthe 1950s by the Navy to help manage very large, complex projects with a highdegree of intertask dependency.jobs → job jobs→ jobjob → jobName ‘:’ deps walltime nrCpus actionswalltime → ‘(’ TIME ‘)’nrCpus → ‘[’ INT ‘]’→ jobName → ID→ ID VARdeps → jobName deps→ actions → action→ actions actionaction → shellCommand→ perlCommand→ setDefinitionshellCommand → TAB SHELLperlCommand → TAB ‘sub{’ PERL ‘}’setDefinition → TAB VAR ‘<-’ SHELL→ TAB VAR ‘<-’ ‘sub{’ PERL ‘}’Fig. 1. Makefile::Parallel DSL grammar.prepare: (5:00)mkdir OutputDatap <- sub{ print "$_\n" for (3..10) }run$p: prepare (20:00:00) [2]runMyProgram -p $p InputData > OutputData/run.$pcleanup: run$p (5:00)for a in @p; do rm -f OutputData/run.${a}.tmp; doneFig. 2. Simple Makefile::Parallel example.2.1 Support for both Shell and Perl actionsAlthough most processes are binary programs, sometimes we need some specificsmall actions to put files in some special places and to change some file contents.With that in mind we support Bash (as makefile) and, because we work especiallywith Perl (and the scheduler is implemented in Perl), we also support actionswritten in Perl.In the example above all actions are in Bash for simplicity.2.2 Support for parametric rulesIt is quite usual to test some specific programs with different algorithms ordifferent parameters. In other cases, we need to split a big job on small chunksto be processed independently.Both situations need the use of parametric rules: rules where variables getreplaced by a set of values. For instance, on the example above, the variable $pon the run rule gets replaced by the values of the p set. This set is defined onthe prepare rule with some Perl code. That Perl code returns a text file (in thestdout) with a value per line. These values are the p set elements.So, p gets the values from 3 to 10: p = {3, 4, 5, 6, 7, 8, 9, 10}. Then, the rulerun$p is replaced by:run3: prepare(20:00:00) [2]runMyProgram -p 3 InputData > OutputData/run.3run4: prepare(20:00:00) [2]runMyProgram -p 4 InputData > OutputData/run.4...run10: prepare(20:00:00) [2]runMyProgram -p 10 InputData > OutputData/run.4In the cleanup rule, something similar happens. The dependency list is ex-panded with p values, and the special list variable @p in the rule action is alsoexpanded to all values it can take. Note that @p is expanded correctly accord-ingly with its context (Bash or Perl). If we did not have parametric rules wewould need to write:cleanup: run3 run4 run5 run6 run7 run8 run9 run10 (5:00)for a in run3 run4 run5 run6 run7 run8 run9 run10; dorm -f OutputData/run.${a}.tmp; doneNote that the variable ${a} is a standard Bash variable that will be instantiatedduring run-time.3 Makefile::Parallel SchedulerThe scheduler — pmake— is written in the Perl language. It takes a specificationfile and schedules jobs accordingly with their dependencies.Since the beginning we had in mind to develop more than one schedulersubsystem to submit jobs. While most clusters use ROCKS, there are otherscheduler systems. Also, a simple SSH scheduler could be created. Thus, thecode was modularized: an abstract class to represent any scheduler, and a set ofsubclasses implementing all the methods for a specific system.do {launch_rules_with_satisfied_dependencies()terminated_processes = gather_terminated_processes()for( process in terminated_processes ) {if ( defines_parameters( process ) ) {parameters = calculate_parameters( process )expand_dependency_graph(parameters)}}save_journal()sleep(10)} while(! all_processes_executed())generate_profiling_information()print_report()Fig. 3. Scheduler engine behavior algorithm.3.1 Scheduler BehaviorThe basic scheduler behavior is specified on figure 3. The scheduler visits thedependency graph specification generated by the parser and, for every job withfulfilled dependencies, send them to run on the selected subsystem. The schedulerthen waits for any of the running jobs to die or to end, by asking their state tothe subsystem.When a process ends, the scheduler gets information from the subsystem(return code, CPU time, memory used). If the return code does not indicatefailure, all variables defined in the rule are instantiated (evaluating the Perl orBash definition), and the dependency graph is modified on-the-fly to reflect theinstantiated variables. Then, all processes not yet executed are browsed, and ifany is found with all dependencies fulfilled, it is submitted to the system. Theprocess continues all over again until no processes need to be executed.From time to time, the scheduler saves some part of his internal state topermanent storage (hard disk), in the form of a journal. This is useful in casethe specification is stopped by an error, or some kind of problem exists with thesubsystem (as power shortages). Later, the user can simply pass an option topmake, and the scheduler will bypass all processes ended correctly.Two subsystem implementations were developed for now: a PBS designed forclusters running mainly ROCKS, and a Local, designed for desktop processing.3.2 PBS SubsystemHigh-performance clusters are the computing tool of choice for a wide range ofscientific disciplines. Yet straightforward software installation, management, andmonitoring for large-scale clusters have been consistent and nagging problemsfor non-cluster experts. The free ROCKS cluster distribution takes a fresh per-spective on cluster installation and management to dramatically simplify versiontracking, cluster management and integration[4]. The toolkit centers around aLinux distribution based on the Red Hat Enterprise line, and includes work frommany popular cluster and grid specific projects.One utility found in any cluster toolkit is a Portable Batch System (PBS).Basically, scheduling software let the cluster run like a batch system, allowingthe allocation of cluster resources, such as CPU time and memory, on a job-by-job basis. Jobs are queued and run as resources become available, subject to thepriorities established[5]. PBS is a powerful and versatile system.When the scheduler emits a process to run on this subsystem, a PBS scriptis generated and sent to the cluster queue using the command qsub. To checkif a process is still running Makefile::Parallel uses the qstat program with theappropriate parameters.Eventually some processes finish. When the scheduler detects that (using theprevious call to this subsystem), it asks for additional information about thedead process using the output of the tracejob program. This program returnsdetailed information, including (real) CPU time, return code, and memory usageof the process.If the scheduler needs to stop a running process, the subsystem can call qdelto kill the process if it already running or to remove it from the queue otherwise.While ROCKS supports dependencies between jobs (you can specify jobsdependency when submitting a job) it is not versatile enough for most usersneeds.3.3 Local Scheduler SubsystemSince most of modern computers are becoming multiprocessor by nature, a localscheduler is a good way of exploring parallelization on small to medium work-flows.With this in mind, a local scheduler subsystem was implemented. Runningjobs on the development desktop machine allowed faster bug tracking and lesstime of coding. At the same time, one could not have access to a multicomputercluster so this subsystem can be used in a variety of situations.pmake local scheduler operation resembles the GNU make program on manyaspects. First of all, if invoked with no parameters, pmake takes the specificationand run it sequentially — the jobs are run as if a single pipeline exists. Thiscould be optimal on a desktop machine.However, as the level of multiprocessors rises the user may want to use the ad-ditional processor power available. For this reason, this subsystem was designedto accept a parameter that specifies how many parallel pipelines the schedulermust support on this execution.This subsystem uses the fork-exec-perror paradigm and trust the operatingsystem the correct map of the job to a free processor (both logical or physical).4 Case StudyWe have at our disposal a multicomputer cluster formed by approximately 140CPUs and 50 nodes. The cluster runs Linux and ROCKS. This means that wecan use the TORQUE Resource Manager to schedule our jobs.Although we are using Makefile::Parallel on two different research fields (BioInformatics and Natural Language Processing) we just present here the later,because it is the most interesting and was the real motivation for this work.The code shown on figure 4 is part of a bigger pmakefile, working in produc-tion for the word-alignment[6] of big parallel texts (bitexts: texts and respectivetranslations) and extraction of translation examples[7]. The word-alignment taskneeds to create big sparse matrixes in memory and for big texts (with more than300 MBytes of text files) this matrix does not fit on main memory. Thus the so-lution is to split the big text in smaller pieces, and process them independently.At the end the processing result is merged up.Figure 6 shows graphically this process. Important to note that this graphis generated by Graphviz[8] at run time with times for each task, and theirdependencies. In fact we are getting 25% to 15% of the time needed in sequentialmode in our tasks (this value highly depends on the current cluster load, asexpected).During process pmake will output progress information to a log file as shownin figure 5. This log shows the processes being launched and finishing, as well asthe time spent for each one. Together with the graph shown in figure 6, pmakegenerates also a file with timing information for easy profiling as can be seen onfigure 7.5 ConclusionsMakefile::Parallel proven to be a useful tool to specify jobs dependencies and toschedule them efficiently.Being able to perform a concise description makes it easier to define depen-dencies correctly than defining them by hand.codify: (20:00:00)nat-codify -id=EurLex EurLex-PT EurLex-ENi <- sub{ $nr = ‘cat EurLex/nat.cnf |grep nr-chunks|cut -f 2 -d "="‘;printf("%03d\n",$_) for (1..$nr); }initmat$i: codify (20:00:00)nat-initmat EurLex/source.$i.crp EurLex/target.$i.crp EurLex/mat.$i.inipfp$i: initmat$i (20:00:00)nat-ipfp 5 EurLex/source.$i.crp EurLex/target.$i.crp \EurLex/mat.$i.in EurLex/mat.$i.outrm -f EurLex/mat.$i.inpostipfp$i: ipfp$i (20:00:00)nat-mat2dic EurLex/mat.$i.out EurLex/dict.$irm -f EurLex/mat.$i.outpostbin$i: postipfp$i (20:00:00)nat-postbin EurLex/dict.$i \EurLex/source.$i.crp.partials EurLex/target.$i.crp.partials \EurLex/source.lex EurLex/target.lex \EurLex/source-target.$i.bin EurLex/target-source.$i.binrm -f EurLex/dict.$idicA: postbin$i (20:00:00)for a in @i; do \nat-dict add EurLex/source-target.bin EurLex/source-target.${a}.bin; \donefor a in @i; do rm -f EurLex/source-target.${a}.bin; donedicB: postbin$i (20:00:00)for a in @i; do \nat-dict add EurLex/target-source.bin EurLex/target-source.${a}.bin; \donefor a in @i; do rm -f EurLex/target-source.${a}.bin; donedump: dicA dicB (20:00:00)nat-dumpDicts -self EurLexFig. 4. Subset of NATools pmakefile.[...]2006/12/12 10:49:22 The job "ipfp005" is ready to run. Launching2006/12/12 10:49:22 Launched "ipfp005" (23996)2006/12/12 10:49:52 Process 23996 (ipfp005) has terminated [30s]2006/12/12 10:49:52 The job "postipfp005" is ready to run. Launching2006/12/12 10:49:52 Launched "postipfp005" (23997)2006/12/12 10:50:02 Process 23997 (postipfp005) has terminated [10s][...]Fig. 5. Makefile::Parallel log output.example1000006h 28m 13scleanExamples1m 2sipfp00219m 27spostipfp0024sexample2000006h 26m 51spostipfp0053m 12spostbin0051m 46spostipfp0013spostbin0012sngramsB2m 37sexamples57sexample1500006h 16m 40sexample05h 54m 49sexample500006h 7m 57sexample2500004h 7m 3spostipfp0035spostbin0032sipfp00117m 59sdicB1m 34sdump43sngramsA26mngramsA5m 45sngramsB411m 2sdicA1m 35spostipfp00415spostbin0042sngramsB25m 56sipfp00421m 34spostbin0022sinitmat0024m 36sfilter32sipfp00532sinitmat0044m 34sinitmat0058sngramsA411m 54sinitmat0032m 27sipfp00319m 28scodify9m 17sngramsA39m 23sinitmat0011m 39sngramsB37m 51sFig. 6. Process dependency graph.ID Start Time End Time Elapsedcodify 2006-12-12T10:41:10 2006-12-12T10:49:11 8m 1sngramsA 2006-12-12T10:49:11 2006-12-12T11:07:46 18m 34sngramsB 2006-12-12T10:49:11 2006-12-12T11:05:44 16m 33sinitmat001 2006-12-12T10:49:11 2006-12-12T10:50:12 1minitmat002 2006-12-12T10:49:11 2006-12-12T10:50:43 1m 31sinitmat003 2006-12-12T10:49:11 2006-12-12T10:51:03 1m 51s[...]Fig. 7. Makefile::Parallel report output.The syntax is versatile enough to be applied in more than one research area.The parser is quite simple, and the fact of supporting the embed of externallanguages makes it yet more powerful.Report facilities and graph generation are useful for week report generationabout cluster usage and project evolution. The generated graphs are useful toexplain how software works, how the different jobs interact together, and whatare the critical jobs (and paths).Worth saying thatMakefile::Parallel has almost no running cost since it spendsmost of the time sleeping, waiting for events to happen.References1. Campbell, D., Grevstad, C.: A tutorial for make. In: ACM’85: Proceedings of the1985 ACM annual conference on The range of computing : mid-80’s perspective,New York, NY, USA, ACM Press (1985) 374–3802. Douglas, D.E.: Pert and simulation. In: WSC ’78: Proceedings of the 10th conferenceon Winter simulation, Piscataway, NJ, USA, IEEE Press (1978) 89–983. Desarmenien, F.: Parse::Yapp — perl extension for generating and using lalr parsers.Perl module (2001) http://search.cpan.org/dist/Parse-Yapp/.4. Sacerdoti, F.D., Chandrai, S., Bhatia, K.: Grid systems deployment & manage-ment using rocks. IEEE International Conference on Cluster Computing, San Diego(September 2004)5. Sloan, J.D.: High Performance Linux Clusters with OSCAR, Rocks, OpenMosix,and MPI. O’Reilly (2004)6. Simões, A.M., Almeida, J.J.: Natools – a statistical word aligner workbench. SEPLN(Sep. 2003)7. Simões, A., Almeida, J.J.: Combinatory examples extraction for machine transla-tion. In Lønning, J.T., Oepen, S., eds.: 11th Annual Conference of the EuropeanAssociation for Machine Translation, Oslo, Norway (19–20, June 2006)8. Gansner, E.R., North, S.C.: An open graph visualization system and its applicationsto software engineering. Software — Practice and Experience 30(11) (2000) 1203–1233

Makefile : parallel dependency specification language

http://repositorium.sdum.uminho.pt/bitstream/1822/16530/1/MP07.pdf

Makefile : parallel dependency specification language

Abstract

Similar works

Full text

Available Versions

Universidade do Minho: RepositoriUM