Badia Sala, Rosa Maria

Lezzi, Daniele

Lordan Gomis, Francesc-Josep

UPCommons. Portal del coneixement obert de la UPC

Cyberinfrastructure programming with COMPSsFrancesc Lordan∗, Daniele Lezzi∗, Rosa M. Badia∗∗Barcelona Supercomputing Center (BSC)E-mail: {francesc.lordan, daniele.lezzi,rosa.m.badia}@bsc.esKeywords—Distributed Computing, Workflow Managers, Pro-gramming Models, Edge Computing, Internet of Things, CloudComputingI. EXTENDED ABSTRACTCyberinfrastructures are compositions of computing sys-tems, data repositories and storage systems, data collectioninstruments and visualization environments all linked togetherto improve research productivity and enable breakthroughs nototherwise possible.As for previous computing infrastructure transformations,developers had to adopt new execution paradigms to exploitthem. Distributed systems deprecated the traditional monolithicmodel in favor of service-oriented applications. To exploit theCloud and offer SaaS, developers embraced the microservicesmodel so that services could adjust the number of instancesof each microservice to the current workload. Bringing downthe computation from the Cloud to the Edge mitigates thenetwork issues - latency and bandwidth - and enables newopportunities. On Fog infrastructures, computing devices canjoin in or leave at their own will. For dealing with suchdynamicity, microservices became serverless and stateless.IoT devices have sensors that permanently produce data.These devices can monitor this data themselves and, when acertain condition is met, trigger a response that may requireheavy computation capabilities, or they can provide otherdevices with this information so they process it on a real-timebasis. To support the former scenario, developers can turn toFunction as a Service (FaaS): functions executed on the under-lying platform in a serverless, stateless approach. To supportthe latter, developers need to code using stream processingframeworks such as Kafka streams or Spark Streams.A. The COMPSs Programming ModelCOMPSs [1], [2] is a framework that aims to ease thedevelopment of distributed applications. Its core it is a task-based programming model [3], [4] and a runtime toolkitexecuted along with the application which automatically de-tects its parallelism and orchestrates the execution of its taskson the available computing nodes. Thus, it is able to exploitdistributed, heterogeneous and highly-dynamic infrastructureswhile keeping the code totally unaware of the infrastructureand parallelism details.To develop a distributed application with COMPSs, pro-grammers code the logic of the application in a sequential,infrastructure-unaware fashion with no API invocations as ifthe code was to be run on a single-core computer. For theruntime system to detect the tasks composing the application,developers select a set of methods whose invocations createnew asynchronous tasks by annotating them with the @tasktag. In order to guarantee the sequential consistency of thecode, the runtime system monitors the data values accessed byeach tasks to find dependencies among them. So the runtimecan better exploit the application parallelism, developers needto describe how the method operates (reads, generates orupdates) on each data value by indicating its directionality (IN,OUT, INOUT, respectively).The code snippet in Figure 1 shows a sample applicationwith three tasks: two of them (generateReport) process theirrespective input files generating two Report objects that aremerged in a third task – merge report.@task(file=FILE IN, returns=Report)def generate report(file):...return report;@task(r1=INOUT, r2=IN)def merge report(r1, r2):...def main(files):final report = Nonefor file in files:partial report = generateReport(file)if final report:merge report(final report, partial report)else:final report = partial reportFig. 1. Sample COMPSs application using PythonB. Envisaged Computing PatternsIn cyberinfrastructure environments, authors envisage threepossibilities that require computation. On the first one, knownas sense-process-actuate, the infrastructure is supposed to givea proper response to an event detected on one of its sensors;for instance, turning on the AC system when temperature in aroom reaches more than 27 degrees. A second approach thatmight require computation is stream processing. Sensors maycontinuosly produce data that need to be processed in real-time to produce a response or update the data visualizationcomponents. For instance, a camera could video-stream theinside of the room, and the AC system could be turned whenthe processing of such stream identifies typical reactions toheat. Finally, the third scenario that might require computationis batch processing. Cyberinfrastructures collect, generate andstore big amounts of data. All this information can be analyzedto produce new knowledge; for instance, machine learningtechniques could be applied to improve the models detectingpeople reactions and change the behavior of the infrastructure.C. Runtime System InfrastructureUnlike previous environments with a single entry point tostart an application, on cyberinfrastructures any device can6th BSC Severo Ochoa Doctoral Symposium48detect an event and trigger a computation. Thus, the eachdevice must be able to work in a standalone manner and runthe computation in an efficient way: detecting the parallelisminherent in the computation and run the detected tasks onthe computing elements embedded on the device. For doingso, each node of the infrastructure will host the executionof a COMPSs agent, a deamon process waiting for executionrequests. When a the node triggers a new execution, it contactsthis agent which runs the corresponding code detecting thetasks composing it and orchestrates their execution on theavailable computing resources.On the one hand, the computing resources locally em-bedded on the same device might not be enough to host theexpected computation. On the other hand, not all the devicesof the infrastructure will trigger executions; indeed, even thosethat raise events may not compute anything temporarily. Thus,if COMPSs agents were able to only exploit the local proces-sors, most of the computing power would remain idle. In orderto better exploit the computing power of the infrastructureand shorten the execution time of the application, COMPSsagents must interact with each other and offload part of theircomputation. For that purpose, one COMPSs agent can callthe same method that started a computation on another agentto execute a task in the remote agent. This agent will run thetask code, analyze it detecting new tasks on the task code andorchestrate the execution of these tasks on other agents of theinfrastructure.For infrastructures with a small number of devices, aCOMPSs agent can individually manage the execution of thetasks it has detected on any of the other COMPSs agentscomposing the infrastructure. However, the bigger the in-frastructure is, the higher the complexity of the schedulingproblem becomes. Assigning tasks to resources may becomea computational load bigger than the actual computation toperform, or for very large infrastructures, it may simply not fitin resource-scarce devices.To solve that problem, we propose to organize all the agentsof the infrastructure in a hierarchic way. The topology of suchorganization can be represented as an acyclic graph where eachnode of the graph represents an agent of the infrastructure,and edges illustrate the possibility of one agent to submittasks onto the agent represented by the other end of the edge.Thus, one agent can decide to run tasks on its local devicesor offload them onto any of the other agents with whom itis directly connected on the hierarchy. In turn, the receivingagent could decide to run the task on its local computingdevices or onto any of its direct neighbors. Thus, a computationtriggered by any of the agents could use as many resourcesof the infrastructure as the computation needs to ensure thatthe application runs and it does so in the shortest executiontime possible without adding a significant overhead due to thescheduling.An important aspect to bear in mind when dealing withcyberinfrastructures is the potential mobility of its components.Smartphones, tablets and more complex devices attached to abattery with wireless network connectivity may be part of thesystem. These kind of devices may join in the infrastructure orleave it at their own will. When a device joins in, it is includedin the system topology to add its resources into the pool soother agents can use them and, on the other hand, the mobiledevice can exploit the rest of the infrastructure to enhanceits performance. When a device gets disconnected from theinfrastructure, a communication link is broken and an edge ofthe graph is removed. At that point, the agents at both ends ofthe edge need to re-schedule the execution of the tasks alreadyoffloaded onto the other agent and assign them to resources towhich it remains connected. Thus, any computation alreadystarted can go on and finish its execution.Good criteria to build the topology are the stability andlatency of the network. Establishing the topology according tothat ensures that agents will always try to offload tasks ontonearby resources, on the fog, rather than submitting them tothe Cloud achieving a higher performance. Besides, in the caseof network disruption, the system could remain usable evenwithout the cloud.II. ACKNOWLEDGMENTThis work is supported by the Horizon 2020 research andinnovation programme under grant 730929 (mF2C Project).Salis, A., Mancini, G., Bulla, R., Cocco, P., Lezzi, D. andLordan, F. (2019) Benefits of a Fog-to-Cloud Approach inProximity Marketing. In: Mencagli G. et al. (eds) Euro-Par2018: Parallel Processing Workshops.Lordan, F. and Badia, R.M. (2017) COMPSs-Mobile: Par-allel Programming for Mobile Cloud Computing. In Journalof Grid Computing, vol 15, 357.REFERENCES[1] R. M. Badia and et al., “COMP superscalar, an interoperableprogramming framework,” SoftwareX, vol. 3, pp. 32–36, 12 2015.[Online]. Available: http://dx.doi.org/10.1016/j.softx.2015.10.004[2] Workflows and Distributed Computing - Barcelona SupercomputingCenter (BSC). (2018) COMP Superscalar. [Online]. Available:http://compss.bsc.es[3] F. Lordan et al., “ServiceSs: an interoperable programming frameworkfor the Cloud,” Journal of Grid Computing, vol. 12, no. 1, pp. 67–91, 32014. [Online]. Available: https://digital.csic.es/handle/10261/132141[4] E. Tejedor et al., “PyCOMPSs: Parallel computational workflows inPython,” The International Journal of High Performance ComputingApplications (IJHPCA), vol. 31, pp. 66–82, 2017. [Online]. Available:http://dx.doi.org/10.1177/1094342015594678Francesc Lordan obtained the Ph.D. in ComputerArchitecture from the Universitat Politecnica deCatalunya in 2018 after defending his Ph.D. thesis:”Programming Models for Mobile Environments”.Since 2010, Francesc is part of the Workflows andDistributed Computing group of the Barcelona Su-percomputing Center. His efforts have focused on theCOMPSs programming model: a task-based modelfor developing parallel applications running on largedistributed infrastructures such as clusters, super-computers, grids and clouds. Francesc has publishedmore than 20 articles in International conferences and journals, and he has beendirectly involved in the European projects mF2C, ASCETIC and OPTIMIS.He has also provided support to other collaborative projects such as Venus-C,EU-Brasil OpenBio, Transplant and the Human Brain Project. His researchfocuses on programming models that aim to ease the development of parallelapplications by hiding the technical concerns of heterogeneous and distributedinfrastructures.6th BSC Severo Ochoa Doctoral Symposium49

Cyberinfrastructure programming with COMPSs

UPCommons

https://upcommons.upc.edu/bitstream/2117/167719/1/20_Cyberinfrastructure_programming.pdf

Cyberinfrastructure programming with COMPSs

Abstract

Similar works

Full text

Available Versions

UPCommons. Portal del coneixement obert de la UPC

UPCommons