Golasowski, Martin

Ptošek, Vít

Slaninová, Kateřina

Vojtek, David

Ševčík, Jiří

English

DSpace at VSB Technical University of Ostrava

SERVER-SIDE NAVIGATION SERVICE BENCHMARKINGTOOLMartin Golasowski, Katerˇina Slaninova´, Jiˇr´ı Sˇevcˇ´ık, Vı´t Ptosˇek, David VojtekIT4InnovationsVSˇB - Technical University of Ostrava, Ostrava, Czech Republicmartin.golasowski@vsb.cz, katerina.slaninova@vsb.cz, jiri.sevcik@vsb.cz,vit.ptosek@vsb.cz, david.vojtek@vsb.czAbstract: Convenient access to a complex benchmarking and monitoring tool was the mainmotivation for implementation of the proposed benchmarking tool. This paper provides abrief summary of its design and implementation. Its main purpose is to test an experimentalserver-side navigation service which has to comply to a multi-criteria service level agreement.The service is deployed on a heterogenous high performance computing infrastructure whichis monitored by the tool. Consistent test environment is provided by the tool for transparentanalysis and optimization of the service performance.Keywords: navigation, routing, benchmark, high performance computing1 IntroductionThere is a significant demand for new forms of vehicle routing algorithms. For example, Prob-abilistic time-dependent vehicle routing problem which could use precise route computation asan input was proposed in [1]. In the context of smart city, large number of routing requestscan be issued in a very short time while there can be periods with very low or nonexistenttraffic. Service which provides such functionality has to be flexible enough in order to satisfythe dynamic load while maintaining a given service level agreement (SLA).These services are usually deployed on a heterogenous architecture consisting for exampleof both virtual machines and high performance computing (HPC) nodes. The SLA thereforeconsists of multiple criteria, such as response time or resources budget. These criteria canchange over time. Maintaining the SLA over time is possible by using technologies from theANTAREX project [10], such as the autotuning framework and a domain specific language.In order to design and develop such a service a proper benchmarking and monitoring tool hasto be provided. The goal of the tool is to provide users a way how to test the mentioned multi-criteria SLA against a given set of parameters of the service and its underlying technologies.The tool should provide a consistent and robust benchmarking environment which should beused in a transparent way for optimizing the service. Simultaneously, the tested service isprovided as a black box and the tool only exposes only a subset of the service parameterswhich are relevant for testing. The tool can be used to obtain unified presentation of the testresults, which is convenient for comparison of different parameter settings of the service.The service provides navigation on a graph representation of the road network. It finds apath between a pair of two nodes or between a set of waypoints according to a selected criteria(speed, road/vehicle type, etc.). The performance metric of the service in our case is requestthroughput per second and average time per request. The input data set consists of a set oforigin/destination pairs and parameters of the routing pipeline (PTDR, sample count, routingalgorithm type, etc.). The service implements a subset of the HTTP communication protocoland uses simple JSON-based data structures.1.1 Related workThere is an increased demand for architectures following the Anything as a Service (XaaS)paradigm [4]. Even though there are many robust and complicated protocols for providingsuch services; APIs based on top of the HTTP protocol are quite popular these days.DSpace VŠB - TUO           http://hdl.handle.net/10084/133448                      December 2018This is the post peer-review accepted manuscript of: GOLASOWSKI, Martin, Kate?ina SLANINOVÁ, Ji?í ŠEV?ÍK, Vít PTOŠEK a David VOJTEK. Server-side navigation service benchmarking tool. Slovenian Society Informatika, 2017. The published version is available online at: http://fgg-web.fgg.uni-lj.si/~/sdrobne/sor/SOR'17%20-%20Proceedings.pdf © 2017 Lidija Zadnik Stirn – Mirjana Kljaji? Borštnar – Janez Žerovnik – Samo Drobne Any service provided over a computer network needs to be properly tested in order todetermine possible performance bottlenecks in the architecture or possible errors in its output.Obtaining an energy efficient operating mode of the service is also a significant factor in testing,especially in big data centres [2, 3]. Demand for energy efficient adaption of this paradigm onpowerful computing nodes is getting more significant with the overlap of cloud and traditionalhigh-performance computing.There are many benchmarking tools for HTTP-based services implemented for variousplatforms. They are usually implemented as a stand-alone terminal applications [5, 6]. Inputspecification is often realised by a template of the request or by a set of pre-defined requeststo be sent. Some of the benchmarks provide more comprehensive statistical evaluation of thetest, however, the output is often in plain text form, which is harder to parse and prone toerrors. On the other hand, there are comprehensive tools, such as Locust [7], which providesall-in-one solution together with the analysis and visualisation front-end and result database.Such tools offer many features, which are unnecessary for our use-case and can be hard toalter and optimise. Therefore, we propose a custom light-weight solution which is designed forservices running on heterogenous HPC infrastructure.Next Section 2 is focused the benchmarking process including an example of the server-side navigation service benchmark used within the ANTAREX project [10]. The benchmarkimplementation is described in Section 3 and the asynchronous back-end which provides anexecution and management of large number of small tasks follows in Section 3.2. Section 4then concludes the paper.2 Benchmarking ProcessBenchmarking can be described as a process in which performance of a system is evaluated ona set of its parameters and input data. Performance of the system is then evaluated by relevantset of metrics (throughput, load, energy consumption, etc.). Subject of the testing is a relationbetween the parameter values and the selected metrics. Tested parameters can be for exampleI/O strategy, memory access pattern or various source code optimizations. The system is testedagainst a set of the parameter values while input data and run-time environment remain static.Crucial property of a good benchmark is consistency. Individual tests share the sameconditions through the entire test. It is often hard to assure completely consistent conditionsfor all the tests in real-world scenarios (for example in multi-task operating systems), thereforeinfluence of the external factors should be kept to a sane minimum. For example, unnecessaryservices should be turned off as well as scheduling of maintenance tasks (backup, logging, etc.),in order to minimise performance fluctuation of the target hardware platform.2.1 Server-side Navigation BenchmarkIn our case, the subject for benchmarking is the server-side navigation service. Core of theservice is the navigation pipeline which is deployed on a HPC cluster. This design allows flexibleoptimization of the pipeline according to various criteria to correspond with the current SLA.Parameters of the benchmark are divided into three parts.1. Data sets - type, size2. Static routing - routing algorithm, route type3. Probabilistic time-dependent routing - enabled, number of samples, autotuningstrategyThe data set of origin-destination point pairs is sent to the service from a large number ofconcurrent processes. This approach allows us to simulate a large number of concurrent usersDSpace VŠB - TUO           http://hdl.handle.net/10084/133448                      December 2018Figure 1: HPC Routing Benchmark Datasetof the service. The service then logs each individual requests. A benchmark is defined by thetime-frame between sending the first pair of the input data to handling reception of a responsefor the last pair of the input data. The metrics are computed from the service log within thetime-frame. Total amount of the requests sent is controlled by initial size of the selected dataset and its size parameter, which can be used to artificially inflate the number of sent requestsand to create specialized use case (such as smart city).2.2 Data SetsThe points in current version of the benchmark are selected from the region of the CzechRepublic, see Figure 1. The region is divided to 77 units according to the LAU1 (LocalAdministrative Units Level 1) division, which corresponds to individual districts of the CzechRepublic. Each district has its administrative center which usually corresponds to its biggestcity. There are 72 district towns as some of districts share one district town. There are 10randomly selected points, that correspond to nodes of a routing graph, inside each of the 72district cities. The resulting point data set is stored in a PostgreSQL database with a spatialextension PostGIS as a basic data set. Source data for the basic dataset are obtained from theOpen Street Map project (OSM)[11].Since the source data for the routing graph are updated regularly, the IDs of the graph nodes(the selected points of the basic dataset), can change. Therefore, it is practical to generatederived data sets only by a database query stored as a procedure or materialized view. Thedata sets are versioned according to the current processed OSM data.In the current version of the benchmark, there are three data sets, defined by their size andtype of the location:DSpace VŠB - TUO           http://hdl.handle.net/10084/133448                      December 2018Figure 2: Overview of the application architecture• Smart city - points within single LAU (city, 90 points)• Transit - points between three main cities (Praha, Brno, Ostrava, 300 points)• All - all 72 LAU1 regions (250 000 points)3 Benchmark ApplicationThe application design is roughly divided into several parts. The monitoring and benchmark-ing front-end is the first part while the asynchronous task and database back-end is the secondpart. Overview of the application architecture is in the Figure 2. Web applications are in-herently synchronous and stateless which makes them unsuitable for our purposes. The userfront-end should only serve as a tool for control and monitoring of the benchmarking process.Therefore, an asynchronous approach has to be implemented, such that the process executionis independent on the user requests to the application front-end.3.1 ImplementationThe frontend is implemented in a popular Python web framework Flask [8]. In the contextof benchmarking, it provides an user interface for setup, execution and presentation of thebenchmark results. The main page contains a form for submitting the benchmark task anda table of results of the past runs. Detail of a particular benchmark run can be accessedfrom the results table (Figure 3). The detail presented in Figure 4 contains information aboutparameters of the viewed run and its results. In this particular case, the presentation of theresults is split into data about the run itself and several metrics computed from the service log.Content of the entire page is generated on the server side directly from the service logs. Thisapproach allows fairly easy extension of the analysis with new types of metrics.3.2 BackendThe asynchronous back-end of the application is implemented with the help of the Celeryframework [9]. It provides a convenient way of execution and management of large number ofsmall tasks asynchronously and in a distributed fashion. The client part of the framework isDSpace VŠB - TUO           http://hdl.handle.net/10084/133448                      December 2018Figure 3: Interface for parameter specificationFigure 4: Visualisation of the benchmark resultused by the front-end to create the benchmarking task and pass the parameters. The Celerythen uses a type of in-memory concurrent storage (such as REDIS or RabbitMQ) to distributethe task across available worker processes. The worker processes communicate with the clientthrough the in-memory storage, which is also used for storing the intermediate results. Thismechanism is very useful in our case as it can be used to launch a large number of taskswhich are processed by a pool of workers running on a HPC cluster. Celery takes care of thetask scheduling itself and the benchmarking pipeline is implemented using the available tasksynchronization constructs.The individual service requests are executed in a celery worker tasks. Each request is sentto the service in a blocking call to HTTP requests library and the result of the call is stored inthe in-memory database.After all requests are sent, all results are passed to a finalization task,which computes aggregated statistics and stores them in the SQL database for presentation onthe front-end.The worker processes are deployed on a HPC cluster to ensure proper saturation of theservice in order to maintain consistency of the benchmark.4 Conclusion and future workThe authors presented benchmarking application which is used for transparent testing andfurther optimization of a server-side navigation service. The application implements an initialversion of a consistent benchmarking environment. It can be further extended for testing othernetwork-based services. Current version is used for testing of the autotuning framework andother tools developed within the H2020 project ANTAREX as well as for performing acceptanceDSpace VŠB - TUO           http://hdl.handle.net/10084/133448                      December 2018tests of the navigation service. It will be also used for testing of various types of navigationalgorithms for development of vehicle routing problem algorithm.In the next version, we plan to implement another type of the test, which will bypass thenetwork connection handling layer of the service and test only the routing pipeline running onthe cluster.5 AcknowledgementsThis work was supported by The Ministry of Education, Youth and Sports from the Na-tional Programme of Sustainability (NPU II) project ”IT4Innovations excellence in science- LQ1602”, by the IT4Innovations infrastructure which is supported from the Large Infras-tructures for Research, Experimental Development and Innovations project ”IT4InnovationsNational Supercomputing Center LM2015070”, and by ANTAREX, a project supported bythe EU H2020 FET-HPC program under grant 671623.References[1] Rezˇnar, T., Martinovicˇ, J., Slaninova´, K., Grakova, E., Vondra´k, V.: Probabilistic time-dependentvehicle routing problem (2016). Central European Journal of Operations Research, pp. 1–16.[2] Yichao Jin and Yonggang Wen and Qinghua Chen: Energy efficiency and server virtualization indata centers: An empirical investigation (2012). In 2012 Proceedings IEEE INFOCOMWorkshops,pp. 133–138.[3] K. Le and R. Bianchini and T. D. Nguyen and O. Bilgir and M. Martonosi: Capping the brownenergy consumption of Internet services at low cost (2010). In International Conference on GreenComputing, pp. 3–14.[4] Y. Duan and G. Fu and N. Zhou and X. Sun and N. C. Narendra and B. Hu: Everything asa Service (XaaS) on the Cloud: Origins, Current and Future Trends (2015). In 2015 IEEE 8thInternational Conference on Cloud Computing, pp. 621–628.[5] The Apache Benchmark. (2017) The Apache Software Foundationhttps://httpd.apache.org/docs/2.4/programs/ab.html [Accessed 20/05/2017].[6] Siege (2012) Jeffrey Fulmer at al.https://www.joedog.org/siege-home/ [Accessed 20/05/2017].[7] Locust (2016). Locusthttp://locust.io/ [Accessed 20/05/2017].[8] Flask (2010). Armin Ronacherhttp://flask.pocoo.org/ [Accessed 20/05/2017].[9] Celery - Distributed Task Queue (2017). Ask Solemhttp://docs.celeryproject.org/en/latest/index.html [Accessed 20/05/2017].[10] Silvano, C., Agosta, G., Cherubin, S., Gadioli, D., Palermo, G., Bartolini, A., Bispo, J., et. al.(2016). The ANTAREX approach to autotuning and adaptivity for energy efficient HPC systems.In Proceedings of the ACM International Conference on Computing Frontiers (pp. 288–293). ACM.[11] Haklay, M., Weber, P. (2008). Openstreetmap: User-generated street maps. IEEE Pervasive Com-puting, 7(4), pp. 12-18.DSpace VŠB - TUO           http://hdl.handle.net/10084/133448                      December 2018

Server-side navigation service benchmarking tool

http://dspace.vsb.cz/bitstream/10084/133448/2/2017p85.pdf

Server-side navigation service benchmarking tool

Abstract

Similar works

Full text

Available Versions

DSpace at VSB Technical University of Ostrava