2 research outputs found
On the design of a framework integrating an optimization engine with streaming technologies
A number of streaming technologies have appeared in the last years as a result of the rising of Big
Data applications. Nowadays, deciding which technology to adopt is not an easy task due not only to
the number of available data streaming processing projects, but also because they are continuously
evolving. In this paper, we focus on how these issues have affected jMetalSP, a framework for dynamic
multi-objective optimization that incorporates streaming features. jMetalSP allows the development of
three tier optimization workflows where the central component is an optimizer that is continuously
solving a dynamic multi-objective optimization problem. This problem can change as a consequence of
the analysis of data streams carried out by components that use the Apache Spark streaming engine.
A third kind of components receive and process the Pareto front approximations being yielded by
the optimization algorithm. However, all jMetalSP elements are tightly coupled and linked to Spark,
making it difficult to use a different streaming system. To overcome this issue, we have redesigned the
jMetalSP architecture to make it flexible enough to avoid the dependence of any particular streaming
system. This way, popular Apache projects such as Spark Structured Streaming, Kafka Streams, or Flink
can be used without requiring to change the rest of components of the application. Furthermore, Kafka
can be used for inter-process communication, what enables the execution of components in different
nodes of a cluster, independently of their implementation languages thanks to the serialization of data
streams with Apache Avro. We show how the embraced solution provides a high degree of flexibility
that enhances the usability of jMetalSP. To this end, a representative case study based on a transport
problem is conducted that focuses on data representation and performance evaluation of the Spark,
Flink, and Kafka systems.Ministerio de Educaci贸n y Ciencia TIN2017-86049-
Recommended from our members
OptPlatform: metaheuristic optimisation framework for solving complex real-world problems
This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonWe optimise daily, whether that is planning a round trip that visits the most attractions within a given holiday budget or just taking a train instead of driving a car in a rush hour. Many problems, just like these, are solved by individuals as part of our daily schedule, and they are effortless and straightforward. If we now scale that to many individuals with many different schedules, like a school timetable, we get to a point where it is just not feasible or practical to solve by hand. In such instances, optimisation methods are used to obtain an optimal solution. In this thesis, a practical approach to optimisation has been taken by developing an optimisation platform with all the necessary tools to be used by practitioners who are not necessarily familiar with the subject of optimisation. First, a high-performance metaheuristic optimisation framework (MOF) called OptPlatform is implemented, and the versatility and performance are evaluated across multiple benchmarks and real-world optimisation problems. Results show that, compared to competing MOFs, the OptPlatform outperforms in both the solution quality and computation time. Second, the most suitable hardware platform for OptPlatform is determined by an in-depth analysis of Ant Colony Optimisation scaling across CPU, GPU and enterprise Xeon Phi. Contrary to the common benchmark problems used in the literature, the supply chain problem solved could not scale on GPUs. Third, a variety of metaheuristics are implemented into OptPlatform. Including, a new metaheuristic based on Imperialist Competitive Algorithm (ICA), called ICA with Independence and Constrained Assimilation (ICAwICA) is proposed. The ICAwICA was compared against two different types of benchmark problems, and results show the versatile application of the algorithm, matching and in some cases outperforming the custom-tuned approaches. Finally, essential MOF features like automatic algorithm selection and tuning, lacking on existing frameworks, are implemented in OptPlatform. Two novel approaches are proposed and compared to existing methods. Results indicate the superiority of the implemented tuning algorithms within constrained tuning budget environment