Hyper-Heuristics based on Reinforcement Learning, Balanced Heuristic Selection and Group Decision Acceptance

In this paper, we introduce a multi-objective selection hyper-heuristic approach combining Reinforcement Learning, (meta)heuristic selection, and group decision-making as acceptance methods, referred to as Hyper-Heuristic based on Reinforcement LearnIng, Balanced Heuristic Selection and Group Decision AccEptance (HRISE), controlling a set of Multi-Objective Evolutionary Algorithms (MOEAs) as Low-Level (meta)Heuristics (LLHs). Along with the use of multiple MOEAs, we believe that having a robust LLH selection method as well as several move acceptance methods at our disposal would lead to an improved general-purpose method producing most adequate solutions to the problem instances across multiple domains. We present two learning hyper-heuristics based on the HRISE framework for multi-objective optimisation, each embedding a group decision-making acceptance method under a different rule: majority rule (HRISE_M) and responsibility rule (HRISE_R). A third hyper-heuristic is also defined where both a random LLH selection and a random move acceptance strategy are used. We also propose two variants of the late acceptance method and a new quality indicator supporting the initialisation of selection hyper-heuristics using low computational budget. An extensive set of experiments were performed using 39 multi-objective problem instances from various domains where 24 are from four different benchmark function classes, and the remaining 15 instances are from four different real-world problems. The cross-domain search performance of the proposed learning hyper-heuristics indeed turned out to be the best, particularly HRISE_R, when compared to three other selection hyper-heuristics, including a recently proposed one, and all low-level MOEAs each run in isolation

Fuzzy adaptive parameter control of a late acceptance hyper-heuristic

A traditional iterative selection hyper-heuristic which manages a set of low level heuristics relies on two core components, a method for selecting a heuristic to apply at a given point, and a method to decide whether or not to accept the result of the heuristic application. In this paper, we present an initial study of a fuzzy system to control the list-size parameter of late- acceptance move acceptance method as a selection hyper-heuristic component. The performance of the fuzzy controlled selection hyper-heuristic is compared to its fixed parameter version and the best hyper-heuristic from a competition on the MAX-SAT problem domain. The results illustrate that a fuzzy control system can potentially be effective within a hyper-heuristic improving its performance

An iterated multi-stage selection hyper-heuristic

There is a growing interest towards the design of reusable general purpose search methods that are applicable to different problems instead of tailored solutions to a single particular problem. Hyper-heuristics have emerged as such high level methods that explore the space formed by a set of heuristics (move operators) or heuristic components for solving computationally hard problems. A selection hyper-heuristic mixes and controls a predefined set of low level heuristics with the goal of improving an initially generated solution by choosing and applying an appropriate heuristic to a solution in hand and deciding whether to accept or reject the new solution at each step under an iterative framework. Designing an adaptive control mechanism for the heuristic selection and combining it with a suitable acceptance method is a major challenge, because both components can influence the overall performance of a selection hyper-heuristic. In this study, we describe a novel iterated multi-stage hyper-heuristic approach which cycles through two interacting hyper-heuristics and operates based on the principle that not all low level heuristics for a problem domain would be useful at any point of the search process. The empirical results on a hyper-heuristic benchmark indicate the success of the proposed selection hyper-heuristic across six problem domains beating the state-of-the-art approach

A case study of controlling crossover in a selection hyper-heuristic framework using the multidimensional knapsack problem

Hyper-heuristics are high-level methodologies for solving complex problems that operate on a search space of heuristics. In a selection hyper-heuristic framework, a heuristic is chosen from an existing set of low-level heuristics and applied to the current solution to produce a new solution at each point in the search. The use of crossover low-level heuristics is possible in an increasing number of general-purpose hyper-heuristic tools such as HyFlex and Hyperion. However, little work has been undertaken to assess how best to utilise it. Since a single-point search hyper-heuristic operates on a single candidate solution, and two candidate solutions are required for crossover, a mechanism is required to control the choice of the other solution. The frameworks we propose maintain a list of potential solutions for use in crossover. We investigate the use of such lists at two conceptual levels. First, crossover is controlled at the hyper-heuristic level where no problem-specific information is required. Second, it is controlled at the problem domain level where problem-specific information is used to produce good-quality solutions to use in crossover. A number of selection hyper-heuristics are compared using these frameworks over three benchmark libraries with varying properties for an NP-hard optimisation problem: the multidimensional 0-1 knapsack problem. It is shown that allowing crossover to be managed at the domain level outperforms managing crossover at the hyper-heuristic level in this problem domain. © 2016 Massachusetts Institute of Technolog

A tensor-based selection hyper-heuristic for cross-domain heuristic search

Hyper-heuristics have emerged as automated high level search methodologies that manage a set of low level heuristics for solving computationally hard problems. A generic selection hyper-heuristic combines heuristic selection and move acceptance methods under an iterative single point-based search framework. At each step, the solution in hand is modified after applying a selected heuristic and a decision is made whether the new solution is accepted or not. In this study, we represent the trail of a hyper-heuristic as a third order tensor. Factorization of such a tensor reveals the latent relationships between the low level heuristics and the hyper-heuristic itself. The proposed learning approach partitions the set of low level heuristics into two subsets where heuristics in each subset are associated with a separate move acceptance method. Then a multi-stage hyper-heuristic is formed and while solving a given problem instance, heuristics are allowed to operate only in conjunction with the associated acceptance method at each stage. To the best of our knowledge, this is the first time tensor analysis of the space of heuristics is used as a data science approach to improve the performance of a hyper-heuristic in the prescribed manner. The empirical results across six different problem domains from a benchmark indeed indicate the success of the proposed approach

Markov Chain Selection Hyper-heuristic for the Optimisation of Constrained Magic Squares

UKCI 2015: UK Workshop on Computational Intelligence, University of Exeter, UK, 7-9 September 2015A square matrix of size n × n, containing each of the numbers (1, . . . , n2) in which every row, column and both diagonals has the same total is referred to as a magic square. The problem can be formulated as an optimisation problem where the task is to minimise the deviation from the magic square constraints and is tackled here by using hyper-heuristics. Hyper-heuristics have recently attracted the attention of the artificial intelligence, operations research, engineering and computer science communities where the aim is to design and develop high level strategies as general solvers which are applicable to a range of different problem domains. There are two main types of hyper-heuristics in the literature: methodologies to select and to generate heuristics and both types of approaches search the space of heuristics rather than solutions. In this study, we describe a Markov chain selection hyper-heuristic as an effective solution methodology for optimising constrained magic squares. The empirical results show that the proposed hyper-heuristic is able to outperform the current state-of-the-art method

A comparative study of fuzzy parameter control in a general purpose local search metaheuristic

There is a growing number of studies on general purpose metaheuristics that are directly applicable to multiple domains. Parameter setting is a particular issue considering that many of such search methods come with a set of parameters to be configured. Fuzzy logic has been used extensively in control applications and is known for its ability to handle uncertainty. In this study, we investigate the potential of using fuzzy systems to control the parameter settings of a threshold accepting (TA) metaheuristic for improving the overall effectiveness of a cross-domain approach. We have evaluated the performance of various general purpose local search metaheuristics which mix multiple heuristics at random and apply the TA metaheuristic with fixed threshold, crisp (non-fuzzy) rule-based control of the threshold and various fuzzy systems controlling the threshold. The empirical results show that the approach using the TA with crisp rule-based control performs the best across six problem domains from a benchmark

A grouping hyper-heuristic framework: application on graph colouring

Grouping problems are hard to solve combinatorial optimisation problems which require partitioning of objects into a minimum number of subsets while a given objective is simultaneously optimised. Selection hyper-heuristics are high level general purpose search methodologies that operate on a space formed by a set of low level heuristics rather than solutions. Most of the recently proposed selection hyper-heuristics are iterative and make use of two key methods which are employed successively; heuristic selection and move acceptance. In this study, we present a novel generic selection hyper-heuristic framework containing a fixed set of reusable grouping low level heuristics and an unconventional move acceptance mechanism for solving grouping problems. This framework deals with one solution at a time at any given decision point during the search process. Also, a set of high quality solutions, capturing the trade-off between the number of groups and the additional objective for the given grouping problem, is maintained. The move acceptance mechanism embeds a local search approach which is capable of progressing improvements on those trade-off solutions. The performance of different selection hyper-heuristics with various components under the proposed framework is investigated on graph colouring as a representative grouping problem. Then, the top performing hyper-heuristics are applied to a benchmark of examination timetabling instances. The empirical results indicate the effectiveness and generality of the proposed framework enabling grouping hyper-heuristics to achieve high quality solutions in both domains. ©2015 Elsevier Ltd. All rights reserved

A modified choice function hyper-heuristic controlling unary and binary operators

Hyper-heuristics are a class of high-level search methodologies which operate on a search space of low-level heuristics or components, rather than on solutions directly. Traditional iterative selection hyper-heuristics rely on two key components, a heuristic selection method and a move acceptance criterion. Choice Function heuristic selection scores heuristics based on a combination of three measures, selecting the heuristic with the highest score. Modified Choice Function heuristic selection is a variant of the Choice Function which emphasises intensification over diversification within the heuristic search process. Previous work has shown that improved results are possible in some problem domains when using Modified Choice Function heuristic selection over the classic Choice Function, however in most of these cases crossover low-level heuristics (operators) are omitted. In this paper, we introduce crossover low-level heuristics into a Modified Choice Function selection hyper-heuristic and present results over six problem domains. It is observed that although on average there is an increase in performance when using crossover low-level heuristics, the benefit of using crossover can vary on a per-domain or per-instance basis

A dynamic multiarmed bandit-gene expression programming hyper-heuristic for combinatorial optimization problems

Hyper-heuristics are search methodologies that aim to provide high-quality solutions across a wide variety of problem domains, rather than developing tailor-made methodologies for each problem instance/domain. A traditional hyper-heuristic framework has two levels, namely, the high level strategy (heuristic selection mechanism and the acceptance criterion) and low level heuristics (a set of problem specific heuristics). Due to the different landscape structures of different problem instances, the high level strategy plays an important role in the design of a hyper-heuristic framework. In this paper, we propose a new high level strategy for a hyper-heuristic framework. The proposed high-level strategy utilizes a dynamic multiarmed bandit-extreme value-based reward as an online heuristic selection mechanism to select the appropriate heuristic to be applied at each iteration. In addition, we propose a gene expression programming framework to automatically generate the acceptance criterion for each problem instance, instead of using human-designed criteria. Two well-known, and very different, combinatorial optimization problems, one static (exam timetabling) and one dynamic (dynamic vehicle routing) are used to demonstrate the generality of the proposed framework. Compared with state-of-the-art hyper-heuristics and other bespoke methods, empirical results demonstrate that the proposed framework is able to generalize well across both domains. We obtain competitive, if not better results, when compared to the best known results obtained from other methods that have been presented in the scientific literature. We also compare our approach against the recently released hyper-heuristic competition test suite. We again demonstrate the generality of our approach when we compare against other methods that have utilized the same six benchmark datasets from this test suite