Call Admission Control and Routing in Integrated Services Networks Using Neuro-Dynamic Programming

Abstract—We consider the problem of call admission control (CAC) and routing in an integrated services network that handles several classes of calls of different value and with different resource requirements. The problem of maximizing the average value of admitted calls per unit time (or of revenue maximization) is naturally formulated as a dynamic programming problem, but is too complex to allow for an exact solution. We use methods of neuro-dynamic programming (NDP) [reinforcement learning (RL)], together with a decomposition approach, to construct dynamic (state-dependent) call admission control and routing policies. These policies are based on state-dependent link costs, and a simulation-based learning method is employed to tune the parameters that define these link costs. A broad set of experiments shows the robustness of our policy and compares its performance with a commonly used heuristic. Index Terms—ART neural networks, communication system control, communication system routing, dynamic programming, Markov processes. I

Wavelength assignment in optical burst switching networks using neuro-dynamic programming

Cataloged from PDF version of article.All-optical networks are the most promising architecture for building large-size, hugebandwidth transport networks that are required for carrying the exponentially increasing Internet traffic. Among the existing switching paradigms in the literature, the optical burst switching is intended to leverage the attractive properties of optical communications, and at the same time, take into account its limitations. One of the major problems in optical burst switching is high blocking probability that results from one-way reservation protocol used. In this thesis, this problem is solved in wavelength domain by using smart wavelength assignment algorithms. Two heuristic wavelength assignment algorithms prioritizing available wavelengths according to reservation tables at the network nodes are proposed. The major contribution of the thesis is the formulation of the wavelength assignment problem as a continuous-time, average cost dynamic programming problem and its solution based on neuro-dynamic programming. Experiments are done over various traffic loads, burst lengths, and number of wavelength converters with a pool structure. The simulation results show that the wavelength assignment algorithms proposed for optical burst switching networks in the thesis perform better than the wavelength assignment algorithms in the literature that are developed for circuit-switched optical networks.Keçeli, FeyzaM.S

A review of connection admission control algorithms for ATM networks

The emergence of high-speed networks such as those with ATM integrates large numbers of services with a wide range of characteristics. Admission control is a prime instrument for controlling congestion in the network. As part of connection services to an ATM system, the Connection Admission Control (CAC) algorithm decides if another call or connection can be admitted to the Broadband Network. The main task of the CAC is to ensure that the broadband resources will not saturate or overflow within a very small probability. It limits the connections and guarantees Quality of Service for the new connection. The algorithm for connection admission is crucial in determining bandwidth utilisation efficiency. With statistical multiplexing more calls can be allocated on a network link, while still maintaining the Quality of Service specified by the connection with traffic parameters and type of service. A number of algorithms for admission control for Broadband Services with ATM Networks are described and compared for performance under different traffic loads. There is a general description of the ATM Network as an introduction. Issues to do with source distributions and traffic models are explored in Chapter 2. Chapter 3 provides an extensive presentation of the CAC algorithms for ATM Broadband Networks. The ideas about the Effective Bandwidth are reviewed in Chapter 4, and a different approach to admission control using online measurement is presented in Chapter 5. Chapter 6 has the numerical evaluation of four of the key algorithms, with simulations. Finally Chapter 7 has conclusions of the findings and explores some possibilities for further work

On the Orchestration and Provisioning of NFV-enabled Multicast Services

The paradigm of network function virtualization (NFV) with the support of software-defined networking has emerged as a prominent approach to foster innovation in the networking field and reduce the complexity involved in managing modern-day conventional networks. Before NFV, functions, which can manipulate the packet header and context of traffic flow, used to be implemented at fixed locations in the network substrate inside proprietary physical devices (called middlewares). With NFV, such functions are softwarized and virtualized. As such, they can be deployed in commodity servers as demanded. Hence, the provisioning of a network service becomes more agile and abstract, thereby giving rise to the next-generation service-customized networks which have the potential to meet new demands and use cases. In this thesis, we focus on three complementary research problems essential to the orchestration and provisioning of NFV-enabled multicast network services. An NFV-enabled multicast service connects a source with a set of destinations. It specifies a set of NFs that should be executed at the chosen routes from the source to the destinations, with some resources and ordering relationships that should be satisfied in wired core networks. In Problem I, we investigate a static joint traffic routing and virtual NF placement framework for accommodating multicast services over the network substrate. We develop optimal formulations and efficient heuristic algorithms that jointly handle the static embedding of one or multiple service requests over the network substrate with single-path and multipath routing. In Problem II, we study the online orchestration of NFV-enabled network services. We consider both unicast and multicast NFV-enabled services with mandatory and best-effort NF types. Mandatory NFs are strictly necessary for the correctness of a network service, whereas best-effort NFs are preferable yet not necessary. Correspondingly, we propose a primal-dual based online approximation algorithm that allocates both processing and transmission resources to maximize a profit function that is proportional to the throughput. The online algorithm resembles a joint admission mechanism and an online composition, routing, and NF placement framework. In the core network, traffic patterns exhibit time-varying characteristics that can be cumbersome to model. Therefore, in Problem III, we develop a dynamic provisioning approach to allocate processing and transmission resources based on the traffic pattern of the embedded network service using deep reinforcement learning (RL). Notably, we devise a model-assisted exploration procedure to improve the efficiency and consistency of the deep RL algorithm

The application of Approximate Dynamic Programming techniques

Koole, G.M. [Promotor]Bhulai, S. [Copromotor

An Approximate Dynamic Programming Approach to the Scheduling of Impatient Jobs in a Clearing System.

A single server is faced with a collection of jobs of varying duration and urgency. Before service starts, all jobs are subject to an initial triage, i.e., an assessment of both their urgency and of their service requirement, and are allocated to distinct classes. Jobs in one class have independent and identically distributed lifetimes during which they are available for service. Should a job's lifetime example before its service begins then it is lost from the system unserved. The goal is to schedule the jobs for service to maximise the expected number served to completion. Two heuristic policies have been proposed in the literature. One works well in a "no loss" limit while the other does so when lifetimes are short. Both can exhibit poor performance for problems at some distance from the regimes for which they were designed. We develop a robustly good heuristic by an approximative approach to the application of a single policy improvement step to the first policy above, in which we use a fluid model to obtain an approximation for its value function. The performance of the proposed heuristic is investigated in an extensive numerical study. This problem is substantially complicated if the initial triage is subject to error. We take a Bayesian approach to this additional uncertainty and discuss the design of heuristic policies to maximise the Bayes' return. We identify problem features for which a high price is paid for poor initial triage and for which improvements in initial job assessment yield significant improvements in service outcomes. An analytical upperbound for the cost of imperfect classification is developed for exponentially distributed lifetime cases. An extensive numerical study is conducted to explore the behaviour of the cost in more general situations