Using POMDP as Modeling Framework for Network Fault Management

For highדּpeed networks, it is important that fault management be proactive--i.e., detect, diagnose, and mitigate problems before they result in severe degradation of network performance. Proactive fault manageשּׂent depends on monitoring the network to obtain the data on which to base manager decisions. However, monitoring introduces additional overhead that may itself degrade network performance especially when the network is in a stressed state. Thus, a tradeoff must be made be﫠tween the amount of data collected and transferred on one hand, and the speed and accuracy of fault detection and diagnosis on the other hand. Such a tradeoff can be naturally formulated as a Partially Observable Markov decision process (POMDP).Since exact solution of POMDPs for a realistic number of states is computationally prohibitive, we develop a reinforcementשּׁearningﬢased fast algorithm which learns the decisionגּule in an approximate network simulator and makes it fast deployable to the real network. Simulation results are given to diagnose a switch fault in an ATM network. This approach can be applied to centralized fault management or to construct intelligent agents for distributed fault management

Solving POMDP by On﬐olicy Linear Approximate Learning Algorithm

This paper presents a fast Reinforcement Learning (RL) algorithm to solve Partially Observable Markov Decision Processes (POMDP) prob﫠lem. The proposed algorithm is devised to provide a policyשּׂaking frame﫠work for Network Management Systems (NMS) which is in essence an engineering application without an exact model.The algorithm consists of two phases. Firstly, the model is estimated and policy is learned in a completely observable simulator. Secondly, the estimated model is brought into the partially observed real﬷orld where the learned policy is then fineהּuned.The learning algorithm is based on the onאּolicy linear gradientﬤescent learning algorithm with eligibility traces. This implies that the Qזּalue on belief space is linearly approximated by the Qזּalue at vertex over the belief space where onשּׁine TD method will be applied.The proposed algorithm is tested against the exact solutions to exten﫠sive small/middleדּize benchmark examples from POMDP literature and found near optimal in terms of averageﬤiscountedגּeward and stepהּo﫠goal. The proposed algorithm significantly reduces the convergence time and can easily be adapted to large stateאַumber problems

Stability of Wireless Networks for Mode S Radar

Stability issues in a connectionless, one-hop queueing system featuringservers with overlapping service regions (e.g. a Mode Select (Mode S) Radarcommunications network or part of an Aeronautical Telecommunications Network (ATN) network) are considered, and a stabilizing policy is determined in closed-loop form. The cases of queues at the sources (aircraft) and queues at the servers (base stations) are consideredseparately. Stabilizability of the system with exponential service times and Poisson arrival rates is equivalent to the solvability of a linear program and if the system is stabilizable, a stabilizing open loop routingpolicy can be expressed in terms of the coefficients of the solution to thelinear program. We solve the linear program for the case of a single class of packets.The research and scientific content in this material has beenpublished under the same title in the Proceedings of the 32nd Conference onInformation Sciences and Systems; Princeton, NJ; March 1998.</Center

Supervisory Control of Real-Time Systems Using Prioritized Synchronization

The theory of supervisory control of discrete event systems is extended to the real-time setting. The real-time behavior of a system is represented by the set of all possible timed traces of the system. This is alternatively specified using timed automata where each transition is associated with an event occurrence time set during which time the transition can occur. Our model for time is more general in that the time advances continuously as compared to a model where time advances discretely. We extend the notion of prioritized synchronous composition to the real-time setting to use it as the control mechanism. It is shown that a suitable extension of the controllability condition to the real-time setting yields a condition for the existence of a supervisor achieving a desired timed behavior. Although the real-time controllability is similar in form to its untimed counterpart, they are different in the sense that one does not imply the other and vice-versa

A Canonical Form for Controllable Singular Systems.

A new canonical form for the action of restricted system equivalence on controllable singular systems is given. The construction of this form is based on the Weierstrass decomposition of the singular system into a slow and a fast subsystem. Both subsystems are transformed into Hermite canonical form. The resulting Hermite canonical form for singular systems has a particularly simple structure and is expected to be useful for e.g. identification purposes. Continuity properties of the Hermite form are investigated and the nonexistence of a globally defined continuous canonical form for controllable singular systems is shown

Centralized and Decentralized Supervisory Control of Nondeterministic Systems Under Partial Observation

In this paper we extend our earlier work on supervisory control of nondeterministic systems using prioritized synchronization as the mechanism of control and trajectory model as the modeling formalism by considering design of supervisors under partial observation. We introduce the notion of observation-compatible systems and show that prioritized synchronous composition of observation-compatible systems can be used as a mechanism of control of nondeterministic systems under partial observation in presence of driven events. Necessary and sufficient conditions, that depend on the trajectory model as opposed to the language model of the plant, are obtained for the existence of centralized as well as decentralized supervision. Our work on centralized control shows that the results of the traditional supervisory control can be &quot;extended&quot; to the above setting provided the supervisor is deterministic and the observation mask is projection type. On the other hand, our work on decentralized co..

Topology of the Orbit Space of Generalized Linear Systems.

We investigate the topology of the orbit space of controllable generalized linear (descriptor) systems modulo restricted system equivalence. We compute the singular homology groups in the complex case, and prove that in both the real and complex cases, this space is a smooth compactification of the orbit space of controllable state space systems modulo system similarity