Resource allocation problems in stochastic sequential decision making

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Civil and Environmental Engineering, 2009.Includes bibliographical references (p. 159-162).In this thesis, we study resource allocation problems that arise in the context of stochastic sequential decision making problems. The practical utility of optimal algorithms for these problems is limited due to their high computational and storage requirements. Also, an increasing number of applications require a decentralized solution. We develop techniques for approximately solving certain class of resource allocation problems that arise in the context of stochastic sequential decision making problems that are computationally efficient with a focus on decentralized algorithms where appropriate. The first resource allocation problem that we study is a stochastic sequential decision making problem with multiple decision makers (agents) with two main features 1) Partial observability Each agent may not have complete information regarding the system 2) Limited Communication - Each agent may not be able to communicate with all other agents at all times. We formulate a Markov Decision Process (MDP) for this problem. The features of partial observability and limited communication impose additional computational constraints on the exact solution of the MDP. We propose a scheme for approximating the optimal Q function and the optimal value function associated with this MDP as a linear combination of preselected basis functions. We show that the proposed approximation scheme leads to decentralization of the agents' decisions thereby enabling their implementation under limited communication. We propose a linear program, ALP, for selecting the parameters for combining the basis functions. We establish bounds relating the approximation error due to the choice of the parameters selected by the ALP with the best possible error given the choice of basis functions.(cont.) Motivated by the need for a decentralized solution to the ALP, which is equivalent to a resource allocation problem with separable, concave objective function, we analyze a general class of resource allocation problems with separable concave objective functions. We propose a distributed algorithm for this class of problems when the objective function is differentiable and establish its convergence and convergence rate properties. We develop a smoothing scheme for non-differentiable objective functions and extend the algorithm for this case. Finally, we build on these results to extend the decentralized algorithm to accommodate non-negativity constraints on the resources. Numerical investigations on the performance of the developed algorithm show that our algorithm is competitive with its centralized counterpart. The second resource allocation problem that we study is the problem of optimally accepting or rejecting arriving orders in a Make-To-Order (MTO) manufacturing firm. We model the production facility of the MTO manufacturing firm as a queue and view the time of the production facility as a resource that needs to be optimally allotted between current and future orders. We formulate the Order Acceptance Problem under two arrival processes - Poisson process (OAP-P), and Bernoulli Process (OAP-B) and formulate both problems as MDPs. We provide insights into the structure of the optimal order acceptance policy for OAP-B under the assumption of First Come First Served (FCFS) scheduling of accepted orders.(cont.) We investigate a class of randomized order acceptance policies for OAP-B called static policies that are practically relevant due to their ease of implementation and develop a procedure for computing the policy gradient for any static policy. Using these results for OAP-B, we propose 4 heuristics for OAP-P. We numerically investigate the performance of the proposed heuristics and compare their performance with other heuristics reported in literature. One of our proposed heuristics, FCFS-ValueFunction outperforms other heuristics under a variety of conditions while also being easy to implement.by Hariharan Lakshmanan.Ph.D

Resource management in QoS-aware wireless cellular networks

2011 Summer.Includes bibliographical references.Emerging broadband wireless networks that support high speed packet data with heterogeneous quality of service (QoS) requirements demand more flexible and efficient use of the scarce spectral resource. Opportunistic scheduling exploits the time-varying, location-dependent channel conditions to achieve multiuser diversity. In this work, we study two types of resource allocation problems in QoS-aware wireless cellular networks. First, we develop a rigorous framework to study opportunistic scheduling in multiuser OFDM systems. We derive optimal opportunistic scheduling policies under three common QoS/fairness constraints for multiuser OFDM systems--temporal fairness, utilitarian fairness, and minimum-performance guarantees. To implement these optimal policies efficiently, we provide a modified Hungarian algorithm and a simple suboptimal algorithm. We then propose a generalized opportunistic scheduling framework that incorporates multiple mixed QoS/fairness constraints, including providing both lower and upper bound constraints. Next, taking input queues and channel memory into consideration, we reformulate the transmission scheduling problem as a new class of Markov decision processes (MDPs) with fairness constraints. We investigate the throughput maximization and the delay minimization problems in this context. We study two categories of fairness constraints, namely temporal fairness and utilitarian fairness. We consider two criteria: infinite horizon expected total discounted reward and expected average reward. We derive and prove explicit dynamic programming equations for the above constrained MDPs, and characterize optimal scheduling policies based on those equations. An attractive feature of our proposed schemes is that they can easily be extended to fit different objective functions and other fairness measures. Although we only focus on uplink scheduling, the scheme is equally applicable to the downlink case. Furthermore, we develop an efficient approximation method--temporal fair rollout--to reduce the computational cost

Matching Theory for Future Wireless Networks: Fundamentals and Applications

The emergence of novel wireless networking paradigms such as small cell and cognitive radio networks has forever transformed the way in which wireless systems are operated. In particular, the need for self-organizing solutions to manage the scarce spectral resources has become a prevalent theme in many emerging wireless systems. In this paper, the first comprehensive tutorial on the use of matching theory, a Nobelprize winning framework, for resource management in wireless networks is developed. To cater for the unique features of emerging wireless networks, a novel, wireless-oriented classification of matching theory is proposed. Then, the key solution concepts and algorithmic implementations of this framework are exposed. Then, the developed concepts are applied in three important wireless networking areas in order to demonstrate the usefulness of this analytical tool. Results show how matching theory can effectively improve the performance of resource allocation in all three applications discussed