Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Coalition Formation under Uncertainty

Many multiagent systems require allocation of agents to tasks in order to ensure successful task execution. Most systems that perform this allocation assume that the quantity of agents needed for a task is known beforehand. Coalition formation approaches relax this assumption, allowing multiple agents to be dynamically assigned. Unfortunately, many current approaches to coalition formation lack provisions for uncertainty. This prevents application of coalition formation techniques to complex domains, such as real-world robotic systems and agent domains where full state knowledge is not available. Those that do handle uncertainty have no ability to handle dynamic addition or removal of agents from the collective and they constrain the environment to limit the sources of uncertainty. A modeling approach and algorithm for coalition formation is presented that decreases the collective\u27s dependence on knowing agent types. The agent modeling approach enforces stability, allows for arbitrary expansion of the collective, and serves as a basis for calculation of individual coalition payoffs. It explicitly captures uncertainty in agent type and allows uncertainty in coalition value and agent cost, and no agent in the collective is required to perfectly know another agents type. The modeling approach is incorporated into a two part algorithm to generate, evaluate, and join stable coalitions for task execution. A comparison with a prior approach designed to handle uncertainty in agent type shows that the protocol not only provides greater flexibility, but also handles uncertainty on a greater scale. Additional results show the application of the approach to real-world robotics and demonstrate the algorithm\u27s scalability. This provides a framework well suited to decentralized task allocation in general collectives

Gaining Insight into Determinants of Physical Activity using Bayesian Network Learning

Contains fulltext : 228326pre.pdf (preprint version ) (Open Access) Contains fulltext : 228326pub.pdf (publisher's version ) (Open Access)BNAIC/BeneLearn 202

Reinforcement Learning

Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. The first 11 chapters of this book describe and extend the scope of reinforcement learning. The remaining 11 chapters show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field

Diversifying agent's behaviors in interactive decision models

Modeling other agents' behaviors plays an important role in decision models for interactions among multiple agents. To optimize its own decisions, a subject agent needs to model what other agents act simultaneously in an uncertain environment. However, modeling insufficiency occurs when the agents are competitive and the subject agent cannot get full knowledge about other agents. Even when the agents are collaborative, they may not share their true behaviors due to their privacy concerns. Most of the recent research still assumes that the agents have common knowledge about their environments and a subject agent has the true behavior of other agents in its mind. Consequently, the resulting techniques are not applicable in many practical problem domains. In this article, we investigate into diversifying behaviors of other agents in the subject agent's decision model before their interactions. The challenges lie in generating and measuring new behaviors of other agents. Starting with prior knowledge about other agents' behaviors, we use a linear reduction technique to extract representative behavioral features from the known behaviors. We subsequently generate their new behaviors by expanding the features and propose two diversity measurements to select top‐ K $K$ behaviors. We demonstrate the performance of the new techniques in two well‐studied problem domains. The top‐ K $K$ behavior selection embarks the study of unknown behaviors in multiagent decision making and inspires investigation of diversifying agents' behaviors in competitive agent interactions. This study will contribute to intelligent systems dealing with unknown unknowns in an open artificial intelligence world

Applications of Machine Learning in Supply Chains

Advances in new technologies have resulted in increasing the speed of data generation and accessing larger data storage. The availability of huge datasets and massive computational power have resulted in the emergence of new algorithms in artificial intelligence and specifically machine learning, with significant research done in fields like computer vision. Although the same amount of data exists in most components of supply chains, there is not much research to utilize the power of raw data to improve efficiency in supply chains.In this dissertation our objective is to propose data-driven non-parametric machine learning algorithms to solve different supply chain problems in data-rich environments.Among wide range of supply chain problems, inventory management has been one of the main challenges in every supply chain. The ability to manage inventories to maximize the service level while minimizing holding costs is a goal of many company. An unbalanced inventory system can easily result in a stopped production line, back-ordered demands, lost sales, and huge extra costs. This dissertation studies three problems and proposes machine learning algorithms to help inventory managers reduce their inventory costs.In the first problem, we consider the newsvendor problem in which an inventory manager needs to determine the order quantity of a perishable product to minimize the sum of shortage and holding costs, while some feature information is available for each product. We propose a neural network approach with a specialized loss function to solve this problem. The neural network gets historical data and is trained to provide the order quantity. We show that our approach works better than the classical separated estimation and optimization approaches as well as other machine learning based algorithms. Especially when the historical data is noisy, and there is little data for each combination of features, our approach works much better than other approaches. Also, to show how this approach can be used in other common inventory policies, we apply it on an $(r,Q)$ policy and provide the results.This algorithm allows inventory managers to quickly determine an order quantity without obtaining the underling demand distribution.Now, assume the order quantities or safety stock levels are obtained for a single or multi-echelon system. Classical inventory optimization models work well in normal conditions, or in other words when all underlying assumptions are valid. Once one of the assumptions or the normal working conditions is violated, unplanned stock-outs or excess inventories arise.To address this issue, in the second problem, a multi-echelon supply network is considered, and the goal is to determine the nodes that might face a stock-out in the next period. Stock-outs are usually expensive and inventory managers try to avoid them, so stock-out prediction might results in averting stock-outs and the corresponding costs.In order to provide such predictions, we propose a neural network model and additionally three naive algorithms. We analyze the performance of the proposed algorithms by comparing them with classical forecasting algorithms and a linear regression model, over five network topologies. Numerical results show that the neural network model is quite accurate and obtains accuracies in $[0.92, 0.99]$ for the hardest to easiest network topologies, with average of 0.950 and standard deviation of 0.023, while the closest competitor, i.e., one of the proposed naive algorithms, obtains accuracies in $[0.91, 0.95]$ with average of 9.26 and standard deviation of .0136. Additionally, we suggest conditions under which each algorithm is the most reliable and additionally apply all algorithms to threshold and multi-period predictions.Although stock-out prediction can be very useful, any inventory manager would like to have a powerful model to optimize the inventory system and balance the holding and shortage costs. The literature on multi-echelon inventory models is quite rich, though it mostly relies on the assumption of accessing a known demand distribution. The demand distribution can be approximated, but even so, in some cases a globally optimal model is not available.In the third problem, we develop a machine learning algorithm to address this issue for multi-period inventory optimization problems in multi-echelon networks. We consider the well-known beer game problem and propose a reinforcement learning algorithm to efficiently learn ordering policies from data.The beer game is a serial supply chain with four agents, i.e. retailer, wholesaler, distributor, and manufacturer, in which each agent replenishes its stock by ordering beer from its predecessor. The retailer satisfies the demand of external customers, and the manufacturer orders from external suppliers. Each of the agents must decide its own order quantity to minimize the summation of holding and shortage cost of the system, while they are not allowed to share any information with other agents. For this setting, a base-stock policy is optimal, if the retailer is the only node with a positive shortage cost and a known demand distribution is available. Outside of this narrow condition, there is not a known optimal policy for this game. Also, from the game theory point of view, the beer game can be modeled as a decentralized multi-agent cooperative problem with partial observability, which is known as a NEXP-complete problem.We propose an extension of deep Q-network for making decisions about order quantities in a single node of the beer game. When the co-players follow a rational policy, it obtains a close-to-optimal solution, and it works much better than a base-stock policy if the other agents play irrationally. Additionally, to reduce the training time of the algorithm, we propose using transfer learning, which reduces the training time by one order of magnitude. This approach can be extended to other inventory optimization and supply chain problems