Eugene A. Feinberg Adam Shwartz This volume deals with the theory of Markov Decision Processes (MDPs) and their applications. Each chapter was written by a leading expert in the re spective area. The papers cover major research areas and methodologies, and discuss open questions and future research directions. The papers can be read independently, with the basic notation and concepts ofSection 1.2. Most chap ters should be accessible by graduate or advanced undergraduate students in fields of operations research, electrical engineering, and computer science. 1.1 AN OVERVIEW OF MARKOV DECISION PROCESSES The theory of Markov Decision Processes-also known under several other names including sequential stochastic optimization, discrete-time stochastic control, and stochastic dynamic programming-studiessequential optimization ofdiscrete time stochastic systems. The basic object is a discrete-time stochas tic system whose transition mechanism can be controlled over time. Each control policy defines the stochastic process and values of objective functions associated with this process. The goal is to select a "good" control policy. In real life, decisions that humans and computers make on all levels usually have two types ofimpacts: (i) they cost orsavetime, money, or other resources, or they bring revenues, as well as (ii) they have an impact on the future, by influencing the dynamics. In many situations, decisions with the largest immediate profit may not be good in view offuture events. MDPs model this paradigm and provide results on the structure and existence of good policies and on methods for their calculation.

This book presents classical Markov Decision Processes (MDP) for real-life applications and optimization. MDP allows users to develop and formally support approximate and simple decision rules, and this book showcases state-of-the-art applications in which MDP was key to the solution approach. The book is divided into six parts. Part 1 is devoted to the state-of-the-art theoretical foundation of MDP, including approximate methods such as policy improvement, successive approximation and infinite state spaces as well as an instructive chapter on Approximate Dynamic Programming. It then continues with five parts of specific and non-exhaustive application areas. Part 2 covers MDP healthcare applications, which includes different screening procedures, appointment scheduling, ambulance scheduling and blood management. Part 3 explores MDP modeling within transportation. This ranges from public to private transportation, from airports and traffic lights to car parking or charging your electric car . Part 4 contains three chapters that illustrates the structure of approximate policies for production or manufacturing structures. In Part 5, communications is highlighted as an important application area for MDP. It includes Gittins indices, down-to-earth call centers and wireless sensor networks. Finally Part 6 is dedicated to financial modeling, offering an instructive review to account for financial portfolios and derivatives under proportional transactional costs. The MDP applications in this book illustrate a variety of both standard and non-standard aspects of MDP modeling and its practical use. This book should appeal to readers for practitioning, academic research and educational purposes, with a background in, among others, operations research, mathematics, computer science, and industrial engineering.

The theory of Markov decision processes focuses on controlled Markov chains in discrete time. The authors establish the theory for general state and action spaces and at the same time show its application by means of numerous examples, mostly taken from the fields of finance and operations research. By using a structural approach many technicalities (concerning measure theory) are avoided. They cover problems with finite and infinite horizons, as well as partially observable Markov decision processes, piecewise deterministic Markov decision processes and stopping problems. The book presents Markov decision processes in action and includes various state-of-the-art applications with a particular view towards finance. It is useful for upper-level undergraduates, Master's students and researchers in both applied probability and finance, and provides exercises (without solutions).

Markov Decision Processes (MDPs) are widely popular in Artificial Intelligence for modeling sequential decision-making scenarios with probabilistic dynamics. They are the framework of choice when designing an intelligent agent that needs to act for long periods of time in an environment where its actions could have uncertain outcomes. MDPs are actively researched in two related subareas of AI, probabilistic planning and reinforcement learning. Probabilistic planning assumes known models for the agent's goals and domain dynamics, and focuses on determining how the agent should behave to achieve its objectives. On the other hand, reinforcement learning additionally learns these models based on the feedback the agent gets from the environment. This book provides a concise introduction to the use of MDPs for solving probabilistic planning problems, with an emphasis on the algorithmic perspective. It covers the whole spectrum of the field, from the basics to state-of-the-art optimal and approximation algorithms. We first describe the theoretical foundations of MDPs and the fundamental solution techniques for them. We then discuss modern optimal algorithms based on heuristic search and the use of structured representations. A major focus of the book is on the numerous approximation schemes for MDPs that have been developed in the AI literature. These include determinization-based approaches, sampling techniques, heuristic functions, dimensionality reduction, and hierarchical representations. Finally, we briefly introduce several extensions of the standard MDP classes that model and solve even more complex planning problems. Table of Contents: Introduction / MDPs / Fundamental Algorithms / Heuristic Search Algorithms / Symbolic Algorithms / Approximation Algorithms / Advanced Notes

This book provides both the research and practitioner communities with a comprehensive coverage of the metaheuristic methodologies that have proven to be successful in a wide variety of real-world problem settings. Moreover, it is these metaheuristic strategies that hold particular promise for success in the future. The various chapters serve as stand alone presentations giving both the necessary background underpinnings as well as practical guides for implementation.

Over the past thirty-five years, a substantial amount of theoretical and empirical scholarly research has been developed across the discipline domains of Transportation. This research has been synthesized into a systematic handbook that examines the scientific concepts, methods, and principles of this growing and evolving field. The Handbook of Transportation Science outlines the field of transportation as a scientific discipline that transcends transportation technology and methods. Whether by car, truck, airplane - or by a mode of transportation that has not yet been conceived - transportation obeys fundamental properties. The science of transportation defines these properties, and demonstrates how our knowledge of one mode of transportation can be used to explain the behavior of another. Transportation scientists are motivated by the desire to explain spatial interactions that result in movement of people or objects from place to place. Its methodologies draw from physics, operations research, probability and control theory.

Data Envelopment Analysis (DEA) is a relatively new “data-oriented” approach for evaluating the performances of a set of entities called Decision- Making Units (DMUs) which convert multiple inputs into multiple outputs. DEA has been used in evaluating the performances of many different kinds of entities engaged in many different kinds of activities in many different contexts. It has opened up possibilities for use in cases which have been resistant to other approaches because of the complex and often unknown nature of the relations between the multiple inputs and outputs involved in many of these activities, which are often reported in non-commeasurable units. DEA has also been used to supply new insights into activities and entities that have previously been evaluated by othermethods. This handbook is intended to represent a milestone in the progression of DEA. Written by experts, who are often major contributors to the topics to be covered, it includes a comprehensive review and discussion of basic DEA models, extensions to the basic DEA methods, and a collection of DEA applications in the areas of banking, education, sports, retail, health care, and a review of current DEA software technology. This handbook’s chapters are organized into three categories: (i) basic DEA models, concepts, and their extensions; (ii) DEA applications; and (iii) xii Preface DEA software packages. The first category consists of eleven chapters.

The Wiley-Interscience Paperback Series consists of selected books that have been made more accessible to consumers in an effort to increase global appeal and general circulation. With these new unabridged softcover volumes, Wiley hopes to extend the lives of these works by making them available to future generations of statisticians, mathematicians, and scientists. "This text is unique in bringing together so many results hitherto found only in part in other texts and papers. . . . The text is fairly self-contained, inclusive of some basic mathematical results needed, and provides a rich diet of examples, applications, and exercises. The bibliographical material at the end of each chapter is excellent, not only from a historical perspective, but because it is valuable for researchers in acquiring a good perspective of the MDP research potential." —Zentralblatt fur Mathematik ". . . it is of great value to advanced-level students, researchers, and professional practitioners of this field to have now a complete volume (with more than 600 pages) devoted to this topic. . . . Markov Decision Processes: Discrete Stochastic Dynamic Programming represents an up-to-date, unified, and rigorous treatment of theoretical and computational aspects of discrete-time Markov decision processes." —Journal of the American Statistical Association

Analysis and Modeling of Manufacturing Systems is a set of papers on some of the newest research and applications of mathematical and computational techniques to manufacturing systems and supply chains. These papers deal with fundamental questions (how to predict factory performance: how to operate production systems) and explicitly treat the stochastic nature of failures, operation times, demand, and other important events. Analysis and Modeling of Manufacturing Systems will be of interest to readers with a strong background in operations research, including researchers and mathematically sophisticated practitioners.

Dynamic Portfolio Strategies: Quantitative Methods and Empirical Rules for Incomplete Information investigates optimal investment problems for stochastic financial market models. It is addressed to academics and students who are interested in the mathematics of finance, stochastic processes, and optimal control, and also to practitioners in risk management and quantitative analysis who are interested in new strategies and methods of stochastic analysis. While there are many works devoted to the solution of optimal investment problems for various models, the focus of this book is on analytical strategies based on "technical analysis" which are model-free. The technical analysis of these strategies has a number of characteristics. Two of the more important characteristics are: (1) they require only historical data, and (2) typically they are more widely used by traders than analysis based on stochastic models. Hence it is the objective of this book to reduce the gap between model-free strategies and strategies that are "optimal" for stochastic models. We hope that researchers, students and practitioners will be interested in some of the new empirically based methods of "technical analysis" strategies suggested in this book and evaluated via stochastic market models.