Kadota, Yoshinobu

Kurano, Masami

Yasuda, Masami

Kyoto University Research Information Repository

TitleRegret-Optimality Equation in Semi-MDP's with an AbsorbingSet (Mathematical Programming Concerning DecisionMakings and Uncertainties)Author(s)Kadota, Yoshinobu; Kurano, Masami; Yasuda, MasamiCitation数理解析研究所講究録 (2004), 1373: 9-16Issue Date2004-05URL http://hdl.handle.net/2433/25520RightType Departmental Bulletin PaperTextversionpublisherKyoto University8Regret-Optimality Equation inSemi-MDP’s with an Absorbing Set(Yoshinobu KADOTA, Faculty of Education, Wakayama University)(Masami KURANO, Faculty of Education, Chiba University) $)$(Masami YASUDA, Faculty of Science, Chiba University)Abstract. In terms of countable state Semi-Markov decision processes, theexpected regret-utility incurred until the first-passage time into the absorbingset is considered. The utility of regret is represented using two variables, oneis the target value and the other is the present value. We call it the regret-utility function. In order to characterize the regret-Optimal policy, we derivethe optimality equation, for which the uniqueness of solution is proved. Asapplication, a few examples of regret-utility functions are given, under whichsome analyses are developed.Keywards: Regret-Optimal policy, Semi-Markov decision processes, Gen-eral regret-utility, Optimality equation.1. Introduction and notationIn social life or in business, the decision making is commonly executed by making theregret incurred from the decision as small as possible, where the regret means the differencebetween the target value and the real payoff. In this paper, the above problem will beconsidered in terms of countable state Semi-Markov decision processes (SMDP’s) with anabsorbing set.The utility of regret is represented by a function using two variables, one is the targetvalue and the other is the real payoff, called regret-utility function, and the problem to besolved is to minimize the expected regret-utility incurred until the first-passage time intothe absorbing set. As for utility functions, refer to $[3, 9]$ and for general utility treatmentof Markov decision processes refer to [1, 2, 5, 6, 7].In the remainder of this section, we define the regret-utility optimization problem forSMDP’s to be examined in the sequel.SMDP’s are specified by (i) a countable state space $S=\{0,1,2,3 . \}$ , (ii) a finiteaction space $A$ , (iii) transition probability matrices $p=\{p_{ij}(a)|i, j\in S, a\in A\}$ , (iv)distribution functions $\{F_{ij}(\cdot|a)|i, j\in S, a\in A\}$ of the time between transitions, (v) animmediate reward $r$ and reward rate $d$ which are functions from $S\cross$ $A$ to $R_{+}$ , where$R_{+}=$ $[0, \infty)$ . When the system is in state $i\in S$ and action $a\in A$ is taken, then itmoves to a new state $j\in S$ with the sojourn time $\tau$ , and the reward $r(i, a)+d(i, a)\tau$ isobtained, where the new state $j$ and the sojourn time $\mathrm{r}$ are distributed with Pi.{a) and$F_{ij}(\cdot|a)$ respectively. This process is repeated from the new state $j\in S.$1373 2004 9-16ioThe sample space is the product space $0=(S\cross A\cross R_{+})^{\infty}$ . Let $X_{n}$ , $\Delta_{n}$ and $\tau_{n+1}$be random quantities such that $X_{n}(\omega)=x_{n}$ , $\Delta_{n}(\mathrm{u})$ $=a_{n}$ and $\tau_{n+1}(\omega)=tn$ 11 for all$JJ$ $=$ $(\mathrm{x}\mathrm{o}, a_{0}, t_{1}, x_{1}, a_{1}, t_{2}, \cdot \cdot\cdot)$ $\in\Omega$ and $n=0,1,2$ , $\cdots$ . A policy $\pi$ $=$ (xo, $\pi_{1},$ $\cdots$ ) is asequence of conditional probabilities $\pi_{n}$ such that $\pi_{n}(A|x_{0}, a_{0}, t_{1}, \cdots, x_{n})=1$ for allhistories $(x_{0}, a_{0}, t_{1}, , . . , x_{n})\in(S\cross A\cross R_{+})^{n}\cross$ S. The set of all policies is denoted by$\Pi$ . A policy $\pi$ $=$ $(\mathrm{x}\mathrm{o}, \pi_{1}, \cdots)$ is called stationary if there exists a function $f$ : $Sarrow A$such that $\pi_{n}(\{f(x_{n})\}|x_{0}, a_{0}, t_{1}, \cdots, x_{n})=1$ for all $n\mathit{2}$ $0$ and $(x_{0}, a_{0}, t_{1}, | \cdot\cdot, x_{n})\in$$(S\cross A\cross R_{+})^{n}\cross S.$ Such a policy is denoted by $f^{\infty}$ .For any $\pi\in\Pi$ , we assume that(i) Prob$(X_{n+1}=/|X_{0}, \Delta_{0}, \tau_{1}, , . . ,X_{n}=i, \Delta_{n}=a)=p_{ij}(a)$and(ii) $Prob(\tau_{n+1}\leqq t|X_{0}, \Delta_{0}, \mathrm{r}_{1}, \cdots, X_{n}=i, \Delta_{n}=a, X_{n+1}=j)$ $=F_{ij}(t|a)$for all $n\geqq 0$ , $i$ , $j\in S$ and $a\in A.$ Then, any initial state $i\in S$ and policy $\pi$ $\in$ II determinethe probability measure $P_{\pi}(\cdot|X_{0}=i)$ on $\Omega$ by a usual way.We make the general assumption: There exists an absorbing set $J_{0}\subset S$ and $J_{0}\neq S,$ forwhich that $\sum p_{ij}(a)=1$ and $r(i, a)=d(i, a)=0$ for all $i\in J_{0}$ and $a\in A.$ Let $J=$ S-Jqand $N$ be$\mathrm{t}\mathrm{h}\mathrm{e}J$first-passage epoch into $J_{0}$ i.e.,$N= \min\{n|X_{n}\in J_{0}, n\geqq 0\}$ , where $\min\emptyset=\infty$ .The present value and the total lapsed time of the process { $X_{n}$ , $\Delta_{n}$ , $\tau_{n+1}$ : $n=0,1,2$ ,, . .} until the $\ell$-th epoch are defined respectively by$\overline{D}_{\ell}=\sum_{n=0}^{\mathit{1}-1}$ $(r(X_{n}, \Delta_{n})+\tau_{n+1}d(X_{n}, \Delta_{n}))$ and$\overline{\tau}$/ $= \sum_{n=1}^{\ell}\tau_{n}$ , $(\mathrm{f} \geqq 1)$ .Let $G$ : $R_{+}\cross R_{+}arrow R$ be a Borel-measurable function, which will be called a regret-utility function. Then, for a constant $g^{*}$ , called a target value, our problem is to minimizethe expected regret-utility with a target $g^{*}$$E_{\pi}(G(g^{*}\tilde{\tau}_{N},\overline{D}_{N})|X_{0}=i)$ over all $\pi\in\Pi$ ,where $E_{\pi}(\cdot|X_{0}=i)$ is the expectation with respect to $P_{\pi}(\cdot|X_{0}=i)$ . We say that $\pi^{*}\in\Pi$is regret-Optimal with a target 9’ if$E_{\pi^{*}}$ ($G(g^{*}\overline{\tau}_{N},\overline{D}_{N})$ $|X_{0}=i)\leqq E_{\pi}(G(g^{*}\overline{\tau}_{N},\overline{D}_{N})$ $|X_{0}=i$ )for all $\pi$ $\in$ II and $i\in S.$In Section 2, under some reasonable assumptions concerning the speed with which thedecision process is driven into $J_{0}$ , we give the optimality equation in order to characterizethe regret-Optimal policy. Also, uniqueness of solution to the optimality equation is proved.In Section 3, as applications of our results, some examples of regret-utility functions aregiven, under which some analyses are developed.112. Regret-Optimality and related optimality equationsTo develop our discussion, the following assumption is needed.Assumption 1. The following $(\mathrm{i})-(\mathrm{i}\mathrm{i})$ holds:(i) $0\leq r(i, a)\leqq M_{1}<00,$ $0\leqq d(i, a)\leqq M_{2}<\infty$ for all $i\in S$ , $a\in A$ and some $M_{1}$and $M_{2}$ .(ii) There exist $L>0$ , $B>0$ with$L \leqq\int_{0}^{\infty}tF_{ij}(dt|a)\leqq B$ for all $i$ , $j\in S$ and $a\in A.$For each $i\in J$ and $n\geqq 0,$ we define $e_{i}(n)$ by$e_{i}(n)= \sup_{\pi\in\Pi}P_{\pi}(X_{n}\in J|X_{0}=i)$ ,which means the maximal probability of being not yet absorbed in $J_{0}$ at the $n$-th epoch.Putting $e(n)= \sup_{i\in J}e_{\dot{\mathrm{t}}}(n)$ , it clearly holds (cf. [4]) that $e(n+1)\leqq e(n)$ and $e(m+n)\leqq$$e(m)e(n)$ for all $m$ , $n\geqq 0.$ The following assumption is needed.Assumption 2. It holds that $\delta_{0}:=\sum_{n=0}^{\infty}e(n)<\infty$ .If the following Assumption 2’ holds, Assumption 2 follows.Assumption 2’. There exist $0<\eta_{0}<1$ and $n_{0}\geqq 1$ such that $e(n_{0})<1-\eta_{0}$ .In fact, if Assumption 2’ holds, we have thatfol$\delta_{0}$ $=$ $\sum_{n=0}^{\infty}e(n)=\sum_{k=0}^{\infty}\sum_{n=0}^{n0-1}e(kn_{0}+n)\leqq\sum_{k=0}^{\infty}n_{0}e(kn_{0})$$\leqq$ $n_{0} \sum_{k=0}^{\infty}e(n_{0})^{k}\leqq n_{0}\eta_{0}^{-1}<\infty$ ,which shows that Assumption 2 holds.Since $P_{\pi}(N>n|X_{0}=i)$ $\leqq e(n)$ for $n\geqq 0,$ it holds that $E_{\pi}(N|X_{0}=i)$ $\leqq\delta_{0}$ , whichimplies $\lim_{narrow\infty}nP_{\pi}(N>n|X_{0}=i)$ $=0$ for any $\pi$ $\in\Pi$ .Here, we define an optimal value function when starting from the initial state $i\in S$ by(2.1) $g_{i}(c_{1}, c_{2})= \inf_{\pi\in\Pi}E_{\pi}(G(c_{1}+g^{*}\tilde{\tau}_{N}, c_{2}+\overline{D}_{N})|X_{0}=i)$ .By the above definition, we observe that $g_{i}$ (cf. $c_{2}$ ) $=$ G(ci, $c_{2}$ ) for $i\in J_{0}$ and $g_{i}(0,0)$ isthe optimal expected regret-utility in our optimization problem.The following assumption is needed to characterize the optimal value function.Assumption 3. There exists a $K>0$ such that(2.2) $\int_{(\}}$”$|G(c_{1}+g^{*}t, c_{2}+r(i, a)+d(i, a)t)-$ $G(ci, c_{2})$ $|F_{j}\dot{.}(di| a)\leqq K$for all $c_{1}$ , $c_{2}\in R_{+}$ , $i$ , $j\in S$ and $a\in A.$12Remark. If $G(c_{1}, c_{2})$ is differentiate and $| \frac{\partial G(c_{1},c_{2})}{\partial c_{1}}|$ and $| \frac{\partial G(c_{1},c_{2})}{\partial c_{2}}|$ are uniformlybounded in (ci, $c_{2}$ ) $\in R_{+}\cross R_{+}$ , Assumption 3 holds from applying the mean value theoremand Assumption 1. Hereafter, Assumption 1, 2 and 3 will be remained operative.Lemma 2.1. There exists a $K’>0$ such that(2.3)$7_{0}$”$|g_{\mathrm{i}}(c_{1} +g^{*}t, c_{2}+r(i, a)+d(i, a)t)$ $-$ 9G(ci, $c_{2}$ ) $|F_{j}$. $(dt| a)\leqq K^{*}$for all ci, $c_{2}\in R_{+}$ , $i$ , $j\in S$ and $a\in A.$Proof. Let $t$ be such that $g_{i}(c_{1}+g^{*}t, c_{2}+r(i, a)+d(i, a)t)-$ 9G(ci, $c_{2}$) $\geqq 0$ and $\epsilon$ $>0.$Then, by (2.1) there exists a policy $\pi=\pi\{i, a, t\}$ (depending on $i$ , $a$ , $t$) satisfying that$|g_{\mathrm{i}}(c_{1} +g^{*}t, c_{2}+r(i, a)+d(i, a)t)$ $-g_{\dot{l}}(c_{1}, c_{2})|$$\leqq E_{\pi}$ ($G(c_{1}+g^{*}t+g^{*}\tilde{\tau}_{N}, c_{2}+r(i, a)+d(i, a)t+\overline{D}_{N})$ $-G(c_{1}+g^{*}\tilde{\tau}_{N}, c_{2}+\overline{D}_{N}))+\epsilon$$= \sum_{n=0}^{\infty}P(N=n)E[E[G(c_{1}’+g^{*}\tau_{n},$ $\mathrm{c}_{2}’+r(X_{n-1}, \Delta_{n-1})+d(X_{n-1}, \Delta_{n-1})\tau_{n})$$-G$ ($c_{1}’’+g^{*}\tau_{n}$ , $4+r(X_{n-1}, \Delta_{n-1})$ $+d(X_{n-1}, \Delta_{n-1})7n$) $|N=n$ , $H_{n}]]+\epsilon$ ,where $d_{1}=c_{1}+g^{*}t+g^{*}\tilde{\tau}_{n-1}$ , $4=c_{2}+r(i, a)+d(i, a)t+D_{n-1}$ ,$d_{1}’=c_{1}+g^{*}\overline{\tau}_{n-1}$ , $\phi’=c_{2}+\overline{D}_{n-1}$ ,$H_{n}=(X_{0}, \Delta_{0}, \tau_{1}, \cdots, X_{n})$ , $P:=P_{\pi}(\cdot|X_{0}=i)$ and $E:=E_{\pi}(\cdot|\mathrm{X}_{0}=i)$ .Applying Assumption 3, we have that$|g_{\mathrm{i}}(c_{1} +g^{*}t, c_{2}+r(i, a)+d(i, a)t)$ $-$ 9G(ci, $c_{2}$ ) $|$$\leqq\sum_{n=0}^{\infty}P(N=n)E[E[(G(d_{1}’)-G(d_{1}’, c_{2}’))|$ $N=n$, $H_{n-1}]+2K]+\epsilon$.$\cdot$.(repeating the same discussion)$\leqq\sum_{n=0}^{\infty}P(N=n)(G(c_{1}+g^{*}t, c_{2}+r(i, a)+d(i, a)t)-G(c_{1}, c_{2})+2nK)+\epsilon$$=G(c_{1}+g^{*}t, c_{2}+r(\mathrm{i}, a)+d(i, a)t)-$ G(ci, $c_{2}$ ) $+2K \sum_{n=0}^{\infty}nP(N=n)+e$Since $\sum_{n=0}^{\infty}nP(N=n)=\sum_{n=0}^{\infty}P(N\mathit{2}n)\leqq\sum_{n=0}^{\infty}e(n)=\delta_{0}$, by letting $\epsilon$ $arrow 0$ in the above, weget that$|g_{\mathrm{i}}(c_{1} +g^{*}t, c_{2}+r(i, a)+d(i, a.)t)$ $-$ 9G(ci, $c_{2}$ ) $|$(2.4)$\leqq|G(c_{1}+g^{*}t, c_{2}+r(i, a)+d(i, a)t)-$ G(ci, $c_{2}$ ) $|+2K\delta_{0}$ .13Similarly, for any $t$ with $g_{i}(c_{1}, c_{2})-g_{i}(c_{1}+g^{*}t, c_{2}+r(i, a)+d(i, a)t)>0,$ we get (2.4).Therefore, applying Assumption 3 again, we obtain (2.3) with $K^{*}=K(2\delta_{0}+1)$ . $\square$In the same way as the proof of Lemma 2.1, we can prove the following.Lemma 2.2. For any $i\in J,$ and $c_{1}$ , $c_{2}\in R_{+}$ , it holds that(2.5) $g_{i}(c_{1}, c_{2})\geqq G(c_{1}, c_{2})-K\delta_{0}$ .i $ , $md$We denote by $B(R_{+}\cross R_{+})$ the set of all bounded Borel measurable functions on $R_{+}\cross$$R_{+}$ . For any set $h=(h_{i} : i\in J)$ with $h_{i}\in B(R_{+}\cross R_{+})$ , we define $U\{h\}(c_{1}, c_{2}|i, a)$ by$U\{h\}(c_{1}, c_{2}|i, a)$ $= \sum_{j\in J}p_{ij}(a)\int_{0}^{\infty}h_{j}$ ( $c_{1}+ttJ,$ $c_{2}+r(i,$ $a)+$ d(i, $a)t$) $F_{\dot{|}\mathrm{j}}$ ( ctoa)(2.6)$+ \sum_{j\in J_{0}}p_{ij}(a)\int_{0}$”$G(c_{1}+g^{*}t, c_{2}+r(i, a)+d(i, a)t)F_{ij}(dt|a)$for $c_{1}$ , $c_{2}\in R_{+}$ , $i\in J$ and $a\in A.$Obviously, for each $i\in J$ and $a$ $\in$ $A$, $\mathrm{U}\{\mathrm{h}\}$ (c|, } $|i,$ $a$) $\in \mathcal{B}(R_{+}\cross R_{+})$ .Here, we can state one of main results, whose proof is done by a slight modification ofthat of Theorem 3.2 and 3.3 in our preceding paper [6], so that the proof is omitted.Theorem 2.1. (i) The optimal value functions $g_{i}$ , $i\in J$ satisfy the following optimalityequation:(2.7) $g_{i}(c_{1}, c_{2})= \min_{a\in A}U\{g\}(c_{1}, c_{2}|i, a)$for all $i\in J,$ and $c_{1}$ , $c_{2}\in R_{+}$ , where $g=$ $(g_{i} : i\in J)$ .(i) Let $\pi^{*}=(\pi_{0}^{*}, \pi_{1}^{*}, \cdots)\in$ II be any policy satisfying(2.8) $\pi_{n}^{*}(A^{*}(g^{*}\overline{\tau}_{n},\overline{D}_{n} : X_{n})|H_{n})=1$ on $\{X_{n}\in J\}$for all $n\geqq 1$ and $H_{n}$ , where$A^{*}(c_{1}, c_{2} : i)= \arg\min_{a\in A}U\{g\}(c_{1}, c_{2}|i, a)$for $c_{1}$ , $c_{2}\in R_{+}and$ $i\in J$ . Tien, $\pi^{*}is$ regret-Optimal with a target $g^{*}$ .The following theorem asserts the uniqueness of solution to the optimality equation(2.7). Theorem 2.2. There exists a unique solution to the optimality equation (2.7) in$C$,where$C=\{h=$ $(h_{i} : i\in J)|h_{i}\in B(R_{+}\cross R_{+})$ for all $i\in J$and $h$ satisfies the statement of Lemma 2.1.}c_{2}\in R_{+}$ an .$ $\pi^{*}$ is. )14Proof. Let $h=$ $(h_{i} : i\in J)$ , $h’=(h_{i}’ : i\in J)$ be solutions to (2.7) and $h$ , $h’\in C.$Then, from (2.6) and (2.7), there is an $\overline{a}$ c- $A$ such that$|/\mathrm{z}i(c_{1}, c_{2})-h_{i}’(c_{1}, c_{2})$ $|$$\leq 5p_{ij}(\overline{a})|\int_{0}^{\infty}h_{j}$ ( $c_{1}+g^{*}t,$ $c_{2}+r(i,$ $a)+$ r(i, $a)t$ ) $F_{ij}(dt|\overline{a})$$\leq\sum_{j\in J}p_{ij}(\overline{a})|\int_{0}^{\infty}h_{j}(c_{1}+g^{*}t, c_{2}+r(i, a)+d(i, a)t)F_{ij}(dt|\overline{a})$(2.9)$- \int_{0}^{\infty}h$(($c_{1}+g^{*}t,$ $c_{2}+r(i,$ $a)+$ r(i, $a)t$) $F_{ij}(dt|\overline{a})$$\leqq\sum_{j\in J}p_{ij}(\overline{a})(|h_{j}(c_{1}, c_{2})-h_{j}’(c_{1}, c_{2})|+2\overline{K}$)for some $K>0.$Repeating the relation (2.9), we get that$|hi(c_{1}, c_{2})$ $-h_{\dot{6}}’(c_{1}, c_{2})| \leqq 2\overline{K}\sum_{n=0}^{\infty}e(n)=2\overline{K}\delta_{0}<\infty$ .So, if we put $||h_{i}-h_{i}’||= \sup_{+c_{1},c_{2}\in R}|1_{i}(c_{1}, c_{2})-h_{i}’(c_{1}, c_{2})|$, then $||h_{i}$ $-h_{i}’||\leqq 2\overline{K}\delta_{0}$ , andfrom the first inequality in (2.9), we get(2.10) $||h_{i}$ $-hs|| \leqq\sum_{j\in J}p_{ij}(\overline{a})||h_{j}-h\mathit{3}||$ for $i\in J.$hh_{i}-h_{i}’||\leqq 2\overline{K}\delta_{0}$$||h_{i}-h_{i}’|| \leqq\sum_{j\in J}p_{ij}(\overline{a})||h_{j}-h_{j}’||$Repeating (2.10), we obtain(2.11) $||h-h’||\leqq e(n)||h-h’||$ for all $n\geqq 1,$where $||h-h’||= \sup_{i\in J}||h_{i}$ $-h\mathrm{S}||$ . Letting $narrow\infty$ and noting that $e(n)arrow 0$ ffom Assumption 2, (2.11) means $||h-h’||=0.$ Thus, $h=h’$ , so that uniqueness of solutions follows.3. ExamplesIn the following examples, the results in the preceding section are applied to the casesof some types of regret-utility functions.Example 1. Consider the case that $G(x, y)=x-y.$ Prom Remark in Section 2, weobserve that Assumption 3 holds. Putting$g_{i}=$ ini $E_{\pi}(g^{*}\overline{\tau}_{N}-D_{N}| X_{0} =i)$ ,We get from (2.1) that$g_{\mathrm{i}}(c_{\mathrm{b}}c_{2})$ $= \inf_{\pi\in\Pi}E_{\pi}(c_{1}+g^{*}\tilde{\tau}_{N}-c_{2}-\overline{D}_{N}|X_{0}=i)$(3.1)$=c_{1}-c_{2}+g_{i}$ $(i\in J, c_{1}, c_{2}\in R_{+})$ .g_{i}(c_{1}, c_{2})$ $ \inf_{\pi\in\Pi}E_{\pi}(c_{1}+g^{*}\tilde{\tau}_{N}-c_{2}-D_{N}|X_{0}=i)$15Thus, the optimality equation (2.7) becomes:(3.2) $g_{i}= \min_{a\in A}\{-R(i, a)+\sum_{j\in J}p_{ij}(a)g_{j}+g^{*}\overline{\tau}(i, a)\}$for $i\in J,$ where $R(i, a)=r(i, a)+$ d(i, $a$) $\overline{\tau}(i, a)$ and$\overline{\tau}(i, a)$ $= \sum_{j\in S}p_{ij}(a)\int_{0}^{\infty}tF_{ij}(dt|a)$ for $i\in J$ and $a\in A.$Applying Theorem 2.1, we can obtain a regret-Optimal policy using the unique solutionof (3.2).Remark. We consider recurrent Semi-MDP’s and put:$J_{0}=\{0\}$ , $N= \min\{n|X_{n}=0, n\geqq 1\}$ and$g^{*}=$ s\pi u\in n $\frac{E_{\pi}(D_{N}|X_{0}=0)}{E_{\pi}(N|X_{0}=0)}$ .Then, (3.2) with $g_{0}--0$ is corresponding to the optimality equation for the average case.In fact, it holds (cf. [8, 10]) that$\min_{a\in A}\{-R(0, a)+\sum_{j\neq 0}p_{0j}(a)g_{j}+g^{*}\overline{\tau}(0, a)\}=0,$so that putting $!70=0$ , (3.2) holds for all $i\in S.$Example 2. Consider the case of the exponential type: $G(x, y)=-e^{-\lambda(x-y)}$ , $(\lambda>0$$)$ . When the target value $g^{*}$ is sufficiently large such that $g^{*}t-r(i, a)-d(i, a)t\mathit{2}$ $0$ forall $t\mathit{2}$ $0$ , $i\in S$ and $a\in A,$ Assumption 3 in Section 2 holds obviously.Let$g_{i}= \inf_{\pi\in\Pi}E_{\pi}(-e^{-\lambda(g^{*}\overline{\tau}_{N}-\overline{D}_{N})}|X_{0}=i)$ ,for $i\in J.$ Then, $g_{i}(c_{1}, c_{2})=e^{-\lambda(c_{1}-c_{2})}g_{i}$ , so that the optimality equation (2.7) comes to$g_{i}= \min_{a\in A}[\sum_{j\in J}p_{ij}(a)R(i, a, j)g_{j}-\sum_{j\in J_{0}}p_{ij}(a)R(i, a, j)]$where $R(i, a, j)$ $= \int_{0}^{\infty}e^{-\lambda(g^{*}t-r(i,a)-d(i,a)t)}F_{ij}(dt|a)$ for $i\in J$, $j\in S$ and $a\in A$Applying Theorem 2.1, we get a regret-Optimal policy for the exponential regret-utilitycase.References[1 ] Chung, K. J. and Sobel, M. J. (1987), Discounted MDP’s: Distribution functionsand exponential utility maximization. SIAM J. Control Optimization 25, 49-62.16[2 ] Denaxdo, $\mathrm{E}.\mathrm{V}$ . and Rothblum, U. G. (1979), Optimal stopping, exponential utilityand linear programming. Math. Prog. 16, 228-244.[3 ] Fishburn, P. C. (1970), Utility Theory for Decision Making. John Wiley & Sons,New York.[4 ] Hinderer, K. and Waldmann, K. H. (2003), The critical discount factor for finiteMarkovian decision processes with an absorbing set. Math. Mech. Oper. ${\rm Res}$ . 57,1-19.[5 ] Howard, R. S. and Matheson, J. E. (1972), Risk-sensitive Markov decision processes.Management Science 8, 356-369.[6 ] Kadota, Y., Kurano, M. and Yasuda, M. (1995), Discounted Markov decision prO-cesses with general utility functions. In Proceeding of APORS’ 94, 330-337, WorldScientific.[7 ] Kadota, Y., Kurano, M. and Yasuda, M. (1998), On the general utility of discountedMarkov decision processes. Int. Tmns. Oper. ${\rm Res}$ . 5, 27-34.[8 ] Lippman, S. A. (1971), Maximal average reward policies for Semi-Markov decisionprocesses with arbitrary state and action space. Ann. Math. Statist 42, 1717-1726.[9 ] Pratt, J. W. (1964), Risk aversion in the small and in the large. Econornetrica 32,122-136.[10 ] Ross, S. M. (1970), Applied Probability Models with Optimization Applications.Holden-Day.

Regret-Optimality Equation in Semi-MDP's with an Absorbing Set (Mathematical Programming Concerning Decision Makings and Uncertainties)

http://repository.kulib.kyoto-u.ac.jp/dspace/bitstream/2433/25520/1/1373-2.pdf

Regret-Optimality Equation in Semi-MDP's with an Absorbing Set (Mathematical Programming Concerning Decision Makings and Uncertainties)

Abstract

Similar works

Full text

Available Versions

Kyoto University Research Information Repository