大野, 勝久

Kyoto University Research Information Repository

Titleニューロ・ダイナミックプログラミングとその応用(最適化問題における確率モデルの展開と応用)Author(s)大野, 勝久Citation数理解析研究所講究録 (2007), 1559: 139-155Issue Date2007-06URL http://hdl.handle.net/2433/81045RightType Departmental Bulletin PaperTextversionpublisherKyoto University\cdot(Katsuhisa Ohno)Faculty of Management and Information Science,Aichi Institute ofTechnology1(Dynamic Programming,DP )(Markov Decision Process, MDP) ,, Bellman 1950, (PolicyIteration Method, PIM), (ValueIteration Method, $M$)$Howard[1]$ , Puterman[2],(Modifled Policy Iteration Method,MP-I-M) ,[3-5] ,, PIM[6] , MDP DP ,,, [7] ,(Reinforcement $Leaming$)$[8,9]$,(Neuro- Dynamic Program-ming, DP) $[10,11]$DP MDP, , ,$=$, MDP,Das et $a1.[12]$ $Gosavi[13]$ , $h/DP$$r*$ (Semi-Markov Decision Process, SMDP)SMART, RELAXED- SMART,He et a1.[14] , PIM $K$$n$ SBPI,Gosavi et al [151 ,SMART (TemporalDifference, TD) $\lambda$–SMART,,, Das[16] Chang et a1.[17],,, [18]==– DP SBMPIM, ,, 2 ,8%, SBMPIM 3, $n$,SBMPIM[19,2011559 2007 139-155 139, [21],$[22,23]$ , $CONWIP[24]$ ,[25], Sutton andBarto[8](Undiscounted MarkovDecision Process, UMDP) , 3$==$ - DPSMART[12,131 $SBPI[14]$4 MPIM DPSBMPIM[19,201 , 5,$==.-$ DP6 , $[19,20]$[26,27,28], SBMPIM2. UMDP, [8] ,,?,,,1. (environment model) ,2.(Learning agent)3. (action) ,(policy)4. (reward ffinction) ,(environmental response)5. (value ffinction),( ),,,( ),,,,,,,,,,,,,( ), ,,,,,,,, $n$$s_{n}$ $a_{n}$ , $n$$r_{n}$ $n+l$ $s_{n+1}$MDP,$h_{n}$ : $n$ (So, $a_{0},r_{0},\cdots,s_{n}$ )140,$p(s,s^{1},a)=P\{s_{n+1}=s’|h_{n-1},s_{n}=s,a_{n}=a\}$$=P\{s_{n+1}=s^{t}|s_{n}=s,a_{n}=a\}$$r(s,a)=E \{r_{n}|h_{n-1},s_{n}=s,a_{n}=E\{r_{n}|s_{n}=s,a_{n}=a\}=\sum_{s^{1}\in S}^{=a\}}p(s,s^{1},a)r(s,s^{1},a)$,,$g$ 1 , $h(s)$,,$g+h(s)= \min_{\in K(s)}\{r(s,.)+\sum_{s’\in S}p(s,s’,a)h(s^{1})\},s\in s(1)$, $S$ ,$K(s)$ : $s$, $s$ (1), $h(s)$$S_{r}$ $h(s_{r})=0$ [1,213. SMART SBPI$==-$ DP$SM\bm{T}T[12]$ SMART ,TD , $g$$Q$-, $Q$- $Q$ ,,$Q$ ,,, $Q$,3 , SMART SMDP,MDP[SMART] [12]1: $s\in S$ $a\in K(s)$$Q- factorQ_{n\alpha\nu}(s,a)=Q_{0}u(s,a)=0$ ,$TC=0$ , 10,$g=0$ , $k=0$ ,2$(\alpha_{0},\alpha_{\tau},p_{0},p_{\tau})$2: $k$ $s$ ,$\alpha_{k}$ , $P_{k}$$\alpha_{k}=\alpha_{0}(\alpha_{\tau}+k)/(k^{2}+k+\alpha_{\tau})$ ,$p_{k}=p_{0}(p_{r}+k)/(k^{2}+k+p_{\tau})$3: $(1-p_{k})$ $Q_{n\cdot w}(s,a)$a , $P_{k}$a $K(s)$ a4: a, $s’$ ,$r(s,s’,a)$5: $Q_{n\cdot w}(s,a)$$Q_{n\cdot\nu}(s,a)=(1-\alpha_{k}p_{0}u(s,a)$$+ \alpha_{k}\{r(s,s’,a)-g+’\min_{\in K(-)},Q_{old}(s’,a’)\}$6: 3 a, $TC$ $g$$TC=TC+r(s,s’,a^{*})$$T=T+1$$g=TC/T$7: $Q_{0}u(s,a)=Q_{n\cdot,\nu}(s,a)$8: $k$$k=k+1,$ $s’$ $s$2$Gosavi[13]$ SMART,RELAXED-SMART[RELAXED-SMART] [13]$3\sim 5,7,8$ [SMART]X $\taurightarrow$ ‘ ‘ 1 : Q-factor $Q_{\sim\nu}(s,a)$$=Q_{old}(s,a)=0$ , $IC=0$ , $T=0$ , $g=0$ ,$k=0$ , $\alpha_{0}$ , $P_{0}$ $\beta_{0}$1412: $k$ $s$ , $\alpha_{k}$ ,$p_{k}$ , $\beta_{k}$$\alpha_{k}=\alpha_{0}/k$ , $p_{k}=p_{0}/k,$ $\beta_{k}=\beta_{0}/k$6: 3 a, $TC,$ $T,$ $g$$TC=(1-\beta_{k})TC+\beta_{k}r(s,s’,a^{*})$$T=(1-\beta_{k})T+\beta_{k}$$g=TC/T$, He et al [14] PIM$=$SBPI (Simulation Based Policy Iteration)[SBPI ] [14]1: $\{f^{0}(s);s\in S\}$ ,$k=0$2:( )2-a: ( $g^{k}$ )i) $s_{0}$$S_{1},\cdots,S_{m}$ii) $g^{k}=0$ , $n=0,\cdots,m-1$$(s_{n},s_{n+1})$ $g^{k}$$g^{k}=(1-\iota/(n+1))_{g^{k}+}(1/(n+\iota))’(s_{n},s_{n+\iota f^{k}())}s_{n}$2-b : ( $h^{k}(s)$ )i) 2-a$s$ii) $s_{0}$ , $S$$I$$L$iii) 1$(s_{0},s_{1},\cdots,s_{N}=s),$ $l=1,\cdots,L$ , ,$(s_{n},s_{n+1})$ $\iota\nu(s_{i})$ , $i=1,\cdots,n$ ,$w(s_{i})=w(s_{i})+\gamma_{i}\lambda^{n-i}d_{n}$,$s_{j}$ ,$0\leq\lambda\leq 1$ ,$d_{n^{=\gamma}}(S_{n},S_{\hslash+\iota,f^{k}())_{-g^{k}+w(s_{n+1})-w(}s_{n})}s_{n}$iv) $h^{k}(s)=w(s)-w(s_{r})$ , $s\in S$3:( )$f^{k+1}(S)=\arg\epsilon m_{b}Kil\{s,s’,as’’ s\in S$4: $f^{k+1}(s)=f^{k}(s),$ $s\in S$$f^{k}(s)$$k=k+1$ 24. MPIM SBMPIM(1)(PIM) ,(MPIM) [1-5] , MPIMPIM,[MPIM] [5]1: $h_{0}(s_{r})=0$$h^{0}$ , $m$ , $f^{0}$ ,$\epsilon$ , $k=0$2:( )$s_{n}\in S$ ,$g^{k+1}(s)= mi\in KP_{s)}\{r(s, a)+\sum_{s’\in S}p(s,s’,n)h^{k}(s’)-h^{k}(s)\}$, $f^{k}(s_{n})$ $g^{k+1}(s)$ ,$f^{k+1}(s)=f^{k}(s)$ , ,$g^{k+1}(s)$ $f^{k+1}(s)$3:( )$w^{0}(s_{n})=h^{k}(s_{n})+g^{k+1}(s_{n})$ , $s_{n}\in S$, $l=0,1,\cdots m-1$ ,$w^{l+1}(s)=r(s,f^{k+1}(s))$$+ \sum_{s’\in S}p(s,$ $s’,$ $f^{k+1}(s))w^{l}(s’),$ $s\in S$,$h^{k+1}(s)=w^{m}(s)-w^{m}(s_{r})$ , $s\in S$$s$ ,142$|h^{k+1}(s_{n})-h^{k}(s_{n})|<\epsilon$ $s=s’$, $k=k+1$ , 2 , $l=m$ 4$\text{ _{}o}$ $l=l+1$ 3, 4:($g$ ) 1, $S$ $g(k)$ $g$$g(k)=TC/m$ ,, $g=(qg+g(k))/(q+1)$,$s_{0}$ , 5:( $h(s)$ ) $S_{T}$$n$ , $v(s)$ \mbox{\boldmath $\lambda$}D $s$ $s$ ,$r$$s$ $h(s)=(S_{\gamma}m\iota\nu()+k^{k_{\mathcal{V}}}ts_{r})/m)s_{r},s_{r}$$r$$==-$ DP $L,$ , $s(\neq s_{r})\in S_{v}$SBMPIM (Simulation-Based $h(S)=(SmS(sm\gamma(s,f())-g-h(S_{\gamma})$Modified Policy Iteration Method)$,$$h(s_{r})=0$ , $k=1$1: $s_{0}$[SBMPIM] [19]$hh\{$$S$ , $S_{0}$$s_{r})=r(s_{r},f(s_{r}))-g$ ,$s)=r(s,f(s))-g-h(s_{r})$, $r$ $m$ , $\lambda$ 6: ( ) $s\in S$$(\lambda\leq 1)$ $Q$$v$$\epsilon>0$ ,$S_{\nu}=\phi$ ( ), $S_{T}=\phi$ , $TC=0$ , $w(s)= \min_{s\in N(s.f())}\{r(s,a)+\sum_{s’\in S}p(s,s’,a)h(s’)\}$$s=s_{0}$ , $k=l=1$ , $q=0$2 :(Schweitzer [29]) $N(s,f(s))$ $K(s)$$\tau$$f(s)$ , $p(s,s’,a)>0$$0<\tau<$ $\min$ $\{1/(1-p(s,s,a))\}$ $s’\not\in S_{v}$ , $S_{v}=S_{v}\cup\{s’\}$$sp\in(S\cdot,\in\cdot)K<\{s\rangle$ , $f(s’)$ s, $r(s,a)$ , $w(s’)=r(s’,f(s’))$ ,$p(s,s’,a)$ $h(s’)=r(s’,f(s’))-r(s_{r},f(s_{r}))$$r(s,a)arrow\tau r(s,a/$,, $w(s)$ $f(s)$ $w(s)$$p(s,s’,a)arrow\tau pss’,a)+(1-\tau)\delta_{s.s^{\iota}}$ , $w(s)$$\delta_{s.s’}=1,$ $s=s’$ ; $=0,$ $s\neq s’$ $f(s)$ $s\in S_{0}$$f(s)$ $q=q+1$3: $s\not\in S_{v}$ , $S_{v}=S_{v}\cup\{s\}$ , , $q=0$ $q\geq Q$$S_{T}=S_{T}\cup\{s\},$ $s$ $v(s)=1$ , $\{g(k-q)/\tau,\cdots,g(k)/\tau\}$, $f(s)$ s $S^{2}$ $g/\tau$ , $q$$s\in S_{v}$ , $s\not\in S_{T}$ , $t$ $\alpha$ $t_{a}(q)$$S_{T}=S_{T}\cup\{s\}$ , $v(s)=1$ , $s\in S_{T}$ ,, $t_{a}(q)S/\sqrt{q+1}<\mathcal{E}$$v(s)=v(s)+1$$g$$s$ $f(s)$ IOO(I-\alpha )%$f$ , $[g/\tau-r_{a}(q)S/\sqrt{q+1},$ $g/\tau+r_{a}(q)S/\sqrt{q+1}]$s’ , $\{f(s),s\in S_{v}\}$$TC=IC+r(s,f(s))$$S_{T}=\phi,$ $TC=0$ ,143$1=1$ , $k=k+1$ 3 $\circ$5. $==$ - DP3, 4 $=n$DP, 3 ,,1, $M$$i$ ,$i=1,\cdots,M$ , , ,, $T_{i}$$L_{j}$ $(>T_{i})$,,$j$$I_{\max:\dot{\iota}}$ ,$J_{m\}c:\dot{\iota}}$ ’ $C_{j}$, C. ,$n,$ $n=1,2,\cdots$ , $C_{j}(n)$ ,, $C_{i:\min}$ ,$n$ $D(n)$ ,, $D_{\dot{m}n}$ ’ $D_{ma}$,$D$,$B_{1nax}$,$i$ , $n$$I_{j}(n)$ $J_{i}(n)$,, $O_{i}(n)$ ,$P_{i}(n)$ $J_{j}(n)$$(i+1)$ ( ), $n$ $i-1$$j$ $Q_{i}(n)$,,,$i=1,\cdots,M$$C_{i}^{J}$ : $i$/$C_{i}^{J}$ : $i+1$$j$ /$C_{i}^{B}$ : $i$/$B_{i}$ : $i$/, $B_{ur}$$C_{In*x}/$MPIM, $i=1,\cdots,M$$L_{j}=1$ , $I_{m\cdot x:i}=I_{m*x}$ , Jmu\iota j=J\sim, $S$$(I_{m}$ $+1)^{M}(J_{mr(}+I_{R}+1)^{M- 1}(J_{Inu}+B_{1t1\iota x}+1)$(2), $I_{mu}=J_{m*x}=B_{m\cdot x}=9$$190^{M}$$M=1$, (2)1000 , 4 MPIM,, 3,4 $==-$DP SBMPIM, SMART,RELAXED-SMART, SBPI$M=L=1$ , $I_{\iota I]\epsilon x}=8$ , $J_{n1*x}=5$ , $B_{1t1*x}=0$ ,$C_{m*x}=100$ , $C=5$ , $C_{\dot{r}n}=3$ , $C^{J}=1$ ,$C^{J}=2$ , $C^{B}=5$ , $B=10$, 144,$P(C(n)=c)=P_{c}$ , $C_{\dot{m}n}\leq c\leq C$$P_{s}=0.7$ , $P_{4}=0.2$ , $P_{3}=0.1$, $D(n),n=1,2,\ldots$ ,144$Pr\{D(n)=D-\frac{1}{2}Q+j\}=(\begin{array}{l}Qj\end{array})(\frac{1}{2})^{Q},0\leq j\leq Q$(3), $D$ ,$Q$ (Q\leq 2D) , $D$ ,$Q/4$, $D=3,$ $Q=2$,MPIM $m=20,$ $\epsilon=10^{-5}$ ,SBMPIM $m=10^{3}$ , $\lambda=0.15$ ,$s_{0}=(0,-10),$ $s=(8,5)$ , $=10^{5}$ ,5 $N(s,f(s))=K(s)$ , SMART$(\alpha_{0},\alpha_{\tau},p_{0},p_{\tau})$$=(0.1,5x10^{5},0.1,5x10^{5})$ , $=$$5x10^{5}$ , RELAXED-SMART$(\alpha_{0},p_{0},\beta_{0})=(0.1,0.1,0.9)$ , 5 $x10^{5}$ ,SBPI $s_{0}=(4,0)$ , $m=10^{3}$ ,$\lambda=0.9$ , Gosavi [15] ,SMTT $Q(s,a)$ SBPI$s$ $s$, $\lambda$ -SMART$\lambda$ -SMART$s_{0}=(3,-10)$ , $=$$=5$ x105 , $(\alpha_{0},\alpha_{\tau},p_{0},p_{\tau},\lambda)=(0.9,5\cross 10^{5}$ ,0.99, $5\cross 10^{5},0.9$),$g$ 1 MPIM5DOSN $\Re(CPU:Pentium42.4GHz$ , $$:2GB$) ,ExcelVBA1 , SBMPIM $==$ - DP, MPIM, $\text{ _{}\wedge}-$ DP, 123 , MPIMMPIM ,1 , 2,* $==$ - DP3 $0$SBMPIM $–=$ - DPJIT ,,$=$.$n$ ,([30], 9 ), $10^{4}$20,7 , 4 ,, $6.403\pm 0.036$1 $==.-$ DP, MPIM ,7% ,$(3, 0)$ , $(4,0)$, $(3, 1)$ 4, 3, $(3, 2)$4, 2, (4, -1)3, 4, $(4, 1)$ 3,3 $(4, - 1)$6.3 ,$=$., SBMPIM,, , ,CONWIP, ,5 ,1 $\ovalbox{\tt\small REJECT}$ ,,1451) [21] $N_{l}$$i,$ $i=1,2,3$ ,$M_{i}$ , $N_{i}$$I_{1}(1)=0$ $J_{1}(1)=S- \sum_{l=2}^{3}(I_{i}(l)+J_{l}(1))$ , $I_{i}(1)=M_{j}$ ,, $I_{l}(1)=M_{j},$ $J_{j}(1)=N_{j}$ $J_{i}(1)=N_{j},$ $i=2,3$, ,$O_{1}(n)=D(n-1)$ , $P_{1}(n)= \min\{I_{1}(n),C_{1}\}$$O_{i}(n)=M_{j}-I_{i}(n)-[-J_{i-1}(n)]^{+}- \sum_{l^{--}0}^{T_{l}- 1}Q_{j}(n-l)$$o_{i}(n)=M_{j}-I_{j}(n)-[-J_{j- 1}(n) r-\sum_{l=0}Q_{l}(n-l)T-]$$P_{j}(n)= \min\{-,\}$ $P_{i}(n)= \min\{N_{i}-[J_{i}(n)r,I_{j}(n)c_{i}\},$ $i=2,3$, (4)-(9)$I_{l}(n+1)=I,(n)+Q_{l}(n-T_{l}+\iota)-P_{l}\langle n$) (4)$J,(n+1)=J_{l}(n)+P_{l}’(n)-o_{+1}(n-L_{l*1}+T_{l*1}+\iota),$ $i=1,2$ 5) [26-28](5) $i,$ $i=1,2,3$$J_{3}(n+\iota)=maxt^{r_{3}}(n)+P_{3}’(n)-D(n),-B_{mu}\}$ (6)$S_{i}$ , $M_{j}$,,$P_{i}’(n)= \min\{P_{i}(n),C_{i}(n)\}$ (7)$Q_{1}(n)=o_{1}(n-L_{1}+\tau_{1})$ (8)$I_{j}(1)=M_{i}$ , $J_{j}(1)=S_{i}$$Q_{l}(n)=\dot{m}n\{o_{i}(n-L, +r_{l})+1^{-J_{i- 1}}(n-\iota)\Gamma$ , $n=1,2,\cdots$$P_{-1}’(n-\iota)+[J_{l- 1}(n-\iota)r$ }, $i=2,3$ (9) $o_{i}(n)= \min\{n-1,nnn-l$2) $[22,23]$ $P,(n)=m\dot{\bm{o}}\{S_{l}-[J_{l}(n)]^{+},I,(n\rangle,c_{i}\}$$i,$ $i=1,2,3$ , (4)–(9)$S_{i}$ ,$I_{j}(1)=0$ , $J_{l}(1)=S_{t}$ 3, $n=1,2,\cdots$$O_{j}(n)=D(n-1)$ , , SBMPIM$P_{i}(n \ovalbox{\tt\small REJECT}\min\{S_{j}-[J_{l}(n)r,I_{j}(n),c_{i}\}$$i=1,2,3$$D(0)=0$(4)-(9) $L_{j}=1,$ $T_{i}=0,$ $I_{Ina)\iota:i}=8,$ $J_{ma’(:i}=5,$ $B_{I]\cdot x}=4$3) CONWIP[24] , (2) 143$iP$ $S$$S= \sum_{i=1}^{3}(I_{j}(1)+J_{i}(1))$ , \langle , (3),, $n=1,2,\cdots$$D=Q=2$ 2, 0.5$O_{1}(n)=D(n-1),$ $O_{i}(n)=J_{i- 1}(n),$ $i=2,3$ , ,$P_{i}(n)= \min\{I_{i}(n),C_{i}\}$ , $i=1,2,3$ $A:P_{5}=1$ ( ), $=5$(4)–(9) $B$ : $P_{5}=0.7$ , $P_{4}=0.2$ , $P_{3}=0.1$ ,$=4.6$4) [25] $C:P=0.4,$ $P=0.3,$ $P=0.2$ $P-0.1$WP $S$, $i=2,3$ 5 4 3$-$2$=4.0$, $M_{i}$ , 3 ,146, , 5), 2), 1), 4), 3)(a)$(C_{1}^{J},C_{2}^{I},C_{2}^{l})=(1,3,6)$ , ,$(C_{1}^{J},C_{2}^{J},C_{3}^{J})=(3,6,12)$ , (a) SBMPIM$(C_{1}^{B},C_{2}^{B},C_{3}^{B})=(20,40,80)$ , , $A\sim c$$(B_{1},B_{2},B_{3})=(40,80,120)$ , $C_{mv(}=1000$ 24.13%, 1862%,5.3O% [31](b) , $bRP$ , , TOC,$(C_{1}^{l},C_{2}^{l},C_{3}^{I})=(1,2,4)$ , CONWIP$(C_{1}^{J},C_{2}^{J},C_{3}^{J})=(2,4,6)$ , ,$(C_{1}^{B},C_{2}^{B},C_{3}^{B})=(10,10,30)$ , bfflP,$(B_{1},B_{2},B_{3})=(20,20,60)$ , $C_{\max}=100$, WP2, =. (throughput), SBMPM ,$s=(3,3,3,3,3,5)$ , $s=(8,5,8,5,8,5)$ , ,$0$ ,5$m=10$ , $\lambda=0.99$ , $\tau=0.99$ , $Q=20$ , , 4-(a) (b) ,$\epsilon=0.1$, ,4-(a) (b) SBMPIM$\backslash \backslash$$\backslash$$\sim\backslash \backslash$ (Supply Chain,Management, SCM)$=$. ,,B , ,$C_{ma\}\iota}$ , 1 ,$L_{3}=2,$ $T_{3}=1$, $C_{1}^{B}=C_{2}^{B}=B_{1}=B_{2}=0$ $D=Q=2$ , A,, $(C_{1}^{I},C_{2}^{I},C_{3}^{l})=(1,3,6)$$(C^{J}C^{J}C^{J})=(3,6,12)$, 5 $=$ 1 ’ 2 3$(C_{1}^{B},C_{2}^{B},C_{3}^{B})=(0,0,80)$ ,$(B_{1},B_{2},B_{3})=(0,0,120)$ ,4-(a) (b) $C_{m*]c}=1000$, SBMPIM 2(%) , $i=1,2$( ,SBMPIM )/ $L_{t}=1$ , $T_{i}=0,$ $I_{maJ\dot{g}}=6,$ $J.=6$$m\cdot$(SBMPM ) (%) ,$L_{3}=2$ , $T_{3}=1,$ $I_{mv\iota 3}=10$ , $J_{m*]i3}=10$ ,$($ $)$ 4 $B_{m*\}}=5$147, ii-3) $s\in S_{v}$ ,, $s\not\in S_{T}$ , $S_{T}=S_{T}\cup\{s\}$ , $v(s)=1$(2) , 2096 , $s\in S_{T}$ , $v(s)=v(s)+1$, SBMPIM DOSN(OS:Windows XP $x64$, CPU: iii) $l=m$ 4Core 2 Duo 2. $66GHz$ , :4GB) $l=l+1$ $3- i)^{\text{ }}$, 4 : ($g$ )$S_{T}$ $\nu(s)$ $s$$s_{r}$ , 1SBMPIM Ver.2 $g(k)$ $g$[SBMPIM Ver.2] $g(k)=TC/m$ ,1:( ) $g=(qg+g(k))/(q+1)$s0 $s$ ,$=$ $m$ , $\lambda,$ $\mu$ $\overline{\mathcal{T}}^{\backslash }\backslash$ 5 : $(h(s)$ $ffl^{\wedge}oe)$$(\lambda,\mu\leq 1)$ $Q$$h(s_{0})=(s_{0}ms_{0}\iota^{k_{\mathcal{V}}}()/mks_{0},f(s_{0}))-g$$\epsilon,\epsilon^{1}>0$ ,$S_{\nu}=\phi$ ( ), $S_{T}=\phi$ , $S_{U}=\phi$ , i) $s(\neq s_{0})\in S_{T}$$TC=0$ , $s=s_{0}$ , $k=l=1,$ $q=0$ $h(s)=(smws\lambda^{k_{\mathcal{V}}}()/m)S,S$, $f(s_{0})$ sii) $s\in(S_{v}\cup Sarrow-S_{T}\ovalbox{\tt\small REJECT}’\cdot$2:(Schweitzer [29]) $h(s)=w(s)-g-h(s_{0})$$\tau$$0<\tau<$ min $\{1/(1-p(s,s,a))\}$ iii) $h(s_{0})=0$$t\in s,\cdot\cdot\in Kp(ss,.)<t^{s)}$6:( ), $r(s,a)$ ,$)s\in$$i$ $S$$p(s,s’,a)$$L$$v$$r(s,a)arrow\tau r(s,a)$$w( s)=\min_{s\in N(s,f())}\{r(s,a)+\sum_{s’\in S}p(s,s’,a)h(s’)\}$$p(s,s’,a)arrow\tau p(s,s’,a)+(1-\tau)\delta_{s,s’}$$\delta_{ss},$ $=1,$ $s=s’$ ; $=0,$ $s\neq s’$ $N(s,f(s))$ $K(s)$3:( $n$ ) $f(s)$ , $p(s,s’,a)>0$i) $s$ $f(s)$ $s’\not\in S_{v}\cup S_{U}$ ,$=$ , $S_{U}=S_{U}\cup\{s^{I}\}$$s’$ ,$TC=TC+r(s,f(s))$ $w(s’)=r(s’,f(s$$s=s’$ $h(s’)=r(s’,f(s$$w(s)$ii-l) $s\not\in S_{V}$ $s\not\in S_{U}$ , ,$S_{v}=S_{v}\cup\{s\},$ $S_{T}=S_{T}\cup\{s\},$ $s$ $f(s)$$v(s)=1$ , $f(s)$ s ii) $s\in S_{U}$, $w(s)=r(s,f(s))$$w( s)=\min_{\in N(s,f(s))}\{$ii-2) $S\not\in S_{V}$ $s\in S_{U}$ ,$S_{v}=S_{v}\cup\{s\}$ , $S_{U}=S_{U}-\{s\}$ ,$S_{T}=S_{T}\cup\{s\},$ $s$ $v(s)=1$ $f(s)$, $f(s’)$ s,$’))$ ,$’))-r(s_{0},f(s_{0}))$$f(s)$ $w(s)$$w(s)$$r(s, a)+\sum_{s’\in S}p(s,s’,a)h(s’)\}$$N(s,f(s))$ $K(s)$, $p(s,s’,a)>0$148$s’\not\in S_{v}\cup S_{U}$ ,$h(s’)=r(s’,f(s’))-r(s_{0},f(s_{0}))$$w(s)$ $f(s)$ $w(s)$, $w(s)$$f(s)$7:( )i) $|g(k)-g(k-1)|<\mathcal{E}^{1}$ , $k\geq 2$ ,$q=q+1$ , $ii)^{\text{ }}$$q=0$ , $iv)^{\text{ }}$ii) $q=1$ $s_{0}=s_{r}$ ,$iv)^{\text{ _{ }}}$ , $q<Q$ ,iv) , $q\geq Q$ ,$\text{ ^{}p}iii)^{\text{ _{ }}}$iii) $\{g(k-q)/\tau,\cdots,g(k)/\tau\}$ $S^{2}$$g/\tau$ , $q$ $t$$\alpha$ $t_{a}(q)$ ,$t_{a}(q)S/\sqrt{q+1}<\epsilon$$g$lOO(l-a)%$[g/\tau-t_{\alpha}(q)S/\sqrt{q+1},$ $g/\tau+t_{\alpha}(q)S/\sqrt{q+1}]$, $\{f(s), s\in S_{v}\}$iV)iv) $s=s_{0},S_{T}=\phi,$ $TC=0,$ $l=1,$ $k=k+1$3, 1, 3, s $f(s)$, $s=(I_{1},J_{1},I_{2},J_{2},I_{3},J_{3},Q_{3})$ ,$s^{*}=(Ii,Ji,I_{2}^{*},J_{2},I_{3},J_{3},Q_{3})$$O_{i}=[I_{i}-I_{l}]^{+},i=1,2,$ $O_{3}=[I_{3}^{s}-I_{3}-Q_{3}]^{+}$$P_{i}= \min\{I_{l},C,,J\int-J_{i}\},i=1,2,3$[SBMPIM Ver.2], 5 $D=2$,$D=l$ , $D=l,Q=2$7., $==$ - DPSMTT$[12,13]$$SBPI[14]$ ,,$==$ - DPSBMPIM , $[19,20]$,SBMPIM,,,,$==.-\iota\supset$ DP, SBMPIM Ver.2 ,, $=n$,,,(C) 18510137[1] Howard, R. A. : $Dynamic$ Programmingand Markov ProcessesJ , Cambridge, MITPress (1960) ( , , :, (1971))[2] Puterman, M. L.: $Markov$ Decision$Processes\rfloor$ , John Wiley&Sons, (1994)[3] :‘ ”,. , Vol. 29, No. 6, Pp.333-341(1985)[4] Ohno, K. and Ichiki, K.: “Computingoptimal policies for controlled tandemqueueing systems”, $Opera\hslash ons$ Research,Vol. 35, No. 1, Pp.121-126 (1987)[5] Ohno, K.: ‘Modified policy iterationalgorithm with nonoptimality tests forundiscounted Markov decision Process”,149Working Paper, Dept. ofInformation System overbooking”, IIE Transactions, Vol. 34, PP.and Management Science, Konan University, 729-742 (2002)Japan (1985) [161 Gosavi, A.: Simulation-based[6] Ohno, K. , Ishigaki, T. and Yoshii, T. :Optimization : Parametric Optimization“ANew Algorithm for a Multi-item Periodic Techniques and Reinforcement $Leaming\rfloor$ ,Review Inventory System”, ZOR-Math Kluwer Academic (2003)Methods of Oper. Res. Vol. 39, pp. 349-364 [17] Chang, H. S. Fu, M. C., Hu, J. and(1994) Marcus, S. I.: Simulation-based[7] Russell, S. and Norvig, P. ( ): Algorithms for Markov Decision ProcessesJ,, Springer-Verlag (2007)(1997) [181 , , $:”==$ - .[8] Sutton, R. S. and Barto, A. $G$ :$Reinforcement$ Leaming , MIT Press ”,(1998) ( , : , , Vol. 54, No. 5, PP. 316-325(2000)) (2003)[9] , , : “ [19] , : $==-$ .”, / / , Vo145,No.4, pp.186-196 (2001) ”,[10] Bertsekas, D.P. and Tsitsiklis, J.N.: , Vol.55, No.4,$Neuro$-Dynamic ProgrammingJ , Athena pp.174-188 (2004)Scientific (1996) [20] :“[11] Roy, R. V: ‘Neuro-dynamicJI $T’$ ,Proyamming: overview and recent trends,”Pp.431-459, in E. A. Feinberg and A.Schwartz ed. Handbook of Markov , ,Decision Processes $\rfloor$ , Kluwer Academic $pp351- 377(2004)$Publishers (2002) [21] :[12] Das, T. K., Gosavi, A., Mahadevan, S. ,and Marchalleck, N.: “Solving semi-Markov (2006)decision problem using average reward [221 Magee, J. F.: $Produc\downarrow ion$ Planning andreinforcement leaming”, Management $InventoryControl\rfloor,$ $McGraw- Hill(1958)$Science, Vol. 45, No. 4, pp. 560-574 (1999) [23] Clark, A. J. and Scarf, H.: “Optimal[13] Gosavi, A.: Doctor Thesis, policies for multi-echelon inventory$httP://faculty$.uscolo.$edu/gosavi/thesis$.html problem,” Management Science, Vol. 6, PP.(1999) 475-490 (1960)[14] He, $Y$ , Fu, M. C. and Marcus, S. I.: $A$ [24] SPeaman, M. L., Woodruff, D. L. andSimulation- based policy iteration algorithm HoPp, W. J.: ”CONWIP: A pull alternativefor average cost unichain Markov decision to Kanban,” Inter. J Prod Res., Vol. 28, PP.processes, PP. 161-182, in Laguna, M. and 879-894 (1990)Velarde, J. L. $G$ eds., $Computing$ Tools for [25] Bonvik, A. M., Couch, C. E. andModeling, Optimization and $Simulation\rfloor$ , Gershwin, S. B.: “A comparison ofKluwer Academic (2000) production-line control mech- anisms”, Inter.[15] Gosavi, A., Bandla, N. and Das, T. K.: J. Prod. ${\rm Res}.,$ $35,789- 804$ (1997)“A reinforcement learning aPproach to a[26] Buzacott, J. A. and Shanthikumar, I. $G$ :single leg airline revenue management Stochastic Models of Manufacturingproblem with multiple fare classes and $Systems\rfloor$ , NJ, Prentice Hall (1993)150[27] Karaesmen, F. and Dallery, Y.: $A$performance comparison of pull type controlmechanisms for multi-stage manufacturing,”Inter. J Prod. Econ., Vol. 68, pp. 59-71(2000)[28] Zipkin, P. H.: $\lceil$ Foundations ofInventory Management $\rfloor$ , McGraw Hill(2000)[29] Schweitzer, P.J.: “Iterative solution ofthe functional equations of undiscountedMarkov renewal programming,” J MathAnal. APPl. Vol. 34, PP.495-501 (1971)[30] , , , :, (2002)[31] :“ ”,, Vol. 42, No. 7, pp. 546-551(2003)12 $g$151152, 1 * MPIM ,MPIM NDP 1 , 2 ,3 $O$1535 ( $L_{3}=2,$ $T_{3}=1$ )154$\ovalbox{\tt\small REJECT}$$\not\in$$\ovalbox{\tt\small REJECT}$\yenCONWIP. $——–*$1155

ニューロ ダイナミックプログラミング ト ソノ オウヨウ サイテキカ モンダイ ニ オケル カクリツ モデル ノ テンカイ ト オウヨウ

http://repository.kulib.kyoto-u.ac.jp/dspace/bitstream/2433/81045/1/1559-14.pdf

ニューロダイナミックプログラミングトソノオウヨウサイテキカモンダイニオケルカクリツモデルノテンカイトオウヨウ

Abstract

Similar works

Full text

Available Versions

Kyoto University Research Information Repository

ニューロ ダイナミックプログラミング ト ソノ オウヨウ サイテキカ モンダイ ニ オケル カクリツ モデル ノ テンカイ ト オウヨウ

Abstract

Similar works

Full text

Available Versions

Kyoto University Research Information Repository

ニューロダイナミックプログラミングトソノオウヨウサイテキカモンダイニオケルカクリツモデルノテンカイトオウヨウ