Ercetin, Ozgur

Erçetin, Özgür

Gunduz, Deniz

Gündüz, Deniz

Ozfatura, Emre

Salehi Heydar Abad, Mehdi

Özfatura, Emre

English

Content storage at the network edge is a promising solution to mitigate excessive content traffic. To this end, cache-enabled cellular architectures can be utilized to reduce the network cost. However, content storage strategies should be able to take into account user satisfaction, which depends on the age of a dynamic content. The ratio of satisfied users can be increased with more frequent cache refreshment, albeit at an increased network cost. In this paper, we introduce a cache refreshment strategy that strikes a balance between user satisfaction and the infrastructure cost

Sabanci University Research Database

Dynamic Content Updates in HeterogeneousWireless NetworksMehdi Salehi Heydar Abad 1, Emre Ozfatura2, Ozgur Ercetin1, and Deniz Gu¨ndu¨z21Faculty of Engineering and Natural Sciences, Sabanci University2Department of Electrical and Electronic Engineering Imperial College London1{mehdis,oercetin}@sabanciuniv.edu2{m.ozfatura,d.gunduz}@imperial.ac.ukAbstract—Content storage at the network edge is a promisingsolution to mitigate excessive content traffic. To this end, cache-enabled cellular architectures can be utilized to reduce thenetwork cost. However, content storage strategies should be ableto take into account user satisfaction, which depends on the age ofa dynamic content. The ratio of satisfied users can be increasedwith more frequent cache refreshment, albeit at an increasednetwork cost. In this paper, we introduce a cache refreshmentstrategy that strikes a balance between user satisfaction and theinfrastructure cost.Index Terms—Content caching, content refreshment, Markovdecision process, multi-armed banditI. INTRODUCTIONWhile proactive content caching has received significantinterest in recent years, most existing strategies in the literature(both with uncoded [1] and coded placement [2]) are basedon known content popularities. Although it is possible toobserve the global popularity of contents in on-demand videostreaming services, such as YouTube, small-cell base stations(SBSs) usually serve a small geographical area, where thelocal content popularity might not be aligned with the globalpopularity [3]. This mismatch between the local and globalcontent popularities requires the design of predictive cachingpolicies that aim to learn the local content popularity fromuser requests. Predictive caching policies can be classified intotwo groups, namely predictive caching with unknown contentpopularity [4] and predictive caching with time-varying con-tent popularity [5]. In [4], the authors focus on a single SBSand model the predictive caching problem as a multi-armedbandit (MAB) problem, in which the received user requestsare utilized to predict the content popularities, and the optimalcaching strategy is obtained by taking into account the cost ofcontent replacements. In [6], this approach has been extendedto a cooperative caching framework, where the SBSs fetcha requested content from neighboring SBSs if it is cachedthere. In [5], a cache replacement strategy has been introducedfor time-varying popularity scenario to maximize the localservice rate with a minimum replacement cost, while a moretheoretical approach is taken in [7], which studies the cacheupdate policy in the case of time-varying content popularities.This work was in part supported by EC H2020-MSCA-RISE-2015 pro-gramme under grant number 690893.Although the predictive caching framework is highly ef-fective in increasing the efficiency of edge caching, it hascertain limitations. Most of the aforementioned strategies aredesigned to predict only the content popularity. However, inmany applications, e.g., news, weather, etc., freshness of thecontent is an important factor for user satisfaction. Contentcaching and refreshment problem has been previously studiedin [8, 9]. In this paper, we consider a heterogeneous cellularnetwork with cache-enabled SBSs, and propose a cost-awarecontent update policy. We first show that the structure ofthe optimal periodic content update policy that minimizes thenetwork cost for given users’ tolerance to the age of the contentis of threshold type. Then, we formulate the resultant cache-refreshment problem as a MAB problem, where we learnusers’ tolerance to the age of the content. Accordingly, wedesign an online cache refreshment policy to minimize theoverall network cost. To the best of our knowledge, this is thefirst work that utilizes a reinforcement learning framework toanalyze user behavior based on content age.II. SYSTEM MODEL AND PROBLEM FORMULATIONWe consider a cellular network with a macro base station(MBS) and a SBS serving the users in the cell. We assume thatboth the MBS and the SBS are equipped with cache memories,storing a library of N distinct dynamic contents, denoted byS1, . . . ,SN . Each dynamic content has a different popularity,i.e., the probability that a user requests content Sn is pn.Dynamic contents, such as news videos, traffic and weatherupdates may change frequently over time. We assume that theMBS, thanks to its relatively high-bandwidth connection tothe content server in the core network, always has the mostup-to-date contents, whereas the stored contents at the SBSare less frequently refreshed to reduce the load on its limited-bandwidth backhaul connection. We consider a discrete timesystem model with equal-length time slots. At the beginningof each time slot the SBS decides on which contents to beupdated. The decision vector at time t is denoted by d(t) =(d1, . . . ,dN), where we set dn(t) = 1 if content Sn is updatedat the end of time slot t, and dn(t) = 0 otherwise. We denoteby hn(t) the age of content Sn in the SBS cache at time slot t.We assume a maximum age Tmax at which a content becomesobsolete. In other words, the age of a content increases until2019 15th Annual Conference on Wireless On-demand Network Systems and Services (WONS)ISBN 978-3-903176-13-3© IFIP 107Content requestUser unsatisfiedSBSMBS123456Figure 1: User requests are first off-loaded to the SBS;however, users who are unsatisfied with the freshness of thecontent served by the SBS are re-directed to the MBS.it becomes obsolete. Accordingly, the age of content Sn, n=1, . . . ,N, evolves over time as follows:hn(t+1) =max{(1 dn(t))(hn(t)+1),Tmax}. (1)We denote the length-N vector of ages associated with all thedynamic contents in the library by h(t).A. User behaviorLet l (t) denote the number of users that request a contentat time slot t. Whenever a user requests a content, the requestis first off-loaded to the SBS. Users have different tolerancelevels to the age of the contents they receive. Hence, weconsider that, with probability P(n)redirect(h), a user is not satisfiedwith the age h of content Sn; and thus, it places another requestfor Sn, which is served directly by the MBS with a freshcontent. Let ln(t) be the number of users that request contentn, n= 1, . . . ,N, in time slot t, i.e., ÂNn=1ln(t) = l (t), which isgoverned by the popularity profile pn. The users that requestcontent n are split into two disjoint sets, where the first set con-sists of lrn(t) users redirected to the MBS, while the secondset consists of lan(t) users satisfied with the content servedby the SBS. Note that lrn(t) and lan(t) are governed by therandom process P(n)redirect(hn(t)). Let  r(t)= (lr1(t), . . . ,lrN(t))be the vector associated with the number of redirected usersfor each content. Expected values of parameters ln and lrnare given by:E[ln|l ] = l pn, and E[lrn|l ,hn] = l pnP(n)r (hn). (2)B. Decision model and the problem formulationLet C( r(t),d(t)) be the cost associated with serving theusers redirected to the MBS at time t, and the backhaul costassociated with updating the contents, if there is any. In thiswork, we assume that this cost is linear in lrn1. Hence,C( r(t),d(t)) =NÂn=1Cn(lrn(t),dn(t)). (3)Similarly, we define the back-haul cost function CBH(d(t))which is also a linear function.CBH(d(t)) =NÂn=1C(n)BH(dn(t)), (4)where C(n)BH(dn(t)) = dn(t)En, with En being the average back-haul cost of updating content n. If the SBS decides to updatecontent n (or multiple contents), the age of the content isupdated at the end of the time slot. Hence,Cn(lrn(t),dn(t)) = bn+anlrn(t)+dn(t)En. (5)Note that updating a content has an immediate cost which islarger than not-updating. However, the incurred extra cost inupdating the content enables more users to be served by theSBS. We aim at minimizing the average total cost as follows:mind(t)2{0,1}NlimT!•1TTÂt=1E⇥C( r(t),d(t))⇤. (6)III. MDP FORMULATIONDefine the state of the system to be h. We denote byV (h) the differential value function at state h. The differentialBellman equations can be written as:V (h)+µ⇤ =mindVd(h), (7)where µ⇤ is the optimal average cost and Vd(h) is thedifferential action-value function defined by:Vd(h), C¯(h,d)+P(h´|h,d)V (h´), (8)where P(h´|h,d) is the transition probability from state h intoh´ when action d is taken which is governed by (1), andC¯(h,d) =Â⇤P( r =⇤)C( r,d). (9)The MDP associated with the average cost minimizationproblem can be solved by the well known value iterationalgorithm. However, the cardinality of the state (i.e., (Tmax)N)and action spaces (i.e., 2N) grow exponentially with thenumber of contents. Hence, the curse of dimensionality is thebottleneck for an efficient solution. To bypass this bottleneck,we note that the cost function in (3) is linear and the transitionprobabilities of a content does not affect the others. Hence, wecan separate the value function in (7) into N independent valuefunctions each representing a distinct content. For each contentn, we haveV (n)(hn)+µ⇤n =mindnV (n)d (hn) (10)1For example, in OFDMA, a user re-directed to the MBS is assigned asubcarrier, and the power allocated to that subcarrier adds linearly to theenergy cost.2019 15th Annual Conference on Wireless On-demand Network Systems and Services (WONS)ISBN 978-3-903176-13-3© IFIP 108V (n)d (hn) = C¯n(hn,dn)+dnV(n)(0)+(1 dn)V (n)(hn+1).(11)We have developed a framework that has enabled distributedpolicies with respect to individual contents. We will show thatfor each content, there exists a threshold policy on the age ofthe content for which it is optimal to update the content. Thefollowing lemma establishes the key property used to provethe structure of the optimal policy.Lemma 1. The differential value function V (n)(hn) for alln = 1, . . . ,N is non-decreasing with respect to the age of thecontent, hn.Proof. We use the value iteration algorithm to prove thelemma. We start by an arbitrary V (n)0 (h) differential valuefunction and obtain the k-step differential value functionV (n)k (h) as follows:V (n)k+1(h) =mind( µ⇤n +C¯n(hn,dn)+dnV(n)k (0)+(1 dn)V (n)k (h+1)) (12)Note that limk!•V(n)k (h) =V(n)(h). The proof is by induction.For k = 1, V (n)1 (h) = mindn( µ⇤n + C¯n(hn,dn) + dnV (n)0 (0) +(1  dn)V (n)0 (hn + 1)), which is the minimum of two non-decreasing functions, and thus, itself is a non-decreasingfunction in hn. Assume that the lemma holds for k. Then,according to (12), V (n)k+1(hn) is also a non-decreasing functionwith respect to hn. By letting k! •, we conclude the proofby showing that V (n)(hn) is also non-decreasing in hn.Theorem 1. For each content the optimal policy minimizingthe average cost is a threshold policy.Proof. The monotonicity of the differential value functionsprove the optimality of the threshold policy [10, Chapter 7].Intuitively, due to the non-decreasing property of the V (n)(hn),at some age, it would be optimal to update the content. Sincethe differential value function is non-decreasing, a larger, orsmaller age would not be able to yield a smaller average cost.IV. LEARNING CONTENT POPULARITY AND AGETOLERANCEIn the previous section, we showed that the problem isseparable and thus, the optimization can be performed foreach content separately. Second, we proved that the policyminimizing the cost is a threshold policy. Hence, the SBSby monitoring the age of the contents individually, needs tooptimize according to a single threshold for each content.Under the threshold policy, the age of a content increaseslinearly until it reaches the threshold wherein the content willbe updated and the age will refresh to a value of zero. Thusthe minimum cost associated with content n is the solution of:µ⇤n =minHn1Hn+1✓ HnÂh=0C¯n(h,0)+En◆, (13)whereC¯n(h,0) = bn+anl pnP(n)redirect(h). (14)Considering the linearity of the cost functions, the averagecost optimization becomes:minHn(bn+En+anpnÂHnh=0P(n)redirect(h)Hn+1). (15)The equivalent optimization problem depends on the redirec-tion probabilities P(n)redirect(h), that are unknown. Hence, in thefollowing we resort to reinforcement learning methods to inferthe redirection probabilities.We consider a sequential learning framework in which theSBS at each iteration of the learning algorithm faces choosinga threshold 0  Hn  Tmax. After choosing the threshold, theSBS will observe a random cost associated with its decision;Cˆn(Hn) =En+ÂHnt=1Cn(lrn(t),0)Hn+1(16)The learning algorithm should provide the SBS a method toadjust its strategy by observing the outcomes of its decisions.This can be formulated as MAB problem, where each action(i.e., thresholds) has an expected return value, correspondingto the value of that action. We denote the true value ofaction Hn by qn(Hn) = 1Hn+1✓ÂHhn=0 C¯n(h,0) + En◆. If theSBS knows the q(·) values, it can simply choose the actionwith the minimum expected cost. A well-studied algorithm forlearning those values is the e-greedy algorithm [11], whichstarts by an arbitrary estimate Qn(·) of the value of theactions, and interacts with the environment to update its initialestimates, eventually converging to the true estimates. Twocritical aspects of the e-greedy algorithm are the exploita-tion and exploration stages. The agent utilizing the estimatesgreedily chooses an action, and thus, it exploits what it knowsalready. Meanwhile, if it chooses an action completely randomregardless of the estimates we say that it explores. Exploitationis necessary to act upon the experience while exploration helpsto improve the estimate values and it facilitates convergenceto the true action values. The e-greedy algorithm is presentedin Algorithm 1.Algorithm 1 e-greedy1: for i= 1,2, . . . do2: Hn (argmaxHn Qn(Hn) with probability 1  e,random action with probability e.3: Apply Hn and observe Cˆn(Hn)4: Qn(Hn) (1 z )Qn(Hn)+zCˆn(Hn)V. NUMERICAL RESULTSIn this section, we aim at evaluating the performance ofthe e-greedy algorithm in finding the optimal thresholds thatminimize the total cost of the system. Due to the separability,we consider only one content and we note that the learningprocesses for all the contents are the same. The popularity2019 15th Annual Conference on Wireless On-demand Network Systems and Services (WONS)ISBN 978-3-903176-13-3© IFIP 109Figure 2: Average regret of the e-greedy algorithm for e =0,0.05,0.1 in a non-stationary environment.of the contents are modeled by a Zipf distribution with anexponent of 1.1. We assume that on average a given userbecomes dissatisfied with a content of age h with probabilityof e 0.4h. Users arrive at the system according to a Poissondistribution with rate 100 users per time slot. The cost ofre-directed users to MBS is Cn(lrn(t)) = 10 · lrn(t) and thebackhaul cost is assumed to be 500 initially and then attime t = 300 the backhaul cost decreases to a value of400 to simulate a time-varying scenario. We illustrate theperformance of the e-greedy algorithm for e = 0, 0.05, 0.1 byadopting average regret as the metric. The regret of a learningalgorithm is defined to be the difference between the costachieved by the learner and the optimal cost. Here, we obtainthe optimal cost by assuming that Predirect(h) is known, and bynumerically solving (13). Note that the estimates of the actionvalues, Q(H), is initialized to be 0 for all H = 0, . . . ,Tmax. Notealso that, the greedy algorithm’s (i.e., e = 0) regret convergesto zero initially (t < 300) even if it always exploits. Thisis not too surprising considering that the action-values areinitialized opportunistically (i.e., the initial costs are believedto be zero by the agent). At the beginning, i.e., t = 0, thegreedy algorithm believes that every action returns a valueof zero. However, by trying each action it gets disappointedin that action and tries the rest. In other words the estimatesare biased. Opportunistic initialization is a simple method toincentivize exploration. However, it can only happen once andat the beginning of time. This method quickly fails in non-stationary environments.We can see that the greedy algorithm cannot adapt to thenon-stationary (t > 300) environment and it gets stuck in asub-optimal threshold. Meanwhile, for 0.05 and 0.1-greedyalgorithm, it is able to adapt to the environment thanks totheir exploration strategy. A large value of e results in moreexploration, and thus, we can see that 0.1-greedy algorithm hasa faster decay in terms of the average regret compared to the0.05-greedy algorithm. However, note that e-greedy algorithmis expected to be at least e away from the optimal cost. Thus,there is a trade-off between the rate of convergence and thevalue of convergence.VI. CONCLUSIONWe have considered a cost minimization problem in a dy-namic content caching setting, and seeked a balance betweenthe number of unsatisfied users redirected to the MBS, and thecost of accessing the backhaul link by the SBS to refresh itsdynamic contents. We have formulated the cost minimizationproblem as an MDP, and proved that a threshold policy in theage of the contents is optimal. Subsequently, to identify theoptimal thresholds, we resorted to learning algorithms sinceusers’ preferences are not known and vary over contents. Tothat extend, we represented the problem in MAB frameworkand through numerical results, we showed that it is possible tomake the expected regret of the learning algorithm arbitrarilyclose to zero. As a future work, we extend the model byconsidering non-linear cost functions and the heterogeneouscellular network architecture with energy harvesting SBSs.REFERENCES[1] K. Poularakis, G. Iosifidis, and L. Tassiulas, “Approx-imation Algorithms for Mobile Data Caching in SmallCell Networks,” IEEE Trans. Comm., Oct 2014.[2] K. Shanmugam, N. Golrezaei, A. Dimakis, A. Molisch,and G. Caire, “FemtoCaching: Wireless Content DeliveryThrough Distributed Caching Helpers,” IEEE Trans. Inf.Theory, vol. 59, no. 12, pp. 8402–8413, Dec 2013.[3] G. Ma, Z. Wang, M. Zhang, J. Ye, M. Chen, and W. Zhu,“Understanding performance of edge content caching formobile video streaming,” IEEE J. Sel. Areas in Commun.,vol. 35, no. 5, pp. 1076–1089, May 2017.[4] P. Blasco and D. Gunduz, “Learning-based optimizationof cache content in a small cell base station,” in IEEEInt. Conf. Commun. (ICC), June 2014, pp. 1897–1903.[5] A. Sadeghi, F. Sheikholeslami, and G. Giannakis, “Op-timal and scalable caching for 5G using reinforcementlearning of space-time popularities,” IEEE J. Sel. TopicsSignal Proc., vol. 12, no. 1, pp. 180–190, Feb 2018.[6] J. Song, M. Sheng, T. Quek, C. Xu, and X. Wang,“Learning-based content caching and sharing for wirelessnetworks,” IEEE Trans. Comm., Oct 2017.[7] B. N. Bharath, K. G. Nagananda, D. Gu¨ndu¨z, and H. V.Poor, “Caching with time-varying popularity profiles: Alearning-theoretic perspective,” IEEE Trans. Commun.,vol. 66, no. 9, pp. 3837–3847, Sept 2018.[8] C. Kam, S. Kompella, G. D. Nguyen, J. E. Wieselthier,and A. Ephremides, “Information freshness and popular-ity in mobile caching,” in IEEE ISIT, June 2017.[9] R. D. Yates, P. Ciblat, A. Yener, and M. Wigger, “Age-optimal constrained cache updating,” in IEEE ISIT, June2017, pp. 141–145.[10] D. Bertsekas, Dynamic programming and optimal con-trol. Athena Scientific, 1995.[11] R. S. Sutton and A. G. Barto, Reinforcement Learning:An Introduction. Cambridge, MA: MIT Press, 1998.2019 15th Annual Conference on Wireless On-demand Network Systems and Services (WONS)ISBN 978-3-903176-13-3© IFIP 110

Dynamic content updates in heterogeneous wireless networks

http://research.sabanciuniv.edu/37102/1/154.pdf

Dynamic content updates in heterogeneous wireless networks

Abstract

Similar works

Full text

Available Versions

Sabanci University Research Database