Greedy Confidence Bound Techniques for Restless Multi-armed Bandit Based Cognitive Radio

학위논문 (석사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2013. 8. 이정우.인지 라디오는 제한된 스펙트럼 리소스를 활용하는 효율적인 기술이다. 이러한 소프트웨어 기반 인지 라디오 시스템은 무선 통신환경이 가지고 있는 통계적인 특성을 인식 하고 그것에 기반하여 스스로를 적응시키는 동작을 수행한다. 인지 라디오 시스템이 추구하는 목표는 크게 신뢰도 높은 통신을 언제 어디서나 구축하는것과 스펙트럼을 효율적으로 사용하는 두가지로 요약할수있다. 연속성이있는 자원의 효율적인 할당문제는 MAB로 알려진 알고리즘적 방식을 통해 모델링하여 반복적인 실험을 통해 해를 구할수있다. MAB 문제에서는 현재의 적절한 보상을 달성하기위한 전략과 현재의 이득을 어느정도 포기하고 미래에 예측가능한 이득을 증진시키는 두가지 전략을 적절히 배합하여 최적의 해를 찾는다. 일반적인 MAB 관련 문제에서는 확률적인 보상을 생성하는 다중 Arm 또는 이에 상응하는 등가의 미리 정의된 동작이 있으며, 플레이어는 예상되는 총 보상을 극대화하기 위한 K개의 Arm을 연속적으로 선택하는 정책을 찾는다. 이 논문에서는 보다 도전적인 문제인 비 베이지안 RMAB 문제를 다룬며 이때 Markov chain의 파라미트들은 미리 알려지지 않았다고 가정된다. 또한 이를 이용해 인지 라디오 시스템을 위한 실용적인 동적 스펙트럼 센싱에에대한 연구를 진행한다. 만약 채널 사용에 있어 우선순위를 갖는 1차 사용자의 채널 사용 패턴이 동일하고 독립적인 39 Markov chain으로 모델링 할때 이를 비 베이지안 RMAB문제로 다룰수있다. 본 논문의 기여도는 통계적 특성이 알려지지 않은 동적 채널에서 2시간 슬롯 Greedy 신뢰도 범위 알고리즘을 (Two slot GCB) 이용해 효율적인 다중 채널 스펙트럼 센싱 알고리즘을 개발한데에 있다. 이는 Markov 파라미터 추정기법과 채널 센싱기법이 결합된 형태를 띄고 있으며 이를 통해 기존의 UCB 알고리즘의 Regret rate이 ln(t) 을 가짐을 보일수있다. 마지막으로 본 연구를 통해 제한조건이 없는 연속 시간 Markov chain 채널 모델을 적용한 인지 라디오 시스템에서의 채널 접근 정책을 제안한다.The electromagnetic radio spectrum is a natural resource, the use of which by transmitters and receivers is licensed by governments. The underutilization of the electromagnetic spectrum leads us to think in terms of spectrum holes, where a spectrum hole is a band of frequencies assigned to a primary user, but, at a particular time and specific geographic location, the band is not being utilized by that user. Spectrum utilization can be improved significantly by making it possible for a secondary user (who is not being serviced) to access a spectrum hole unoccupied by the primary user at the right location and the time in question. Cognitive radio is viewed as a novel approach for improving the utilization of a precious natural resource: the radio electromagnetic spectrum. The cognitive radio, built on a software-defined radio, is defined as an intelligent wireless communication system that is aware of its environment and uses the methodology of understanding-by-building to learn from the environment and adapt to statistical variations in the input stimuli, with two primary objectives in mind: Highly reliable communication whenever and wherever neededEfficient utilization of the radio spectrum. Multi-armed bandit (MAB) problems are a class of sequential resource allocation problems, which has fundamental conflict between a strategy yielding high present reward and a strategy sacrificing present gain for better future reward. In a multi-armed bandit problem, there are multiple (N) arms which generate stochastic reward, and a player seeks a policy to select multiple K≥1 arms in order to maximize the expected total reward over multiple time-slots. A particularly challenging variant of MAB problems is the restless multi-armed bandit problem (RMAB), in which all arms evolve as Markov chains. Even in the Bayesian case, where the parameters of the Markov chains are known, this problem is difficult to solve, and has been proved to be PSPACE hard. We consider more challenging non-Bayesian RMAB problems, in which the parameters of the Markov chain are further assumed to be unknown a priori. We demonstrate our approach on a practical problem related to dynamic spectrum sensing for cognitive radio applications. If the primary user occupancy on each channel is modeled as an identical but independent Markov chain with unknown parameters, we obtain a non-Bayesian RMAB. Our main contribution in this work is that we develop an efficient new multi-channel spectrum sensing algorithm for unknown dynamic channels based on the two-slot greedy confidence bound algorithm (Two-slot GCB), which combines the Markov Chain parameter estimation and the channels sensing simultaneously and provides a new analysis which shows that UCB algorithms have the regret rate of ln(t). And finally we give out another solution for the accessing policy for Cognitive Radio of unconstrained continuous time Markov chain channel model.Contents Abstract .................................................................................................................. i Contents ............................................................................................................... iii List of Figures ......................................................................................................... v Chapter 1 ............................................................................................................... 1 Introduction ........................................................................................................... 1 1.1 Cognitive Radio (CR) .............................................................................. 1 1.2 MAB problem ........................................................................................ 2 1.3 Combination .......................................................................................... 3 Chapter 2 ............................................................................................................... 5 MAB algorithms with a Markov Chain Reward Process ........................................ 5 2.1 The multi-armed bandit problem .......................................................... 5 2.2 UCB algorithm for MAB ......................................................................... 7 2.3 A simple induction for UCBs regret rate of log(t) ................................. 7 2.4 CR with Markov Chain channel model and Greedy algorithm ............ 12 2.4.1 Problem Modeling ........................................................................... 12 2.4.2 Belief Vector based Greedy Algorithm ............................................ 12 2.5 UCB algorithm for CR Markov Chain channel model .......................... 13 2.6 Two and One slot algorithm for CR Markov Chain channel model ..... 14 2.6.1 Two-slot Algorithm ...................................................................... 14 2.6.2 One-slot Algorithm ...................................................................... 15 2.6.3 Combination of Two-slot and One-slot algorithms ..................... 16 2.7 Simulation ............................................................................................ 17 2.7.1 Single User Single Play ..................................................................... 17 2.7.2 Single User Multi play ...................................................................... 18 2.7.3 Decentralized Multi User Single Play ............................................... 21 Chapter 3 ............................................................................................................. 26 Un-slotted channel access policy ........................................................................ 26 3.1 Continuous-time Markov channel model ............................................ 26 iv 3.2 Channel access policy without distribution constraints ....................... 27 3.2.1 Channel access behavior and mathematic modeling ....................... 27 3.2.2 One general channel access policy ..................................................... 31 3.2.3 Another general channel access policy ............................................... 32 3.3 Another model for collision time constraint .............................................. 34 Chapter 4 ............................................................................................................. 36 Conclusions .......................................................................................................... 36 개 요 .................................................................................................................... 38 Bibliography ......................................................................................................... 40Maste

Dynamic Lidar Ratio Calculation and Aerosol Vertical Extinction Coefficient Retrieval Based on Observed Visibility

Micropulse lidar (MPL) cannot directly retrieve the aerosol extinction coefficient under cloudy conditions and at night. Therefore, we used ground visibility, Fernald’s near-end solution method, and the linear correlation between the near-end lidar signal (photons) and ground aerosol extinction coefficient (correlation coefficient = 0.98), to calculate the lidar constant and lidar ratio (LR). We compared the aerosol optical depth (AOD) retrieved from MPL and the AOD retrieved from the multifilter rotating shadowband radiometer (MFRSR-7) at the same band (532 nm). The correlation coefficient was 0.77. The vertical distribution of aerosols in daytime and nighttime during summer was obtained from lidar in July at 00:00 and 12:00 Beijing time (UTC+8). In daytime, under clear sky conditions, the distribution displayed a unimodal and peak at approximately 2000 m. The distribution at night was more complicated than that in the day, with three results. The first was monotonically decreasing from ground to upper layer, with a peak at 600 m and two peaks at approximately 1200 m. In general, the aerosol extinction coefficient at nighttime is higher than that at daytime below 1200 m. The near-ground extinction coefficient at night is higher than in the day

Comparisons of methods for linkage analysis and haplotype reconstruction using extended pedigree data

We compare and contrast the performance of SIMPLE, a Monte Carlo based software, with that of several other methods for linkage and haplotype analyses, focusing on the simulated data from the New York City population. First, a whole-genome scan study based on the microsatellite markers was performed using GENEHUNTER. Because GENEHUNTER had to drop individuals for many of the pedigrees, we performed a follow-up study focusing on several regions of interest using SIMPLE, which can handle all pedigrees in their entirety. Second, 3 haplotyping programs, including that in SIMPLE, were used to reconstruct haplotypic configurations in pedigrees. SIMPLE emerges clearly as a preferred tool, as it can handle large pedigrees and produces haplotypic configurations without double recombinant haplotypes. For this study, we had knowledge of the simulating models at the time we performed the analysis

Influence of the stoichiometry of tin-based 2D/3D perovskite active layers on solar cell performance

2D/3D mixed tin perovskites have the advantages of high crystallinity and preferential orientation compared to pure 3D tin perovskite. However, solar cells based on 2D/3D mixed tin perovskites are still limited by low power conversion efficiency (PCE) when compared to their lead-based counterparts. It is essential to gain deeper insight into the factors that limit the performance of these solar cells in order to further improve them. In this work, we demonstrate that the starting stoichiometry of 2D/3D (PEA(0.08)FA(x)SnI(3)) tin perovskite films influences their crystallization and photophysical properties as well as the solar cell performance. The reference 2D/3D film (x = 0.92, where x refers to the stoichiometry of the precursors) is highly crystalline with the 3D phase preferentially oriented and a small amount of 2D phase located at the bottom of the film. The reference solar cell delivers a PCE of about 8.0%. 2D/3D films with even higher FA concentration (x > 1.0) mainly consist of poorly crystalline and randomly oriented 3D phases, with much higher trap density compared to the reference film. The corresponding solar cells therefore suffer from severe trap-assisted charge recombination, and deliver a poor PCE o

Improvement of hybrid grouper (Epinephelus fuscoguttatus ♀ × E. lanceolatus ♂) by enzyme-digested poultry by-product: Growth performance, amino acid and peptide transport capacity, and intestinal morphology

BackgroundAt present, fish meal (FM) resources are in short supply, and competition for food between humans and animals is becoming increasingly critical. Finding non-grain protein sources that can replace FM is the key to solving the rapid development of aquaculture.MethodsSeven trial diets were prepared with 0 g/kg (EP0), 30 g/kg (EP3), 60 g/kg (EP6), 90 g/kg (EP9), 120 g/kg (EP12), 150 g/kg (EP15), and 180 g/kg (EP18) of enzyme-digested poultry by-product meal (EPBM) by replacing of FM. A total of 630 hybrid groupers (Epinephelus fuscoguttatus ♀ × E. lanceolatus ♂) were equally portioned into 21 tanks. At 8:00 and 16:00 each day, groupers were fed until they were full for a cumulative period of 8 weeks.ResultsThe results showed that 30 g/kg of EPBM significantly increased the rates of weight gain and special growth (P < 0.05). Significantly higher activities of serum glutamic pyruvic transaminase, glutamic oxaloacetic transaminase, catalase, and superoxide dismutase were observed in the EP3 group (P < 0.05). The categories and numbers of the top 10 dominant bacteria in the phylum and genus levels were not significantly influenced by feed (P > 0.05). In the proximal intestine and distal intestine, there were significantly higher expressions of SNAT3, LAAT1, CAT2, and CAT1 in the EP3 group compared with the EP0 group (P < 0.05). In the EP3 group, the expressions of PepT1, LAAT1, B0, +AT, and CAT2 were significantly increased in MI than those in all other groups (except the EP0 group, P < 0.05).ConclusionWhen FM was replaced by 30 g/kg of EPBM, growth performance, antioxidant capacity, and the ability to transport amino acids and peptides of hybrid grouper were significantly improved

Impact of the Hole Transport Layer on the Charge Extraction of Ruddlesden-Popper Perovskite Solar Cells

Recent works demonstrate that polyelectrolytes as a hole transport layer (HTL) offers superior performance in Ruddlesden-Popper perovskite solar cells (RPPSCs) compared to poly(3,4-ethylenedioxythiophene) polystyrene sulfonate (PEDOT:PSS). The factors contributing to such improvement need to be systematically investigated. To achieve this, we have systematically investigated how the two HTLs affect the morphology, crystallinity, and orientation of the Ruddlesden-Popper perovskite (RPP) films as well as the charge extraction of the RPPSCs. PEDOT:PSS as a HTL leads to RPP films of low crystallinity and with a number of large pinholes. These factors lead to poor charge carrier extraction and significant charge recombination in the RPPSCs. Conversely, a PCP-Na HTL gives rise to highly crystalline and pinhole-free RPPSC films. Moreover, a PCP-Na HTL provides a better energy alignment at the perovskite/HTL interface because of its higher work function compared to PEDOT:PSS. Consequently, devices using PCP-Na as HTLs are more efficient in extracting charge carriers

Tuning the Energetic Landscape of Ruddlesden-Popper Perovskite Films for Efficient Solar Cells

Ruddlesden-Popper perovskite films deposited with different methods show very diverse phase segregation and composition. When DMSO is used as solvent, the conventional method based on spin-coating and annealing produces very poor devices, whereas the vacuum-assisted method proposed here allows obtaining devices with efficiency up to 14.14%. The conventional method gives rise to a three-dimensional (3D)-like phase on the top of the film but dominant n = 2 phase with large domains (∼40 μm) at the bottom of the film. These n = 2 domains are oriented with their inorganic slabs parallel to the substrate and inhibit the charge transport in the vertical direction. Consequently, severe monomolecular and bimolecular charge recombination occurs in the solar cells. Conversely, the vacuum-assisted method yields films with a 3D-like phase dominant throughout their entire thickness and only a small amount of n ≤ 2 domains of limited dimensions (∼3 μm) at the bottom, which facilitate charge transport and reduce charge recombination.</p

Enhancing the Performance of the Half Tin and Half Lead Perovskite Solar Cells by Suppression of the Bulk and Interfacial Charge Recombination

In this article it is investigated how the hole extraction layer (HEL) influence the charge recombination and performance in half tin and half lead (FASn(0.5)Pb(0.5)I(3)) based solar cells (HPSCs). FASn(0.5)Pb(0.5)I(3) film grown on PEDOT:PSS displays a large number of pin-holes and open grain boundaries, resulting in a high defect density and shunts in the perovskite film causing significant bulk and interfacial charge recombination in the HPSCs. By contrast, FASn(0.5)Pb(0.5)I(3) films grown on PCP-Na, an anionic conjugated polymer, show compact and pin-hole free morphology over a large area, which effectively eliminates the shunts and trap states. Moreover, PCP-Na is characterized by a higher work function, which determines a favorable energy alignment at the anode interface, enhancing the charge extraction. Consequently, both the interfacial and bulk charge recombination in devices using PCP-Na HEL are considerably reduced giving rise to an overall improvement of all the device parameters. The HPSCs fabricated with this HEL display power conversion efficiency up to 16.27%, which is 40% higher than the efficiency of the control devices using PEDOT:PSS HEL (11.60%). Furthermore, PCP-Na as HEL offers superior performance in larger area devices compared to PEDOT:PSS

Photocatalytic Degradation of Organic Pollutants in Water Using Graphene Oxide Composite

Developing sustainable and less-expensive technique is always challenging task in water treatment process. This chapter explores the recent development of photocatalysis technique in organic pollutant removal from the water. Particularly, advantages of graphene oxide in promoting the catalytic performance of semiconductor, metal nanoparticle and polymer based photocatalyst materials. Owing to high internal surface area and rapid electron conducting property of graphene oxide fostering as backbone scaffold for effective hetero-photocatalyst loading, and rapid photo-charge separation enables effective degradation of pollutant. This chapter summaries the recent development of graphene oxide composite (metal oxide, metal nanoparticle, metal chalcogenides, and polymers) in semiconductor photocatalysis process towards environmental remediation application