강화학습을 적용한 실용적인 건물 시스템 제어

학위논문(석사) -- 서울대학교대학원 : 공과대학 건축학과, 2021.8. 조성권.HVAC 및 조명과 같은 기존 시스템과 간헐적 재생 에너지, 에너지 저장 시스템 등과 같은 새로운 시스템에도 대응해야 하므로 현대 건물 시스템 제어는 복잡해지고 있습니다. 이에 따라, 건물 시스템 제어기는 건물의 동적 거동에 스스로 적응할 수 있어야 하고 다목적 최적화 결과를 반영할 수 있어야 한다. 강화학습 (reinforcement learning, RL)을 사용하여 전술된 건물 제어기의 성능을 달성할 수 있다는 것은 널리 알려져 있지만, RL을 실제 건물에 적용하기 위해서는 해결해야 할 과제들이 있다: (1) RL의 초기 훈련 기간 동안 불안정한 제어는 예상치 못한 비용을 야기할 수 있다. (2) 여전히 대부분의 RL 기반 제어 전략은 일상적 실무에 적용하기에는 시설 관리자 입장에서 이해하기 어렵고 제어 전략에 대한 해석을 수행할 수 없다. RL 알고리즘을 건물 제어에 적용한다는 것은 의사결정의 주체가 인공지능이 된다는 것을 의미한다. 이때, 건물의 소유주와 운영자는 인공지능 기반 건물 제어기의 의도 및 의사결정 과정에 대한 해석 및 이해를 할 필요가 있다. 첫 번째 과제를 해결하기 위해, RL 에이전트를 사전 학습하고 이를 위해 새로운 개념의 시뮬레이션 모델인 연합 모델이 제안된다. 연합 모델은 빌딩 시스템을 물리적 인과 관계에 따라 모듈로 나누고 각 모듈을 데이터 기반 모델로 개발하여 빌딩 시스템에 대한 시뮬레이션을 수행하는 통합 데이터 기반 모델이다. 대상 건물의 냉방 시스템 시뮬레이션 모델은 6개의 모듈로 구성되고 각 모듈은 BEMS에서 수집된 데이터를 사용하여 개발된다. 연합 모델은 제1법칙 기반 시뮬레이션 모델의 한계 (예: 위상 규칙, 모델 보정)를 극복할 수 있다. Deep Q-Network (DQN)은 냉방 시스템의 동적 거동을 학습하고 건물에 냉방을 공급하는 동시에 에너지 사용을 줄일 수 있는 제어 전략을 모색하는 데 적용된다. DQN의 제어 성능을 현재 건물 운영자들이 적용하는 기존 제어 성능과 비교함으로써 RL 제어기가 시스템의 제어 효율성을 크게 개선할 수 있으며 연합 모델은 강화학습 기반 제어기의 학습을 위한 가상 환경을 제공할 수 있음을 증명한다. DQN 에이전트의 해석성을 높이기 위해 의사결정 트리를 사용하여 에이전트의 의사결정 프로세스에 대한 설명을 추출한다. 에이전트에서 생성된 상태-작업 (state-action) 쌍이 의사결정 트리를 훈련하는 데 사용된다. 얕지만 쉽게 해석할 수 있는 모델을 사용한 사후 해석은 강화 학습의 투명성과 해석성을 향상시킨다. 또한 의사결정나무가 만든 분류 결과는 인공지능이 만든 제어 전략을 단순화시킨 'If-then' 규칙을 도출한다. 추출된 규칙 (reduced rule) 기반 제어의 성능과 DQN 제어기의 성능을 비교하여 두 제어기 사이의 에너지 절약량 차이가 2.8%로 미미함을 보인다. 즉, 규칙 기반 제어가 충분한 성능을 보인다는 것을 증명한다. 본 연구는 기축 사무실 건물의 냉방 제어를 위한 설명 가능한 RL의 적용 방안에 대해 수행된다. 의사 결정 트리를 훈련된 DQN 에이전트에 적용한 다음 일련의 단순화된 제어 규칙을 도출한다. 이 연구는 설명 가능한 강화학습을 이용한 정량화된 규칙 도출 프레임워크를 제안하고, 복잡한 강화학습 알고리즘과 비교하여 단순하지만 정량적인 평가가 수행된 규칙이 충분한 성능을 보여줄 수 있음을 보여준다. 이 연구의 의의는 건물 통제에 대한 정량적 평가를 통해 규칙을 도출하는 방법을 제안하는 데 있다.Building controls are becoming complicated because modern building systems must respond to not only conventional systems like HVAC and lighting, but also to novel systems such as intermittent renewables, energy storage systems, and more. Therefore, the advanced building controllers must balance the trade-off between multiple objectives and automatically adapt to dynamic environment. Although it is widely acknowledged that reinforcement learning (RL) can be beneficially used for better building control, there are several challenges that should be addressed for real life application of RL: (1) unstable and poor control actions during early training period of RL may cause unexpected costs; (2) many RL-based control actions still remain unexplainable for daily practice of facility managers. By applying RL algorithms as artificial intelligences that are the subject of decision-making, owners and operators of buildings need to be reassured about the controllers intentions. To address the first challenge, federated model, a novel concept of simulation model, is proposed for pre-training RL agents. The federated model is an integrated data-driven model that divides a building system into several modules based on physical causality and develops each module into a data-driven model to perform simulations on building systems. A federated model of a complex cooling system of a target building is realized using six modules, each developed using data gathered from BEMS. By developing the federated model, limitations of physics-based simulation models (eg. topology rules, model calibration) are overcome. Deep Q-network (DQN) is applied to learn the dynamics of the cooling system and explore control strategies that can reduce energy use while providing cold for the building. By comparing the control performance of DQN with the performance of baseline control, it is shown that RL controller can significantly enhance control efficiency of the system and the federated model can provide sufficient virtual experience for the controller. To enhance interpretability of the DQN agent, decision tree is used to extract explanation of the decision making process of the agent. State-action pairs generated by the agent is used train a decision tree. Post-hoc interpretation using a shallow but easily interpretable model enhances transparency and interpretability of reinforcement learning. Also, the result of classification made by the decision tree provides If-then rules which are reduced version of control strategies made by the artificial intelligence. The performance of the reduced rule-based control is also compared to the performance of DQN controller. It is demonstrated that the reduced rule is good-enough and the difference in energy savings between the two is marginal, resulting in 2.8%. This study reports the development of explainable RL for cooling control of an existing office building. A decision tree is applied to trained DQN agent and then a set of reduced-order control rules are suggested. This study proposes rule reduction framework using explainable reinforcement learning and demonstrates that reduced rules can perform as well as complex reinforcement learning algorithms. The significance of this study lies in proposing how to derive rules with quantitative evaluation for building control.1. Introduction 1 1.1 Control of building systems 1 1.2 Problem Description 2 1.3 Goal 4 1.4 Thesis Outline 5 2. Deep Q-network (DQN) 7 2.1. Summary of reinforcement learning 7 2.1.1 Elements of reinforcement learning 7 2.1.2 Value function 9 2.2. Deep Q-learning 12 2.2.1 Temporal difference (TD) learning and Q-learning 12 2.2.2 Deep Q-learning 14 2.3. Previous works to implement reinforcement learning to existing buildings 16 2.4. Conclusion 19 3. Decision Trees 21 3.1 Summary of decision tree 21 3.2 Classification And Regression Trees (CART) 23 3.3 Interpreting reinforcement learning using decision tree 24 3.4 Conclusion 26 4. Target building and Federated model 27 4.1 Parallel cooling system 27 4.2 Federated model 31 5. Explainable deep Q-network and rule reduction for building control 40 5.1 DQN implementation framework 40 5.2 Control results of DQN 46 5.3 Rule reduction from DQN agent 50 5.4 Discussion 54 6. Conclusion 55 6.1 Summary and conclusion 55 6.2 Future works 57 Reference 58석

Pregnane X Receptor agonist Increases the Expression Levels of the Plasma Membrane Monoamine Transporter

We evaluated the effect of the pregnane X receptor agonist, pregnenolone 16 alpha-carbonitrile (PCN) on the expression levels of plasma monoamine transporter (PMAT) in the intestine. Male C57/BL6 mice were divided into two 2 groups: mice in the PCN group (n=3) were administered PCN once a day for 4 days, while those in the control group (n=3) received the same volume of vehicle once a day for 4 days. After the mice were killed 24 h after administration of the last dose of PCN or vehicle, and the expression levels of PMAT in the intestine tissues were isolated and measured the expression level of PMAT using immunohistochemical and western blotting analyses. The expression level of PMAT expression levels in the small intestine increased after PCN treatment. These results suggest that the induction of PMAT may play a clinically significant role by increasing intestinal absorption of PMAT substrates such as metformin.ope

Verapamil decreases the glucose-lowering effect of metformin in healthy volunteers

AIM: The organic cation transporter 1 (OCT1) plays a key role in the cellular transport of metformin and its subsequent glucose-lowering effect. A recent non-clinical study reported that metformin uptake into hepatocytes is regulated via OCT1, and that uptake was strongly inhibited by verapamil. Therefore, we investigated the effects of verapamil co-administration on the pharmacokinetics and pharmacodynamics of metformin in humans. METHODS: We evaluated the pharmacokinetics and the anti-hyperglycaemic effects of metformin using an oral glucose tolerance test (OGTT) in 12 healthy participants, before (day 1) and after metformin treatment (day 2), and again on days 15 and 16 after co-administration with verapamil. RESULTS: Verapamil inhibited the ability of metformin to reduce maximum blood glucose concentrations (ΔGmax ) by 62.5% (P = 0.008) and decreased the area under the glucose concentration-time curve (ΔAUCgluc ) by 238% (P = 0.015). However, verapamil did not significantly alter the Cmax and the AUC of metformin, nor its renal clearance. CONCLUSIONS: Our results suggest that verapamil remarkably decreases the glucose-lowering effect of metformin, possibly by acting as a competitive inhibitor of OCT1.ope

Discovery of URAT1 SNPs and association between serum uric acid levels and URAT1

OBJECTIVES: Human urate transporter 1 (URAT1) is a member of the organic anion transporter family (SLC22A12) that primarily regulates the renal tubular reabsorption of uric acid. This case-control study was designed to analyse whether hURAT1 might also be a candidate gene for hyperuricaemia or hypouricaemia. SETTING: We recruited 68 healthy volunteers and divided them into two groups: a normal uric acid group and a hyperuricaemia group. We analysed the sequence of the URAT1 gene and found five significant single nucleotide polymorphisms (SNPs). We then selected 900 male subjects from the 262,200 enrolled in the Korean Cancer Prevention Study-II (KCPS-II) cohort for further genetic analysis. PARTICIPANTS: DNA samples from 36 individuals with normal uric acid (8.5 mg/dL) were sequenced. Five significant SNPs (rs7929627, rs75786299, rs3825017, rs11602903 and rs121907892) were identified. We then chose 900 subjects from the KCPS-II cohort consisting of 450 subjects with normal uric acid (UA 8.7 mg/dL). The groups were matched by age, body mass index, metabolic syndrome and use of anti-hypertensive medication. PRIMARY OUTCOME MEASURES: We compared the OR of the incidence of hyperuricaemia by URAT1 genotype. RESULTS: The strongest association with hyperuricaemia was observed for rs75786299 (IVS3+11A/G) with an OR of 32.05. rs7929627 (IVS7-103A/G) and rs3825017 (N82N) showed an association with hyperuricaemia with ORs of 2.56 and 2.29, respectively. rs11602903 (788A/T) and rs121907892 (W258X) were negatively correlated with hyperuricaemia with ORs of 0.350 and 0.447, respectively. Individuals carrying the GATAG haplotype (n=32)-a relatively common variant consisting of rs7929627, rs75786299 and rs3825017-showed the highest risk for hyperuricaemia with an OR of 92.23 (p=9.55×10(-3)). CONCLUSIONS: These results indicate that five newly described SNPs in the hURAT1 gene are significantly associated with uric acid level (4-2008-0318 and 4-2011-0277).ope

Fabrication of ZnO nanotube via atomic later deposition and their practicality as high performance gas sensor

학위논문 (석사)-- 서울대학교 대학원 : 화학부, 2011.2. 이성훈.Maste

음향가진을 이용한 원형제트에서의 보텍스 병합

Thesis (doctoral)--서울대학교 대학원 :기계공학과,1998.Docto

메트포민의 약동, 약력학 및 약물상호 작용에 미치는 약물 수송체의 영향에 대한 연구

Dept. of Medical Science/박사The biguanide derivative, metformin, is the first-line oral hypoglycemic drug for the treatment of type 2 diabetes. Its primary action is to lower hepatic glucose production by inhibiting gluconeogenesis. Generally, the change of drug transporter activity influences its drug exposure in systemic circulation by altering drug absorption and drug excretion especially in biliary and renal excretion. Meanwhile, drug transporting activity into action sites has been known to play important roles in drug actions. Metformin is a substrate of organic cation tranasporters (OCTs) and multidrug and toxin extrusion 1 (MATE1). OCT1 is a transporter located primarily in hepatocyte sinusoidal membranes, whereas OCT2 is localized mainly in the basolateral membrane of the kidney proximal tubule. MATE1 is an H+/organic cation antiporter on the apical membrane of kidney tubules and on bile canaliculi. Since OCT1 is located in the drug action site, metformin may play an important role in lowering glucose. OCT2 and MATE1 may affect metformin’s pharmacokinetics due to its location. There is controversy on pharmacogenetic difference of OCT2 (rs 316019) in metformin’s renal clearance and plasma concentration. Because of the location of MATE1, metformin may play a more important role than OCT2 does. I investigated the role of OCT2 and MATE1 on metformin’s pharmacokinetics and renal clearance through a genotype-enriched prospective clinical trial. OCT1 is related to the hepatic uptake of metformin and is associated with pharmacological action and adverse drug reaction. In this thesis, I investigated how OCT1 works with metformin to lower glucose effect by co-administering rifampin, an OCT1 inducer, and verapamil, and OCT1 inhibitor, with metformin. My thesis is composed of three parts. To assess the influences of polymorphisms MATE1 (rs 2252281) and OCT2 (rs 316019) on metformin’s pharmacokinetics and renal clearance, I conducted a genotype enriched-clinical trial in 48 subjects in Study I by balancing the MATE1 and OCT2 genotypes. In the latter two parts, I evaluated the influence of OCT1 on metformin’s glucose lowering effect in Study II and III. Study II assessed the change in metformin’s glucose lowering effect and metformin’s pharmacokinetics before and after rifampin, an OCT1 inducer, administration. Study III evaluated the change in metformin’s pharmacokinetics and metformin’s glucose lowering effect with and without verapamil administration. Metformin’s glucose effect was calculated by comparing the difference of oral glucose tolerance test (OGTT) before and after metformin administration. Three parameters were used to assess the metformin’s glucose lowering effect. Maximum glucose lowering effect (Gmax) was determined and area under the serum glucose concentration-time curve (ΔAUCgluc) was calculated using the trapezoidal rule. ΔAUCgluc60 was defined as area under the glucose curve from 0 to 60 minutes after glucose ingestion, during which serum glucose concentration increased. Metformin’s pharmacokinetic parameters were evaluated as Cmax, Tmax, AUCmet and t1/2 for plasma pharmacokinetic parameters and renal clearance (CLR) of metformin, Creatinine clearance (CLCr) and renal secretion of metformin (SrCLR) for urine pharmacokinetic parameters.In study I, plasma pharmacokinetic parameters (Cmax, Tmax, AUCmet and t1/2) are not significantly different by MATE1 and OCT2 genotypes. Renal clearance and net tubular secretion of metformin were significantly different by MATE1 genotype (CLR: 617±126 vs. 556±106 vs. 507±104 ml/min, P=0.031 and SrCLR: 517±121 vs. 456±107 vs. 399±107 ml/min, P=0.017). In study II, I found that rifampin increased Gmax by 41.9% (P=0.024) and ∆AUCgluc60 by 54.5% (P=0.020). Renal clearance of metformin was increased 16% by rifampin (P=0.008), and systemic exposure of metformin was slightly increased (13%, P=0.049), possibly due to increased absorption. Rifampin increased OCT1 mRNA levels 4.1-fold in peripheral blood cells (P=0.001). In study III, Verapamil treatment decreased mean Gmax by 62.5% (16 mg/dl vs. 6 mg/dl; P = 0.010). The glucose lowering effect of metformin was not observed after verapamil treatment in mean ΔAUCgluc60 levels (594±500 mg/dl•min vs. -6±556 mg/dl•min; P = 0.008) and mean ΔAUCgluc levels (509±1224 mg/dl•min vs. -702±1103 mg/dl•min; P = 0.015). Pharmacokinetic parameters, AUC, Cmax, CLR and SrCLR of metformin did not change when treated with verapamil.OCT1 plays a key role in metformin’s glucose-lowering effect. OCT1 based drug-drug interaction is important in the response of metformin presumably by affecting the effective concentration of metformin in the target organ. The important drug transporter on metformin’s renal clearance and secretion is not OCT2, but the MATE1 polymorphism (rs2252281).ope

美國 마약정책의 변화와 실패: 거버넌스의 이중성

UN은 국제마약밀매의 수익을 년 4천억 달러로 추산하고 있다. 현재 미국은 명실공히 세계 최대의 마약소비국이다. 미국의 마약지수(인구 10만명 당 마약사용자수)는 세계에서 가장 높은 256을 기록하고 있다. 같은 맥락에서 마약판매는 미국에서 나타나는 각종 불법수익 중에서 가장 많은 부분을 차지하고 있다. 간단한 최근의 통계자료만 보더라도 1998년 미국에서 마약으로 인한 경제비용은 1.434 억 달러로 추산되었고 2000년의 경우 더욱 증가하여 1，600억 달러로 추산하고 있다. 이것은 상원에서 통과된 2003년도 3.551 억 달러에 달하는 美국방예산의 거의 50% 에 가까운 비용이다. UN 국제마약통제위원회 (INCB)의 보고에 따르면 인터넷 주문을 이용한 마약밀매가 점차적으로 증가하고 있는 추세를 감안할 때 인터넷 사용이 가장 발달된 미국에서 마약으로 인한 경제비용은 시간이 갈수록 증가할 전망이다. 20세기 시작된 이래 미국에서 마약문제는 정치 사회적으로 복잡한 양상을 보여왔다. 20세기 동안 미국의 마약사는 크게 세 가지로 분류할 수 있다

South Korean Media Coverage of Official Development Assistance: Topic Modeling Analysis of Six Print Media from 1993 to 2016

본 논문은 1993년부터 2016년의 기간 동안 한국 주요 신문들의 ODA(official developmentassistance) 관련 보도가 어떻게 변화해 왔는지를 탐색한다. 8,407건의 신문기사들의 주제가 어떻게 변했는지를 확률 토픽모형인 잠재 디리클레 할당(Latent Dirichlet Allocation)을 통해 보여준다. 분석 대상이 된 일간지는 조선일보, 동아일보, 한겨레신문, 경향신문, 매일경제신문, 한국경제신문, 총 여섯 개로 진보와 보수 이념, 경제 분야의 주요 언론사를 모두 포함하였다. 연구주제는 크게 두가지이다. 첫째, ODA 기사의 주제는 시기별로 어떤 차이가 있는가? 둘째, ODA 신문 보도의 주제가 신문사의 성격 및 이념적 정향에 따라 어떻게 달라지는가? 분석 결과 한국 언론은 대체로 수출 및 기업 진출, EDCF(Economic Development Cooperation Fund) 차관, 아세안 경제협력, 새마을운동 등 한국의 발전경험과 국익에 초점을 맞추어 보도를 한 것으로 나타났다. 북한의 개발원조에 대한 보도 비중도 높았다. OECD DAC(Development Assistance Committee), UN 등과 관련되는 글로벌 규범에 대한 관심은 특정 시기를 제외하곤 매우 낮았다. 또한 원조추진체계, 원조예산, 인력양성 등 국내 거버넌스와 관련된 주제가 점차 중요해 지는 것으로 나타났다. ODA 보도에 가장 적극적인 언론은 경제신문이었으며, 진보 신문은 보수와 경제신문에 비해 보도량도 적을 뿐 더러 특히, 글로벌 규범에 대한 관심도가 낮은 것으로 분석되었다. 그러나 진보 신문의 북한 문제에 대한 보도 비중은 상대적으로 높았다. 본 연구는 바람직한 ODA 정책추진과 국민인식 제고를 위해 언론의 역할이 중요하다는 점을 강조하고 있다.This article investigates media coverage of official development assistance (ODA) in South Korea with the use of probabilistic topic modeling. It analyzes 8,407 newspaper articles with mentions of ODA related keywords collected from six major South Korean newspapers and examines how main topics of media coverage have evolved during the period, 1993~2016. The Latent Dirichlet Allocation (LDA) analysis suggests that first, South Korean media have paid greater attention to the topics related to South Koreas economic and political interests than global norms or recipient countries needs; secondly, the topics examining ODA delivery structure and governance received an increasing attention and it reflects the consolidation of South Korean ODA; and lastly, liberal media are much less likely to cover ODA related issues than their conservative counterparts. This study contributes to the literature on the role of media in shaping ODA policies and public opinion, and requires further research on the consequences of media coverage on policy and opinion formation.이 논문은 2017년 대한민국 교육부와 한국연구재단의 지원을 받아 수행된 연구임(NRF-2016S1A3A2925085)