Unsupervised Anomaly Detection of High Dimensional Data with Low Dimensional Embedded Manifold

Anomaly detection techniques are supposed to identify anomalies from loads of seemingly homogeneous data and being able to do so can lead us to timely, pivotal and actionable decisions, saving us from potential human, financial and informational loss. In anomaly detection, an often encountered situation is the absence of prior knowledge about the nature of anomalies. Such circumstances advocate for ‘unsupervised’ learning-based anomaly detection techniques. Compared to its ‘supervised’ counterpart, which possesses the luxury to utilize a labeled training dataset containing both normal and anomalous samples, unsupervised problems are far more difficult. Moreover, high dimensional streaming data from tons of interconnected sensors present in modern day industries makes the task more challenging. To carry out an investigative effort to address these challenges is the overarching theme of this dissertation. In this dissertation, the fundamental issue of similarity measure among observations, which is a central piece in any anomaly detection techniques, is reassessed. Manifold hypotheses suggests the possibility of low dimensional manifold structure embedded in high dimensional data. In the presence of such structured space, traditional similarity measures fail to measure the true intrinsic similarity. In light of this revelation, reevaluating the notion of similarity measure seems more pressing rather than providing incremental improvements over any of the existing techniques. A graph theoretic similarity measure is proposed to differentiate and thus identify the anomalies from normal observations. Specifically, the minimum spanning tree (MST), a graph-based approach is proposed to approximate the similarities among data points in the presence of high dimensional structured space. It can track the structure of the embedded manifold better than the existing measures and help to distinguish the anomalies from normal observations. This dissertation investigates further three different aspects of the anomaly detection problem and develops three sets of solution approaches with all of them revolving around the newly proposed MST based similarity measure. In the first part of the dissertation, a local MST (LoMST) based anomaly detection approach is proposed to detect anomalies using the data in the original space. A two-step procedure is developed to detect both cluster and point anomalies. The next two sets of methods are proposed in the subsequent two parts of the dissertation, for anomaly detection in reduced data space. In the second part of the dissertation, a neighborhood structure assisted version of the nonnegative matrix factorization approach (NS-NMF) is proposed. To detect anomalies, it uses the neighborhood information captured by a sparse MST similarity matrix along with the original attribute information. To meet the industry demands, the online version of both LoMST and NS-NMF is also developed for real-time anomaly detection. In the last part of the dissertation, a graph regularized autoencoder is proposed which uses an MST regularizer in addition to the original loss function and is thus capable of maintaining the local invariance property. All of the approaches proposed in the dissertation are tested on 20 benchmark datasets and one real-life hydropower dataset. When compared with the state of art approaches, all three approaches produce statistically significant better outcomes. “Industry 4.0” is a reality now and it calls for anomaly detection techniques capable of processing a large amount of high dimensional data generated in real-time. The proposed MST based similarity measure followed by the individual techniques developed in this dissertation are equipped to tackle each of these issues and provide an effective and reliable real-time anomaly identification platform

Optimization of Reservoirs Operation using the Genetic Algorithm and Seasonally Varied Models: Case of Perak Cascading, Malaysia

The Perak cascading scheme located in the state of Perak, Malaysia, consists of four reservoirs, namely, Temenggor, Bersia, Kenering and Chenderoh. The reservoirs are used for hydroelectric power generation and flood control. The hydroelectric power potential of the cascading scheme is 578 MW, while the annual long-term historical average (HA) hydroelectric power generation was around 228 MW. It was about 39.46% of the potential capacity. Accordingly, the study aimed to improve the hydroelectric power generation of the schem

Exploring Renewable Energy Resources Using Remote Sensing and GIS—A Review

Renewable energy has received noteworthy attention during the last few decades. This is partly due to the fact that fossil fuels are depleting and the need for energy is soaring because of the growing population of the world. This paper attempts to provide an idea of what is being done by researchers in remote sensing and geographical information system (GIS) field for exploring the renewable energy resources in order to get to a more sustainable future. Several studies related to renewable energy resources viz. geothermal energy, wind energy, hydropower, biomass, and solar energy, have been considered in this paper. The focus of this review paper is on exploring how remote sensing and GIS-based techniques have been beneficial in exploring optimal locations for renewable energy resources. Several case studies from different parts of the world which use such techniques in exploring renewable energy resource sites of different kinds have also been included in this paper. Though each of the remote sensing and GIS techniques used for exploration of renewable energy resources seems to efficiently sell itself in being the most effective among others, it is important to keep in mind that in actuality, a combination of different techniques is more efficient for the task. Throughout the paper, many issues relating to the use of remote sensing and GIS for renewable energy are examined from both current and future perspectives and potential solutions are suggested. The authors believe that the conclusions and recommendations drawn from the case studies and the literature reviewed in the present study will be valuable to renewable energy scientists and policymakers

The impact of the European Union Emission Trading Scheme on electricity generation sectors

In order to comply with their commitments under the Kyoto Protocol, France and Germany participate to the European Union Emission Trading Scheme (EU ETS) which concerns predominantly electricity generation sectors. In this paper we seek to know if the EU ETS gives appropriate economic incentives for an e¢ cient and strong system in line with Kyoto commitments. Because if so electricity producers in these countries should include the price of carbon in their costs functions. After identifying the di¤erent sub periods of the EU ETS during its pilot phase (2005-2007), we model the prices of various electricity contracts and look at their volatilities around their fundamentals while evaluating the correlation between the electricity prices in the two countries. We finnd that electricity producers in both countries were constrained to include the carbon price in their cost functions during the …rst two years of operation of the EU ETS. During this period, German electricity producers were more constrained than their French counterparts and the inclusion of the carbon price in the cost function of electricity generation has been so much more stable in Germany than in France. Furthermore, the European market for emission allowances has increased the market power of the historical French electricity producer and has greatly contributed to the partial alignment of the wholesale price of electricity in France with those of Germany. .Carbon Emission Trading, Multivariate GARCH models, Structural break, Non Parametric Approach, Energy prices.

Predictive maintenance in hydropower plants : a case study of valves and servomotors

Digitalization has opened the opportunity for a fourth industrial revolution and the hydropower industry is taking charge of enabling digitalization in their operation. There are a lot of studies on predictive maintenance, however, there are, to our knowledge no studies on system-specific predictive maintenance for hydropower. To bridge this gap, the idea of system-specific, Machine Learning driven Predictive Maintenance is explored. Two systems are chosen as a use-case for this thesis: valves and servomotors. With the increasing amount of intermittent renewable energy resources entering the power system, the need for flexibility in the power grid is unequivocal. Valves and servomotors are key components of hydropower control and thus will play a pivotal role in securing flexibility to the grid. The first system assessed is the main valve. In order to make this analysis easily applicable, the data that is already being collected at Nore 1 hydropower plant is analyzed in order to assess the possibility of maintenance prediction from limited data. Unfortunately, this did not achieve the desired results for the data collected from the valve sensors. This is due to the fact that only one variable was measured, in this case, the opening and closing time-lag of the valve. However, this thesis presents a framework for data collection that allows the use of Machine Learning for predictive maintenance. Various sensors are suggested based on several published works on predictive maintenance. The second system assessed is the servomotor that controls the guiding vanes in a Francis turbine. Servomotors are key components of hydropower control. Due to the data not being collected by Statkraft at the time of the study, this data was provided by one of Statkrafts suppliers. By making use of the historical data of pressure as a function of the piston position, a boundary for where new values should be expected is computed by making use of One Class Support Vector Machine. Another embodiment of this case is presented where force is given as a function of piston position, which yielded better results. When new values are being measured, the data is presented as a bullet chart that visualizes the distance of new values compared to the boundary computed by the One Class Support Vector Machine. This tool could easily be applied to other servomotors which perform other tasks such as controlling water injection to a Pelton turbine or opening and closing of the valve, whether they are butterfly or ball valves. Suggestions for further data collection are presented in order to make use of more data for the use of Machine Learning in Predictive Maintenance.Digitalisering har ledet frem til en fjerde industriell revolusjon og vannkraft bransjen er i ferd med å digitalisere sin operasjon. Under literaturstudien er det ikke funnet noen publiseringer innen systemspesifikk maskinlæringsdrevet predikativ vedlikehold. I denne masteroppgaven blir muligheten for bruk av systemspesifikk, maskinlæringdrevet predikativ vedlikehold innen vannkraftverk utforsket for å vekke interesse innen dette feltet. To av vannkraftverkenes maskiner er brukt som eksempler og utforsket: ventiler og servomotor. Økende mengder uregulerbar strøm er introdusert i kraftnettet og behovet for fleksibilitet øker. Ventiler og servomotor er nøkkeldeler av vannkraftverk regulering og spiller en stor rolle i å sikre flexibilitet til strømnettet. Det første systemet som ble analysert er ventiler. For å gjøre analysen og resultatene enkelt anvendbare, blir data som allerede er innsamlet analysert for å utforske muligheten for predikativ vedlikehold med begrenset data. Analysene basert på data samlet inn fra sensorne montert på ventilene ble dessverre ikke konklusive. Det er utfordrende å forutsi fremtiden når man bare har en variabel å ta utgangspunkt i. Likevel presenteres det et prinsipielt rammeverk for innsamling av data som gjør det mulig å ta i bruk maskinlæring for predikativ vedlikehold. Ulike sensorer er foreslått, basert på relevant litteratur innen ventiler og maskinlæring drevet predikativ vedlikehold. Det andre systemet analysert under studien er servomotorer som styrer vannet i en Francis turbin ved å regulere vinklingen til skovlene. Dataen innsamlet om servomotoren er en god indikator på tilstanden til servomotoren. Ettersom dataen var ikke samlet inn av Statkraft da studien ble utført, ble dataen hentet fra en av Statkraft sine leverandører. En One Class Support Vector Machine ble brukt for å beregne foventet verdi av differansetrykk over stempelkamrene, som funksjon av stempel posisjon. En kulegraf som viser avstanden mellom grensen og nye verdier er visualisert. En annen metode er også presentert hvor man regner ut kraft på begge sider av stempelkamrene gjennom trykk for å vise kraft som funksjon av stempel posisjon. Dette ga bedre valideringsresultater i forventet differansekraft over tempelkamrene. Verktøyet kan enkelt bli anvendt til andre servomotorer som styrer vannmengden i en Pelton turbin eller åpning og lukking av ventilene, uavhengig av om det er spjeld- eller kuleventiler. Forslag til videre data innsamling er presentert for å ta i bruk maskinlæring for predikativ vedlikehold.StatkraftsubmittedVersionM-M

Maakasutuse ja kliimamuutuse mõju Eesti jõgede hüdroenergeetilisele potentsiaalile

A Thesis for applying for the degree of Doctor of Philosophy in Engineering Sciences.Water-related changes are currently receiving the most focus in the context of climate change around the world. The changing climate conditions have already redistributed water resources, including the hydropower potential in Estonian rivers. This phenomenon is expected to continue in the future. However, the effects on land use and climate change on the hydropower potential in Estonia are unclear. The technical hydropower potential was assessed for the territory of Estonia (excluding the Narva River). According to the developed method, the total technically feasible hydropower potential in Estonia was calculated to be approximately 80 MW, which was considerably higher than that previously reported. The SWAT model was employed to assess the potential impact of future climate and land-use change on river flow in various Estonian river watersheds, where hydropower is or has been harvested. The SWAT model demonstrated satisfactory performance in describing the hydrological processes in Estonian rivers by using series of mathematical equations. An evident linear trend between the effects of deforestation and afforestation on yearly river flow was observed. The following general rule can be applied to Estonia; a 5% forest change induces a 1% change in annual average flow. Hydrological modeling indicated a positive change in river flow according to both climate scenarios. Increases in the mean annual flow of 10% and 33% were predicted by the climate projections KNMI and DMI, respectively. Hydropower potential is expected to increase in Estonia by the end of the century, compared with the baseline period. The installation of additional turbines, along with upgrading existing turbines, could increase the installed capacity. Furthermore, with the projected overall increase of hydropower potential in Estonia, the construction of new stations becomes more economically feasible and profitable.Veeressursidega seonduvad küsimused on viimastel kümnenditel olnud ülemaailmselt üks huvipakkuvamaid temaatikaid teadlaste seas. Seda just seetõttu, et vee ajaline ja koguseline kättesaadavus on kliimamuutuse tõttu muutumas, mis on mõjutanud ka Eesti jõgede hüdroenergeetilist potentsiaali. Nende muutuste jätkumine avaldab mõju ka hüdroenergia tootlikkusele, ent kui suuresti võib Eesti jõgede hüdroenergeetiline potentsiaal muutuda, on teadmata. Doktoritöös antakse uus hinnang Eesti (v.a. Narva jõe) tehniliselt rakendatavale hüdroenergeetilisele potentsiaalile. Selleks tuletati meetod, mille kohaselt Eesti jõgede tehniline hüdroenergeetiline potentsiaal seni hinnatust märkimisväärselt suurem, olles ligikaudu 80 MW. Maakasutuse ja kliima muutumise mõju Eesti jõgede vooluhulkadele hinnati hüdroloogilise mudeli SWAT abil, kalibreerides ja valideerides seda Eesti suurema hüdroenergeetilise potentsiaaliga jõgedele. Selgus, et SWAT- mudel suudab piisava täpsusega matemaatiliselt kirjeldada Eesti jõgedes kulgevaid looduslikke protsesse. Ilmnes tugev lineaarne seos metsasuse ja aastakeskmise vooluhulga muutuse vahel. Seda seost võib üldistada järgmiselt: metsasuse viieprotsendine muutus muudab jõe aastakeskmist vooluhulka 1 % võrra. Kliimamuutuse mõju Eesti jõgede aastakeskmisele vooluhulgale on positiivne — olenevalt kasutatud kliimamudelist 10 kuni 33 protsenti. Hüdroenergeetiline potentsiaal on Eestis suurenemas. Selle positiivse mõju ärakasutamiseks on soovitatav olemasolevaid hüdroelektrijaamu laiendada või uuendada. Eesti jõgede hüdroenergeetilise potentsiaali suurenemine teeb uute hüdroelektrijaamade rajamise majanduslikult otstarbekaks.Publication of this dissertation is supported by the Estonian University of Life Science

Pumped Hydropower Conversion and Renewable Hybrid Power Plants at Senja

With an increasing demand for power on a global scale, and an increasing interest in renewable energy sources, both solar and wind power is growing fast. Their efficiency is increasing while the prices are decreasing, and the forecasts for these technologies shows a promising future. Along with these intermittent energy sources, storage solutions are also continuously developing, whereas pumped hydropower is the most prevalent. Even if Norway is mainly self-supplied by renewable hydropower production, the population and energy demand is increasing, and so is installation of wind and solar power. Norway, with its mountains and fjords, have some challenges regarding power supply, since there often is long distances between production and demand. One of these locations is found at the northern part of Senja, where voltage drops are causing severe challenges for the seafood industry and contributing to the rise of this thesis. A total upgrade of the power network at northern Senja is estimated to cost in the order of 45M€, and an alternative solution is being sought to solve this challenge. In this thesis, an evaluation is performed regarding locally produced solar and wind power, so that production is closer to the demand. In addition, power production is normally more profitable than network construction. One of the main challenges for solar and wind power is their intermittent nature, demanding a source for storage. Therefore, the main focus in this thesis is on the possibility of converting already existing hydropower plants into pumped hydroelectricity storage, and by this constructing a renewable hydro power plant. Several suitable solutions are found, and even the most expensive is estimated to cost 4/5 of a total upgrade of the power network. It is also found that solar and wind resources act as complementary sources. While wind power could help off with power production during the heavy load period at wintertime, solar power could work as a good source for seasonal energy storage of pumped hydro. Based on the findings in this thesis, suggestions to topics of further work is given

Recovering resources from abandoned metal mine waters : an assessment of the potential options at passive treatment systems

PhD Thesis Appendices can be consulted at the Philip Robinson Library.Remediation of metal-rich discharges from abandoned mines entails capture of metals within a treatment system and, typically, disposal of the waste. A preferable option would be to recover the metals for reuse. For many long-abandoned mines metal loads are often relatively small, albeit they often cause significant environmental pollution. Low-cost passive treatment systems, in which metals are retained in some form of treatment substrate, such as compost, are often preferred. This thesis investigates the amenability of such treatment systems to resource recovery. Two down-flow compost bioreactors, treating zinc-rich discharges, were the focus of the research: a pilot-scale unit at Nenthead, and a full-scale system at Force Crag, both in Cumbria, England. Laboratory investigations of the Nenthead substrate identified 7,900mg/kg zinc in the upper horizons of the substrate, and 2,400mg/kg in the lower horizons, after two years of operation. Acid leaching tests effectively de-contaminated the substrate with respect to zinc and cadmium. Complete recovery of zinc was observed after ≤30 hours across a range of acid leach tests, although 23-37 days were required before equivalent recovery was achieved by biological leaching. The Force Crag system removed >95% zinc over the first year of operation and, removal rates suggest that after 10 years of operation >20,000mg/kg zinc will have accumulated in the substrate. Substrate de-contamination could offer substantial life-cycle cost savings at passive treatment sites, especially by limiting volumes of material for disposal to landfill. Furthermore, recovery of metals has important implications for resource sustainability and circular economics. Other resource recovery options may exist at abandoned mine sites. At Force Crag 1.6kW of kinetic energy exists in flowing mine water, in addition to thermal energy which could be recovered for space heating applications. Recovering this energy would convert this site into a net-generator of power. Because of their often remote locations, renewable energy may be of particular value to off-grid facilities at some mine sites.partially funded by The Coal Authorit