Remote sensing big data computing: challenges and opportunities

As we have entered an era of high resolution earth observation, the RS data are undergoing an explosive growth. The proliferation of data also give rise to the increasing complexity of RS data, like the diversity and higher dimensionality characteristic of the data. RS data are regarded as RS ‘‘Big Data’’. Fortunately, we are witness the coming technological leapfrogging. In this paper, we give a brief overview on the Big Data and data-intensive problems, including the analysis of RS Big Data, Big Data challenges, current techniques and works for processing RS Big Data

Operation of meshed high voltage direct current (HVDC) overlay grids: from operational planning to real time operation

Energy turnaround from conventional to renewable energy generation needs bulk power long distance transmission. This new transmission objective can be meet with an HVDC overlay grid spanning the existing AC transmission system. This thesis proposes an operation management strategy for future HVDC overlay grids subdivided in tertiary, secondary and primary control instances. The tertiary control ensures coordination among HVDC converters and with the AC system. It determines converter reference values on a regular basis. It is proposed for the case of having multiple as well as a single system operator responsible for the overlay HVDC grid. The secondary control instance locally adapts tertiary control’s converter references to the actual grid requirements (e.g. after disturbances). The primary control ensures DC energy balance. Therefore, a continuous p-v-characteristic is proposed as well as two appropriate parameterization methods. One emulates piecewise linear p-v-characteristics and the other performs an automatic parameterization according to available balancing power provision capabilities on related AC point of common coupling. All control methods are validated by numerical case studies.Weltweit aber besonders in Europa steigt der Bedarf große Leistungen über weite Strecken zu transportieren. Dies ist hauptsächlich in der Energiewende und dem damit zusammenhängenden stark ansteigenden Anteil Erneuerbarer Energien und deren Erzeugungszentren begründet. Ein bedeutender Teil der Erneuerbaren Energien wird zukünftig weitab der Lastzentren produziert. Zur Lösung dieser daraus resultierenden neuen Transportaufgabe ist die Hochspannungsgleichstromübertragung (HGÜ) besonders geeignet. Eine redundante und damit auch wirtschaftliche Ausführung stellt das vermaschte HGÜ-Netz dar, das in der Energieversorgungsnetzhiearchie eine neue Netzebene dargestellt und somit als Overlay-HGÜ-Netz bezeichnet wird. Diese Arbeit widmet sich der Fragestellung der Betriebsführung eines Overlaynetzes. Dazu wird eine dreistufige Betriebsführung vorgeschlagen. In Anlehnung an die im europäischen AC-Verbundnetz bestehende Dreiteiligkeit wird eine Untergliederung in folgende Regelungsinstanzen vorgenommen: Tertiär-, Sekundär und Primärregelung. Die Tertiärregelung übernimmt die Koordinierungsaufgabe der Umrichter untereinander und mit dem unterlagerten AC-Netz im Rahmen einer Betriebsplanung. Es ist ein betriebstypisches Aktualisierungsintervall von 15 Minuten vorgesehen, indem die Umrichtersollwerte vorgegeben werden. Deren Bestimmung erfolgt durch ein auf dieses nichtlineare Problem zugeschnittenen AC/DC Optimal Power Flow. Dieses Verfahren fußt auf der Verfügbarkeit aller AC- und DC-Netzinformationen im Gebiet des Overlaynetzes. Im Falle einer föderalen Organisation eines HGÜ-Overlaynetzes in Europa müssen die Zielsetzungen mehrere Übertragungsnetzbetreiber (ÜNB) bei der Bestimmung eines Umrichtersollwertfahrplans berücksichtig werden. Für diesen Fall wird hier eine Methode vorgeschlagen, die mittels eines Aushandlungsprozesses die ÜNB spezifischen Kostenfunktionen für den Einsatz von HGÜ-Umrichtern in der entsprechenden Regelzone zu einer für das gesamte Overlaynetz gültigen Zielfunktion konsolidiert. Dabei werden Grenzwerte der einzelnen beteiligten ÜNB ebenso berücksichtigt wie lokale Zielfunktionen. Die Sekundärregelung passt die von der Tertiärregelung vorgegebenen Umrichtersollwerte innerhalb des 15-min-Betriebsintervalls vor allem im Fall von Störungen an. Dafür wird ein Verfahren vorgeschlagen, das sich der Informationen eines Weitbereichsüberwachungssystems bedient, um signifikante Abweichung der geplanten Leistungsflüsse zu erfassen. Die Umrichterwirkleistungssollwerte werden entsprechend angepasst. Eine Aufteilung von unplanmäßigen Leistungsflüssen zwischen AC und DC-Netz sorgt für eine Entlastung des AC-Netzes und beugt Betriebsmittelüberlastungen und dadurch verursachten Instabilitätsphänomenen vor. Die Primärregelung gewährleistet das Gleichgewicht zwischen ein- und ausgespeister Wirkleistung in das / aus dem HGÜ-Overlaynetz. Ist die diesbezügliche Leistungsbilanz ausgewogen, ist das Energiegleichgewicht, die sogenanntes Energiestabilität, gewahrt. Die DC-Zeitkonstanten sind klein. Nur eine dezentral (am Umrichterstandort) angeordnete Regelung kann zeitlich angemessen reagieren. Diese nutzt eine p-u-Regelcharakteristik, die die Umrichtersollleistung entsprechend der Abweichung von der DC-Sollspannung anpasst. Dafür werden eine kontinuierliche p-u-Charakteristik sowie Verfahren zu deren Parametrierung vorgeschlagen. Für die Bereitstellung von DC-Regelleistung besonders geeignete AC-Knoten können so angemessen für das HGÜ-Overlaynetz genutzt werden. Die Funktionalität des hier vorgeschlagenen dreiteiligen Bertriebsführungsverfahrens für vermaschte HGÜ-Netze wird anhand von numerischen Fallstudien auf Basis einer typischen Netztsituation in Zentraleuropa validiert.There is an increasing demand for long distance bulk power transmission worldwide and particularly in Europe. Energy turnaround from conventional to renewable energy generation is one of the main drivers. This implies that a significant percentage of electricity production is generated remotely from load centers, by huge wind farms, for example. This new transmission objective can be met with high voltage direct current (HVDC) transmission. An HVDC grid is favored for redundancy as well as economic reasons. As this HVDC grid will be a new network layer above the existing AC transmission layer, it is referred to as an “overlay” HVDC grid. This thesis proposes a three stage operation management strategy for future HVDC overlay grids. The architecture is comprised of tertiary, secondary and primary control instances which reflect the hierarchy of AC system operation. All control methods have been validated by numerical case studies on a reference grid which is a representative of a typical interconnected network situation in central Europe. The proposed tertiary control ensures coordination among all HVDC converters and with the underlaying AC system. It serves as an example of converter reference value determination in a 15 minutes time interval. Therefore a mixed AC/DC optimal power flow method is proposed which is capable of solving this nonlinear optimization problem based on a complete set of topological and other state information of the entire grid. In the event of having different transmission system operators (TSO) operating only a subset of converters of the HVDC overlay grid, the optimization problem becomes increasingly complex since each TSO might have its own optimization objectives. This problem is addressed by another multiple objective function approach. The proposed method superimposes particular cost functions of related TSO which yields system wide cost functions as a basis for AC/DC power flow optimization. The Secondary control instance adapts the tertiary control’s converter reference values within the 15 minute interval to the actual grid requirements, particularly in the event of grid disturbances. An algorithm is proposed that identifies significant deviations from the actual power flow schedule by a wide are monitoring system. Converter power references are adapted in order to optimally share the deviations between the AC system and the HVDC overlay grid. Since data availability is key for the robust operation of this method, backup mechanisms for data acquisition is also proposed. The Primary control ensures DC energy balance, which is referred to as the energy stability of HVDC grids. Converter reference values for active power need to be adjusted in the event of a mismatch between active power fed to and drawn from the HVDC grid. As the time constants within a DC grid are very small, this is a fast, local control based on p v characteristics; the converter’s power reference is adjusted in accordance with deviation of the DC node voltage from its reference. Furthermore, a continuous p v characteristic is proposed as well as two appropriate parameterization methods. One emulates already existing piecewise linear p v characteristics for DC node voltage control and the other performs an automatic parameterization according to available balancing power provision capabilities on related AC point of common couplings. The latter significantly reduces the additional loading of the AC transmission grid with DC balancing power flows as the AC nodes, which are the most technically feasible, are utilized to provide the most DC balancing power

An Efficient Holistic Data Distribution and Storage Solution for Online Social Networks

In the past few years, Online Social Networks (OSNs) have dramatically spread over the world. Facebook [4], one of the largest worldwide OSNs, has 1.35 billion users, 82.2% of whom are outside the US [36]. The browsing and posting interactions (text content) between OSN users lead to user data reads (visits) and writes (updates) in OSN datacenters, and Facebook now serves a billion reads and tens of millions of writes per second [37]. Besides that, Facebook has become one of the top Internet traffic sources [36] by sharing tremendous number of large multimedia files including photos and videos. The servers in datacenters have limited resources (e.g. bandwidth) to supply latency efficient service for multimedia file sharing among the rapid growing users worldwide. Most online applications operate under soft real-time constraints (e.g., ≤ 300 ms latency) for good user experience, and its service latency is negatively proportional to its income. Thus, the service latency is a very important requirement for Quality of Service (QoS) to the OSN as a web service, since it is relevant to the OSN’s revenue and user experience. Also, to increase OSN revenue, OSN service providers need to constrain capital investment, operation costs, and the resource (bandwidth) usage costs. Therefore, it is critical for the OSN to supply a guaranteed QoS for both text and multimedia contents to users while minimizing its costs. To achieve this goal, in this dissertation, we address three problems. i) Data distribution among datacenters: how to allocate data (text contents) among data servers with low service latency and minimized inter-datacenter network load; ii) Efficient multimedia file sharing: how to facilitate the servers in datacenters to efficiently share multimedia files among users; iii) Cost minimized data allocation among cloud storages: how to save the infrastructure (datacenters) capital investment and operation costs by leveraging commercial cloud storage services. Data distribution among datacenters. To serve the text content, the new OSN model, which deploys datacenters globally, helps reduce service latency to worldwide distributed users and release the load of the existing datacenters. However, it causes higher inter-datacenter communica-tion load. In the OSN, each datacenter has a full copy of all data, and the master datacenter updates all other datacenters, generating tremendous load in this new model. The distributed data storage, which only stores a user’s data to his/her geographically closest datacenters, simply mitigates the problem. However, frequent interactions between distant users lead to frequent inter-datacenter com-munication and hence long service latencies. Therefore, the OSNs need a data allocation algorithm among datacenters with minimized network load and low service latency. Efficient multimedia file sharing. To serve multimedia file sharing with rapid growing user population, the file distribution method should be scalable and cost efficient, e.g. minimiza-tion of bandwidth usage of the centralized servers. The P2P networks have been widely used for file sharing among a large amount of users [58, 131], and meet both scalable and cost efficient re-quirements. However, without fully utilizing the altruism and trust among friends in the OSNs, current P2P assisted file sharing systems depend on strangers or anonymous users to distribute files that degrades their performance due to user selfish and malicious behaviors. Therefore, the OSNs need a cost efficient and trustworthy P2P-assisted file sharing system to serve multimedia content distribution. Cost minimized data allocation among cloud storages. The new trend of OSNs needs to build worldwide datacenters, which introduce a large amount of capital investment and maintenance costs. In order to save the capital expenditures to build and maintain the hardware infrastructures, the OSNs can leverage the storage services from multiple Cloud Service Providers (CSPs) with existing worldwide distributed datacenters [30, 125, 126]. These datacenters provide different Get/Put latencies and unit prices for resource utilization and reservation. Thus, when se-lecting different CSPs’ datacenters, an OSN as a cloud customer of a globally distributed application faces two challenges: i) how to allocate data to worldwide datacenters to satisfy application SLA (service level agreement) requirements including both data retrieval latency and availability, and ii) how to allocate data and reserve resources in datacenters belonging to different CSPs to minimize the payment cost. Therefore, the OSNs need a data allocation system distributing data among CSPs’ datacenters with cost minimization and SLA guarantee. In all, the OSN needs an efficient holistic data distribution and storage solution to minimize its network load and cost to supply a guaranteed QoS for both text and multimedia contents. In this dissertation, we propose methods to solve each of the aforementioned challenges in OSNs. Firstly, we verify the benefits of the new trend of OSNs and present OSN typical properties that lay the basis of our design. We then propose Selective Data replication mechanism in Distributed Datacenters (SD3) to allocate user data among geographical distributed datacenters. In SD3,a datacenter jointly considers update rate and visit rate to select user data for replication, and further atomizes a user’s different types of data (e.g., status update, friend post) for replication, making sure that a replica always reduces inter-datacenter communication. Secondly, we analyze a BitTorrent file sharing trace, which proves the necessity of proximity-and interest-aware clustering. Based on the trace study and OSN properties, to address the second problem, we propose a SoCial Network integrated P2P file sharing system for enhanced Efficiency and Trustworthiness (SOCNET) to fully and cooperatively leverage the common-interest, geographically-close and trust properties of OSN friends. SOCNET uses a hierarchical distributed hash table (DHT) to cluster common-interest nodes, and then further clusters geographically close nodes into a subcluster, and connects the nodes in a subcluster with social links. Thus, when queries travel along trustable social links, they also gain higher probability of being successfully resolved by proximity-close nodes, simultaneously enhancing efficiency and trustworthiness. Thirdly, to handle the third problem, we model the cost minimization problem under the SLA constraints using integer programming. According to the system model, we propose an Eco-nomical and SLA-guaranteed cloud Storage Service (ES3), which finds a data allocation and resource reservation schedule with cost minimization and SLA guarantee. ES3 incorporates (1) a data al-location and reservation algorithm, which allocates each data item to a datacenter and determines the reservation amount on datacenters by leveraging all the pricing policies; (2) a genetic algorithm based data allocation adjustment approach, which makes data Get/Put rates stable in each data-center to maximize the reservation benefit; and (3) a dynamic request redirection algorithm, which dynamically redirects a data request from an over-utilized datacenter to an under-utilized datacenter with sufficient reserved resource when the request rate varies greatly to further reduce the payment. Finally, we conducted trace driven experiments on a distributed testbed, PlanetLab, and real commercial cloud storage (Amazon S3, Windows Azure Storage and Google Cloud Storage) to demonstrate the efficiency and effectiveness of our proposed systems in comparison with other systems. The results show that our systems outperform others in the network savings and data distribution efficiency

From online social network analysis to a user-centric private sharing system

Online social networks (OSNs) have become a massive repository of data constructed from individuals’ inputs: posts, photos, feedbacks, locations, etc. By analyzing such data, meaningful knowledge is generated that can affect individuals’ beliefs, desires, happiness and choices—a data circulation started from individuals and ended in individuals! The OSN owners, as the one authority having full control over the stored data, make the data available for research, advertisement and other purposes. However, the individuals are missed in this circle while they generate the data and shape the OSN structure. In this thesis, we started by introducing approximation algorithms for finding the most influential individuals in a social graph and modeling the spread of information. To do so, we considered the communities of individuals that are shaped in a social graph. The social graph is extracted from the data stored and controlled centrally, which can cause privacy breaches and lead to individuals’ concerns. Therefore, we introduced UPSS: the user-centric private sharing system, in which the individuals are considered as the real data owners and provides secure and private data sharing on untrusted servers. The UPSS’s public API allows the application developers to implement applications as diverse as OSNs, document redaction systems with integrity properties, censorship-resistant systems, health care auditing systems, distributed version control systems with flexible access controls and a filesystem in userspace. Accessing users’ data is possible only with explicit user consent. We implemented the two later cases to show the applicability of UPSS. Supporting different storage models by UPSS enables us to have a local, remote and global filesystem in userspace with one unique core filesystem implementation and having it mounted with different block stores. By designing and implementing UPSS, we show that security and privacy can be addressed at the same time in the systems that need selective, secure and collaborative information sharing without requiring complete trust

Blockchain-based Data Management for Smart Transportation

Smart services for Intelligent Transportation Systems (ITS) are currently deployed over centralized system solutions. Conversely, the use of decentralized systems to support these applications enables the distribution of data, only to those entities that have the authorization to access them, while at the same time guaranteeing data sovereignty to the data creators. This approach not only allows sharing information without the intervention of a “trusted” data silo, but promotes data verifiability and accountability. We discuss a possible framework based on decentralized systems, with a focus on four requirements, namely data integrity, confidentiality, access control and persistence. We also describe a prototype implementation and related performance results, showing the viability of the chosen approach

Design of efficient and elastic storage in the cloud

Ph.DDOCTOR OF PHILOSOPH