Deep learning technology for predicting solar flares from (Geostationary Operational Environmental Satellite) data

YesSolar activity, particularly solar flares can have significant detrimental effects on both space-borne and grounds based systems and industries leading to subsequent impacts on our lives. As a consequence, there is much current interest in creating systems which can make accurate solar flare predictions. This paper aims to develop a novel framework to predict solar flares by making use of the Geostationary Operational Environmental Satellite (GOES) X-ray flux 1-minute time series data. This data is fed to three integrated neural networks to deliver these predictions. The first neural network (NN) is used to convert GOES X-ray flux 1-minute data to Markov Transition Field (MTF) images. The second neural network uses an unsupervised feature learning algorithm to learn the MTF image features. The third neural network uses both the learned features and the MTF images, which are then processed using a Deep Convolutional Neural Network to generate the flares predictions. To the best of our knowledge, this work is the first flare prediction system that is based entirely on the analysis of pre-flare GOES X-ray flux data. The results are evaluated using several performance measurement criteria that are presented in this paper

Entorno experimental de procesamiento de datos distribuidos integrando devops en el ciclo de entrega de software

Undoubtedly, the generation of large volumes of data from different sources has allowed organizations to obtain value and knowledge from the data generated. Therefore, companies need specialists who are able to digest this data and turn it into useful information. An important theme is the way in which students can adopt theoretical knowledge in a practical way in big data environments, cloud technologies and or tools demanded in the market avoiding extensive configurations. In this paper we create an experimental big data environment, describing the concept as such, its reference architectures and components, designing and implementing an architecture for a distributed data processing cluster, integrating Devops in a continuous software delivery flow; through an automated deployment of big data processing infrastructure as code in the cloud.Sin duda las generaciones de grandes volúmenes de datos de diferentes fuentes han permitido a las organizaciones obtener valor y conocimiento a partir de los datos generados. Por ello, las empresas necesitan a especialistas que sean capaces de digerir esos datos y convertirlos en información útil. Un tema importante es la manera en que los estudiantes pueden adoptar el conocimiento teórico de manera práctica en entornos big data, tecnologías en la nube y herramientas demandadas en el mercado evitando configuraciones extensas. En este artículo se crea un entorno experimental de big data, describiendo el concepto como tal, sus arquitecturas de referencia y componentes, diseñando e implementando una arquitectura para un clúster de procesamiento de datos distribuido, integrando Devops en un flujo de entrega continua de software; mediante un despliegue automatizado big data procesing de infraestructura como código en la nube

Big data deployment in containerized infrastructures through the interconnection of network namespaces

Big Data applications tackle the challenge of fast handling of large streams of data. Their performance is not only dependent on the data frameworks implementation and the underlying hardware but also on the deployment scheme and its potential for fast scaling. Consequently, several efforts have focused on the ease of deployment of Big Data applications, notably through the use of containerization. This technology was indeed raised to bring multitenancy and multiprocessing out of clusters, providing high deployment flexibility through lightweight container images. Recent studies have focused mostly on Docker containers. Notwithstanding, this article is actually interested in recent Singularity containers as they provide more security and support high-performance computing (HPC) environments and, in this way, they can make Big Data applications benefit from the specialized hardware of HPC. Singularity 2.x, however, does not isolate network resources as required by most Big Data components. Singularity 3.x allows allocating each container with isolated network resources, but their interconnection requires a nontrivial amount of configuration effort. In this context, this article makes a functional contribution in the form of a deployment scheme based on the interconnection of network namespaces, through underlay and overlay networking approaches, to make Big Data applications easily deployable inside Singularity containers. We provide detailed account of our deployment scheme when using both interconnection approaches in the form of a “how-to-do-it” report, and we evaluate it by comparing three Big Data applications based on Hadoop when performing on a bare-metal infrastructure and on scenarios involving Singularity and Docker instances.Peer ReviewedPostprint (author's final draft

Evaluating Performance of Serverless Virtualization

Abstract. The serverless computing has posed new challenges for cloud vendors that are difficult to solve with existing virtualization technologies. Maintaining security, resource isolation, backwards compatibility and scalability is extremely difficult when the platform should be able to deliver native performance. This paper contains a literature review of recently published results related to the performance of virtualization technologies such as KVM and Docker, and further reports a DESMET benchmarking evaluation against KVM and Docker, as well as Firecracker and gVisor, which are being used by Amazon Web Services and Google Cloud in their cloud services. The context for this research is coming from education, where students return their programming assignments into a source code repository system that further triggers automated tests and potentially other tasks against the submitted code. The used environment consists of several software components, such as web server, database and job executor, and thus represents a common architecture in web-based applications. The results of the research show that Docker is still the most performant virtualization technology amongst the selected ones. Additionally, Firecracker and gVisor perform better in some areas than KVM and thus are viable options for single-tenant environments. Lastly, applications that run untrusted code or have otherwise really high security requirements could potentially leverage from using either Firecracker or gVisor

Adaptive learning-based resource management strategy in fog-to-cloud

Technology in the twenty-first century is rapidly developing and driving us into a new smart computing world, and emerging lots of new computing architectures. Fog-to-Cloud (F2C) is among one of them, which emerges to ensure the commitment for bringing the higher computing facilities near to the edge of the network and also help the large-scale computing system to be more intelligent. As the F2C is in its infantile state, therefore one of the biggest challenges for this computing paradigm is to efficiently manage the computing resources. Mainly, to address this challenge, in this work, we have given our sole interest for designing the initial architectural framework to build a proper, adaptive and efficient resource management mechanism in F2C. F2C has been proposed as a combined, coordinated and hierarchical computing platform, where a vast number of heterogeneous computing devices are participating. Notably, their versatility creates a massive challenge for effectively handling them. Even following any large-scale smart computing system, it can easily recognize that various kind of services is served for different purposes. Significantly, every service corresponds with the various tasks, which have different resource requirements. So, knowing the characteristics of participating devices and system offered services is giving advantages to build effective and resource management mechanism in F2C-enabled system. Considering these facts, initially, we have given our intense focus for identifying and defining the taxonomic model for all the participating devices and system involved services-tasks. In any F2C-enabled system consists of a large number of small Internet-of-Things (IoTs) and generating a continuous and colossal amount of sensing-data by capturing various environmental events. Notably, this sensing-data is one of the key ingredients for various smart services which have been offered by the F2C-enabled system. Besides that, resource statistical information is also playing a crucial role, for efficiently providing the services among the system consumers. Continuous monitoring of participating devices generates a massive amount of resource statistical information in the F2C-enabled system. Notably, having this information, it becomes much easier to know the device's availability and suitability for executing some tasks to offer some services. Therefore, ensuring better service facilities for any latency-sensitive services, it is essential to securely distribute the sensing-data and resource statistical information over the network. Considering these matters, we also proposed and designed a secure and distributed database framework for effectively and securely distribute the data over the network. To build an advanced and smarter system is necessarily required an effective mechanism for the utilization of system resources. Typically, the utilization and resource handling process mainly depend on the resource selection and allocation mechanism. The prediction of resources (e.g., RAM, CPU, Disk, etc.) usage and performance (i.e., in terms of task execution time) helps the selection and allocation process. Thus, adopting the machine learning (ML) techniques is much more useful for designing an advanced and sophisticated resource allocation mechanism in the F2C-enabled system. Adopting and performing the ML techniques in F2C-enabled system is a challenging task. Especially, the overall diversification and many other issues pose a massive challenge for successfully performing the ML techniques in any F2C-enabled system. Therefore, we have proposed and designed two different possible architectural schemas for performing the ML techniques in the F2C-enabled system to achieve an adaptive, advance and sophisticated resource management mechanism in the F2C-enabled system. Our proposals are the initial footmarks for designing the overall architectural framework for resource management mechanism in F2C-enabled system.La tecnologia del segle XXI avança ràpidament i ens condueix cap a un nou món intel·ligent, creant nous models d'arquitectures informàtiques. Fog-to-Cloud (F2C) és un d’ells, i sorgeix per garantir el compromís d’acostar les instal·lacions informàtiques a prop de la xarxa i també ajudar el sistema informàtic a gran escala a ser més intel·ligent. Com que el F2C es troba en un estat preliminar, un dels majors reptes d’aquest paradigma tecnològic és gestionar eficientment els recursos informàtics. Per fer front a aquest repte, en aquest treball hem centrat el nostre interès en dissenyar un marc arquitectònic per construir un mecanisme de gestió de recursos adequat, adaptatiu i eficient a F2C.F2C ha estat concebut com una plataforma informàtica combinada, coordinada i jeràrquica, on participen un gran nombre de dispositius heterogenis. La seva versatilitat planteja un gran repte per gestionar-los de manera eficaç. Els serveis que s'hi executen consten de diverses tasques, que tenen requisits de recursos diferents. Per tant, conèixer les característiques dels dispositius participants i dels serveis que ofereix el sistema és un requisit per dissenyar mecanismes eficaços i de gestió de recursos en un sistema habilitat per F2C. Tenint en compte aquests fets, inicialment ens hem centrat en identificar i definir el model taxonòmic per a tots els dispositius i sistemes implicats en l'execució de tasques de serveis. Qualsevol sistema habilitat per F2C inclou en un gran nombre de dispositius petits i connectats (conegut com a Internet of Things, o IoT) que generen una quantitat contínua i colossal de dades de detecció capturant diversos events ambientals. Aquestes dades són un dels ingredients clau per a diversos serveis intel·ligents que ofereix F2C. A més, el seguiment continu dels dispositius participants genera igualment una gran quantitat d'informació estadística. En particular, en tenir aquesta informació, es fa molt més fàcil conèixer la disponibilitat i la idoneïtat dels dispositius per executar algunes tasques i oferir alguns serveis. Per tant, per garantir millors serveis sensibles a la latència, és essencial distribuir de manera equilibrada i segura la informació estadística per la xarxa. Tenint en compte aquests assumptes, també hem proposat i dissenyat un entorn de base de dades segura i distribuïda per gestionar de manera eficaç i segura les dades a la xarxa. Per construir un sistema avançat i intel·ligent es necessita un mecanisme eficaç per a la gestió de l'ús dels recursos del sistema. Normalment, el procés d’utilització i manipulació de recursos depèn principalment del mecanisme de selecció i assignació de recursos. La predicció de l’ús i el rendiment de recursos (per exemple, RAM, CPU, disc, etc.) en termes de temps d’execució de tasques ajuda al procés de selecció i assignació. Adoptar les tècniques d’aprenentatge automàtic (conegut com a Machine Learning, o ML) és molt útil per dissenyar un mecanisme d’assignació de recursos avançat i sofisticat en el sistema habilitat per F2C. L’adopció i la realització de tècniques de ML en un sistema F2C és una tasca complexa. Especialment, la diversificació general i molts altres problemes plantegen un gran repte per realitzar amb èxit les tècniques de ML. Per tant, en aquesta recerca hem proposat i dissenyat dos possibles esquemes arquitectònics diferents per realitzar tècniques de ML en el sistema habilitat per F2C per aconseguir un mecanisme de gestió de recursos adaptatiu, avançat i sofisticat en un sistema F2C. Les nostres propostes són els primers passos per dissenyar un marc arquitectònic general per al mecanisme de gestió de recursos en un sistema habilitat per F2C.Postprint (published version

Modelo de processamento de imagem, com múltiplas fontes de aquisição, para manipulação aplicada à domótica

Este trabalho foca-se em modelos de processamento de imagem para utilização na visão por computador. Modelos de processamento de imagem com multi-aquisição e/ou em multi-perspectiva, para um conhecimento do meio circundante, com possibilidade de comando e controlo na área da domótica e/ou robótica móvel. Os algoritmos desenvolvidos têm a capacidade de serem implementados em blocos de software ou hardware, de forma independente (autónomos), ou integrados como componentes de um sistema mais complexo. O desenvolvimento dos algoritmos privilegiou o seu elevado desempenho, constrangido pela minimização da carga computacional. Nos modelos de processamento de imagem desenvolvidos foram focados 4 tópicos fundamentais de investigação: a) detecção de movimento de objectos e seres humano em ambiente não controlado; b) detecção da face humana, a ser usada como variável de controlo (entre outras aplicações); c) capacidade de utilização de multi-fontes de aquisição e processamento de imagem, com diferentes condições de iluminação não controladas, integradas num sistema complexo com diversas topologias; d) capacidade de funcionamento de forma autónoma ou em rede distribuída, apenas comunicando resultados finais, ou integrados modularmente na solução final de sistemas complexos de aquisição de imagem. A implementação laboratorial, com teste em protótipos, foi ferramenta decisiva no melhoramento de todos os algoritmos desenvolvidos neste trabalho; IMAGE PROCESSING MODELS, WITH MULTIPLE ACQUISITION SOURCES, FOR MANIPULATION IN DOMOTICS Abstract: This work focuses on image processing models for computer vision. Image processing models with multi-acquisition and/or multi-perspective models were developed to acquire knowledge over the surrounding environment, allowing system control in the field of domotics and/or mobile robotics. The developed algorithms have the capacity to be implemented in software or hardware blocks, independently (autonomous), or integrated as a component in more complex systems. The development of the algorithms was focused on high performance constrained by the computational burden minimization. In the developed image processing models it were addressed 4 main research topics: a) movement detection of objects and human beings in an uncontrolled environment; b) detection of the human face to be used as a control variable (among other applications); c) possibility of using multi-sources of acquisition and image processing, with different uncontrolled lighting conditions, integrated into a complex system with different topologies; d) ability to work as an autonomous entity or as a node integrated on a distributed network, only transmitting final results, or integrated as a link in a complex image processing system. The laboratorial implementation, with prototype tests, was the main tool for the improvement of all developed algorithms, discussed in the present wor

Conception et développement d'une solution de diffusion des données géospatiales massives 3D dans un contexte de gestion de risque d'inondations

Au Québec, chaque année, les inondations printanières présentent un défi majeur pour les autorités québécoises. Ainsi, l'élaboration de nouveaux outils et de nouvelles méthodes pour diffuser et visualiser des données massives spatiotemporelles 3D dynamiques d'inondation est très important afin de mieux comprendre et gérer les risques reliés aux inondations. Cette recherche s'intéresse à la diffusion de données géospatiales massives 3D (modèles de bâtiments 3D, arbres, modèles numériques d'élévation de terrain (MNE), données LiDAR, imageries aériennes, etc.) en relation avec les inondations. Le problème est qu'il n'existe pas, à travers la littérature, des systèmes de diffusion efficaces des données massives 3D adaptées aux besoins de cette recherche. En ce sens, notre objectif général consiste à développer un outil de diffusion des données géospatiales massives 3D qui sont des bâtiments 3D et des modèles de terrains de haute résolution à l'échelle de la province du Québec. Les défis de diffusion du flux de données massives, nous ramènent à considérer la technique de tuilage 3D pour convertir les données brutes en formats et structures vectoriels plus légers et adaptés à la diffusion comme la spécification "3D Tiles" pour tuiler les bâtiments 3D, les nuages de points LiDAR et d'autres modèles géoréférencés 3D et le maillage irrégulier, notamment les TIN, pour tuiler les modèles numériques de terrain. Aussi, l'utilisation des techniques de traitement parallèle permet de gérer efficacement les flux massifs de données et d'améliorer le temps de traitement permettant ainsi la scalabilité et la flexibilité des systèmes existants. A cet effet, deux pipelines de tuilage ont été développés. Le premier pipeline concerne la création des tuiles de bâtiments 3D selon la spécification "3D Tiles". Le deuxième est pour créer des tuiles de terrain basées sur des maillages irréguliers. Ces pipelines sont ensuite intégrés dans un système de traitement distribué basé sur des conteneurs Docker afin de paralléliser les processus de traitements. Afin de tester l'efficacité et la validité du système développé, nous avons testé ce système sur un jeux de données massif d'environ 2.5 millions bâtiments 3D situés au Québec. Ces expérimentations ont permis de valider et de mesurer l'efficacité du système proposé par rapport à sa capacité de se mettre à l'échelle (Scalabilité) pour prendre en charge, efficacement, les flux massifs de données 3D. Ces expérimentations ont aussi permis de mettre en place des démarches d'optimisation permettant une meilleure performance dans la production et la diffusion des tuiles 3D.Every year, floods present a major challenge for Quebec authorities. Thus, the development of new tools and methods to disseminate and visualize massive 3D dynamic flood data is very important to better understand and manage flood-related risks. This research focuses on the streaming of massive 3D geospatial data (3D building models, trees, digital elevation models (DEM), LiDAR data, aerial imagery, etc.) related to flooding. The problem is that there is no efficient streaming systems in the literature for massive 3D data adapted to the needs of this research. In this sense, our general objective is to develop a tool for the streaming of massive 3D geospatial data which are 3D buildings and high-resolution terrain models at the scale of the province of Quebec. The challenges of streaming massive data lead us to adopt the 3D tiling technique to convert raw data into lighter vector formats and structures suitable for streaming such as the "3D Tiles" specification to tile 3D buildings, LiDAR point clouds and other 3D georeferenced models and irregular meshes, including TIN, to tile digital terrain models. Also, the use of parallel processing techniques allows efficient management of massive data flows and improve processing time allowing the scalability and the flexibility of existing systems. For this purpose, two tiling pipelines have been developed. The first pipeline is for creating 3D building tiles according to the "3D Tiles" specification. The second is for creating terrain tiles based on irregular meshes. These pipelines are then integrated into a distributed processing system based on Docker containers in order to parallelize the treatment processes. To test the efficiency and validity of the developed system, we tested this system on a massive dataset of about 2.5 million 3D buildings located in Quebec. These experiments allowed us to validate and measure the efficiency of the proposed system to be scalable in order to efficiently handle massive 3D data flows. These experiments also allowed to set up optimization approaches allowing a better performance in the production and the streaming of 3D tiles