Performance Evaluation of Distributed Computing Environments with Hadoop and Spark Frameworks

Recently, due to rapid development of information and communication technologies, the data are created and consumed in the avalanche way. Distributed computing create preconditions for analyzing and processing such Big Data by distributing the computations among a number of compute nodes. In this work, performance of distributed computing environments on the basis of Hadoop and Spark frameworks is estimated for real and virtual versions of clusters. As a test task, we chose the classic use case of word counting in texts of various sizes. It was found that the running times grow very fast with the dataset size and faster than a power function even. As to the real and virtual versions of cluster implementations, this tendency is the similar for both Hadoop and Spark frameworks. Moreover, speedup values decrease significantly with the growth of dataset size, especially for virtual version of cluster configuration. The problem of growing data generated by IoT and multimodal (visual, sound, tactile, neuro and brain-computing, muscle and eye tracking, etc.) interaction channels is presented. In the context of this problem, the current observations as to the running times and speedup on Hadoop and Spark frameworks in real and virtual cluster configurations can be very useful for the proper scaling-up and efficient job management, especially for machine learning and Deep Learning applications, where Big Data are widely present.Comment: 5 pages, 1 table, 2017 IEEE International Young Scientists Forum on Applied Physics and Engineering (YSF-2017) (Lviv, Ukraine

Contributions of architecture Dew Computing to the Internet of Things: comparisons between pilot implementations of both architectures

Dew computing ó la computación de rocío o lágrima ha despertado gran interés en la academia, debido a la separación de los procesos de computación distribuida; donde se encuentran las capas de cloud Computing (computación en la nube), Fog Computing (computación de niebla), Edge Computing (computación de borde) y por último Dew Computing. Estas capas están mencionadas de orden descendente (de mayor a menor) siendo Dew Computing la más cercana al usuario final. Esto se realiza para una mayor comprensión entre las tecnologías y procesos que en ellas se realizan permitiendo su diferenciación. La arquitectura de Internet of Things (IoT) es un paradigma tecnológico que se está formando dentro del ecosistema de computación distribuida, por ende, se requiere resaltar la capa de Dew Computing y su aporte al modelo tecnológico. Es por esto, que se realiza un estado del arte de las arquitecturas Dew Computing e IoT que permitan su comparación con el fin de saber su aporte de forma independiente y en dado caso, cómo podrían integrarse. Se realiza una prueba piloto entre las arquitecturas y una integración de las misma para encontrar los aportes que un modelo del entrega al otro y por último, se plantean posibles escenarios de aplicación que evidencien los beneficios y déficit de la implementación de cada arquitectura en diferentes ámbitos sociales.INTRODUCCIÓN 1. PROBLEMA, PREGUNTA E HIPÓTESIS DE INVESTIGACIÓN 11 2. JUSTIFICACIÓN 11 3. OBJETIVOS DEL PROYECTO 13 3.1 OBJETIVO GENERAL 13 3.2 OBJETIVOS ESPECÍFICOS 13 4. MARCO REFERENCIAL 14 4.1 MARCO CONCEPTUAL 14 4.1.1 Internet of Things 15 4.1.2 Cloud Computing 15 4.1.3 Fog Computing 16 4.1.4 Edge Computing 17 4.1.5 Dew Computing 20 4.2 MARCO TEÓRICO 21 4.3 ESTADO DEL ARTE 22 4.3.1 Revisión sistemática de la literatura 22 4.3.2 Análisis estado del arte 28 4.4 MARCO CONTEXTUAL Y ANTECEDENTES 28 4.5 NORMAS Y ESTÁNDARES 29 4.5.1 Normatividad colombiana 29 4.5.2 Estándares y documentos de referencia 30 4.6 EMPRESAS TECNOLÓGICAS 31 4.6.1 Microsoft Azure IoT Edge 31 4.6.2 Amazon IoT GreenGrass 32 5. DESCRIPCIÓN DEL PROCESO INVESTIGATIVO 34 5.1 ENFOQUE Y TIPO DE INVESTIGACIÓN 34 5.2 FASES Y ACTIVIDADES 34 5.2.1 Elaboración del estado del arte de Dew computing 35 5.2.2 Análisis comparativo entre frameworks para Dew Computing 35 5.2.3 Dispositivo para pruebas 36 5.2.4 Pruebas de ambas arquitecturas 40 5.2.5 Análisis de pruebas 45 6. RESULTADOS 48 6.1 REVISIÓN COMPARATIVA DE DEW COMPUTING E IOT 48 6.2 VENTAJAS Y DESVENTAJAS DE DEW COMPUTING CON IOT. 52 6.2.1 Física 53 6.2.2 Economía 54 6.2.3 Ubicación 54 6.3 OPORTUNIDADES QUE BRINDA DEW COMPUTING 55 6.3.1 Manejo de la energía 55 6.3.2 Procesamiento 55 6.3.3 Almacenamiento 55 6.3.4 Protocolos de comunicación 55 6.3.5 Lenguajes de programación 55 6.3.6 Seguridad de los datos 56 6.3.7 Visualización de los datos 56 7. CONCLUSIONES Y RECOMENDACIONES 57 8. REFERENCIAS 58MaestríaDew Computing or the dew or tear computation has aroused considerable interest in the academy, due to the separation of the processes of distributed computing; where are the layers of Cloud Computing (cloud computing), Fog Computing (fog computing), Edge Computing (edge computing) and finally Dew Computing. These layers are mentioned in descending order (from highest to lowest) with Dew Computing being the closest to the end user. This is done for a better understanding of the technologies and processes that are carried out in them, allowing their differentiation

Comparative Analysis of Open Source Frameworks for Machine Learning with Use Case in Single-Threaded and Multi-Threaded Modes

The basic features of some of the most versatile and popular open source frameworks for machine learning (TensorFlow, Deep Learning4j, and H2O) are considered and compared. Their comparative analysis was performed and conclusions were made as to the advantages and disadvantages of these platforms. The performance tests for the de facto standard MNIST data set were carried out on H2O framework for deep learning algorithms designed for CPU and GPU platforms for single-threaded and multithreaded modes of operation.Comment: 4 pages, 6 figures, 4 tables; XIIth International Scientific and Technical Conference on Computer Sciences and Information Technologies (CSIT 2017), Lviv, Ukrain

Open Source Dataset and Machine Learning Techniques for Automatic Recognition of Historical Graffiti

Machine learning techniques are presented for automatic recognition of the historical letters (XI-XVIII centuries) carved on the stoned walls of St.Sophia cathedral in Kyiv (Ukraine). A new image dataset of these carved Glagolitic and Cyrillic letters (CGCL) was assembled and pre-processed for recognition and prediction by machine learning methods. The dataset consists of more than 4000 images for 34 types of letters. The explanatory data analysis of CGCL and notMNIST datasets shown that the carved letters can hardly be differentiated by dimensionality reduction methods, for example, by t-distributed stochastic neighbor embedding (tSNE) due to the worse letter representation by stone carving in comparison to hand writing. The multinomial logistic regression (MLR) and a 2D convolutional neural network (CNN) models were applied. The MLR model demonstrated the area under curve (AUC) values for receiver operating characteristic (ROC) are not lower than 0.92 and 0.60 for notMNIST and CGCL, respectively. The CNN model gave AUC values close to 0.99 for both notMNIST and CGCL (despite the much smaller size and quality of CGCL in comparison to notMNIST) under condition of the high lossy data augmentation. CGCL dataset was published to be available for the data science community as an open source resource.Comment: 11 pages, 9 figures, accepted for 25th International Conference on Neural Information Processing (ICONIP 2018), 14-16 December, 2018 (Siem Reap, Cambodia

Deep Learning for Fatigue Estimation on the Basis of Multimodal Human-Machine Interactions

The new method is proposed to monitor the level of current physical load and accumulated fatigue by several objective and subjective characteristics. It was applied to the dataset targeted to estimate the physical load and fatigue by several statistical and machine learning methods. The data from peripheral sensors (accelerometer, GPS, gyroscope, magnetometer) and brain-computing interface (electroencephalography) were collected, integrated, and analyzed by several statistical and machine learning methods (moment analysis, cluster analysis, principal component analysis, etc.). The hypothesis 1 was presented and proved that physical activity can be classified not only by objective parameters, but by subjective parameters also. The hypothesis 2 (experienced physical load and subsequent restoration as fatigue level can be estimated quantitatively and distinctive patterns can be recognized) was presented and some ways to prove it were demonstrated. Several "physical load" and "fatigue" metrics were proposed. The results presented allow to extend application of the machine learning methods for characterization of complex human activity patterns (for example, to estimate their actual physical load and fatigue, and give cautions and advice).Comment: 12 pages, 10 figures, 1 table; presented at XXIX IUPAP Conference in Computational Physics (CCP2017) July 9-13, 2017, Paris, University Pierre et Marie Curie - Sorbonne (https://ccp2017.sciencesconf.org/program