19 research outputs found
Performance Evaluation of Distributed Computing Environments with Hadoop and Spark Frameworks
Recently, due to rapid development of information and communication
technologies, the data are created and consumed in the avalanche way.
Distributed computing create preconditions for analyzing and processing such
Big Data by distributing the computations among a number of compute nodes. In
this work, performance of distributed computing environments on the basis of
Hadoop and Spark frameworks is estimated for real and virtual versions of
clusters. As a test task, we chose the classic use case of word counting in
texts of various sizes. It was found that the running times grow very fast with
the dataset size and faster than a power function even. As to the real and
virtual versions of cluster implementations, this tendency is the similar for
both Hadoop and Spark frameworks. Moreover, speedup values decrease
significantly with the growth of dataset size, especially for virtual version
of cluster configuration. The problem of growing data generated by IoT and
multimodal (visual, sound, tactile, neuro and brain-computing, muscle and eye
tracking, etc.) interaction channels is presented. In the context of this
problem, the current observations as to the running times and speedup on Hadoop
and Spark frameworks in real and virtual cluster configurations can be very
useful for the proper scaling-up and efficient job management, especially for
machine learning and Deep Learning applications, where Big Data are widely
present.Comment: 5 pages, 1 table, 2017 IEEE International Young Scientists Forum on
Applied Physics and Engineering (YSF-2017) (Lviv, Ukraine
Contributions of architecture Dew Computing to the Internet of Things: comparisons between pilot implementations of both architectures
Dew computing o虂 la computaci贸n de roc铆o o l谩grima ha despertado gran inter茅s en la academia, debido a la separaci贸n de los procesos de computaci贸n distribuida; donde se encuentran las capas de cloud Computing (computaci贸n en la nube), Fog Computing (computaci贸n de niebla), Edge Computing (computaci贸n de borde) y por 煤ltimo Dew Computing. Estas capas est谩n mencionadas de orden descendente (de mayor a menor) siendo Dew Computing la m谩s cercana al usuario final. Esto se realiza para una mayor comprensi贸n entre las tecnolog铆as y procesos que en ellas se realizan permitiendo su diferenciaci贸n.
La arquitectura de Internet of Things (IoT) es un paradigma tecnol贸gico que se est谩 formando dentro del ecosistema de computaci贸n distribuida, por ende, se requiere resaltar la capa de Dew Computing y su aporte al modelo tecnol贸gico.
Es por esto, que se realiza un estado del arte de las arquitecturas Dew Computing e IoT que permitan su comparaci贸n con el fin de saber su aporte de forma independiente y en dado caso, c贸mo podr铆an integrarse.
Se realiza una prueba piloto entre las arquitecturas y una integraci贸n de las misma para encontrar los aportes que un modelo del entrega al otro y por 煤ltimo, se plantean posibles escenarios de aplicaci贸n que evidencien los beneficios y d茅ficit de la implementaci贸n de cada arquitectura en diferentes 谩mbitos sociales.INTRODUCCI脫N
1. PROBLEMA, PREGUNTA E HIP脫TESIS DE INVESTIGACI脫N 11
2. JUSTIFICACI脫N 11
3. OBJETIVOS DEL PROYECTO 13
3.1 OBJETIVO GENERAL 13
3.2 OBJETIVOS ESPEC脥FICOS 13
4. MARCO REFERENCIAL 14
4.1 MARCO CONCEPTUAL 14
4.1.1 Internet of Things 15
4.1.2 Cloud Computing 15
4.1.3 Fog Computing 16
4.1.4 Edge Computing 17
4.1.5 Dew Computing 20
4.2 MARCO TE脫RICO 21
4.3 ESTADO DEL ARTE 22
4.3.1 Revisi贸n sistem谩tica de la literatura 22
4.3.2 An谩lisis estado del arte 28
4.4 MARCO CONTEXTUAL Y ANTECEDENTES 28
4.5 NORMAS Y EST脕NDARES 29
4.5.1 Normatividad colombiana 29
4.5.2 Est谩ndares y documentos de referencia 30
4.6 EMPRESAS TECNOL脫GICAS 31
4.6.1 Microsoft Azure IoT Edge 31
4.6.2 Amazon IoT GreenGrass 32
5. DESCRIPCI脫N DEL PROCESO INVESTIGATIVO 34
5.1 ENFOQUE Y TIPO DE INVESTIGACI脫N 34
5.2 FASES Y ACTIVIDADES 34
5.2.1 Elaboraci贸n del estado del arte de Dew computing 35
5.2.2 An谩lisis comparativo entre frameworks para Dew Computing 35
5.2.3 Dispositivo para pruebas
36
5.2.4 Pruebas de ambas arquitecturas 40
5.2.5 An谩lisis de pruebas
45
6. RESULTADOS 48
6.1 REVISI脫N COMPARATIVA DE DEW COMPUTING E IOT 48
6.2 VENTAJAS Y DESVENTAJAS DE DEW COMPUTING CON IOT. 52
6.2.1 F铆sica 53
6.2.2 Econom铆a 54
6.2.3 Ubicaci贸n 54
6.3 OPORTUNIDADES QUE BRINDA DEW COMPUTING 55
6.3.1 Manejo de la energ铆a 55
6.3.2 Procesamiento 55
6.3.3 Almacenamiento 55
6.3.4 Protocolos de comunicaci贸n 55
6.3.5 Lenguajes de programaci贸n 55
6.3.6 Seguridad de los datos 56
6.3.7 Visualizaci贸n de los datos 56
7. CONCLUSIONES Y RECOMENDACIONES 57
8. REFERENCIAS 58Maestr铆aDew Computing or the dew or tear computation has aroused considerable interest in the academy, due to the separation of the processes of distributed computing; where are the layers of Cloud Computing (cloud computing), Fog Computing (fog computing), Edge Computing (edge computing) and finally Dew Computing. These layers are mentioned in descending order (from highest to lowest) with Dew Computing being the closest to the end user. This is done for a better understanding of the technologies and processes that are carried out in them, allowing their differentiation
Comparative Analysis of Open Source Frameworks for Machine Learning with Use Case in Single-Threaded and Multi-Threaded Modes
The basic features of some of the most versatile and popular open source
frameworks for machine learning (TensorFlow, Deep Learning4j, and H2O) are
considered and compared. Their comparative analysis was performed and
conclusions were made as to the advantages and disadvantages of these
platforms. The performance tests for the de facto standard MNIST data set were
carried out on H2O framework for deep learning algorithms designed for CPU and
GPU platforms for single-threaded and multithreaded modes of operation.Comment: 4 pages, 6 figures, 4 tables; XIIth International Scientific and
Technical Conference on Computer Sciences and Information Technologies (CSIT
2017), Lviv, Ukrain
Open Source Dataset and Machine Learning Techniques for Automatic Recognition of Historical Graffiti
Machine learning techniques are presented for automatic recognition of the
historical letters (XI-XVIII centuries) carved on the stoned walls of St.Sophia
cathedral in Kyiv (Ukraine). A new image dataset of these carved Glagolitic and
Cyrillic letters (CGCL) was assembled and pre-processed for recognition and
prediction by machine learning methods. The dataset consists of more than 4000
images for 34 types of letters. The explanatory data analysis of CGCL and
notMNIST datasets shown that the carved letters can hardly be differentiated by
dimensionality reduction methods, for example, by t-distributed stochastic
neighbor embedding (tSNE) due to the worse letter representation by stone
carving in comparison to hand writing. The multinomial logistic regression
(MLR) and a 2D convolutional neural network (CNN) models were applied. The MLR
model demonstrated the area under curve (AUC) values for receiver operating
characteristic (ROC) are not lower than 0.92 and 0.60 for notMNIST and CGCL,
respectively. The CNN model gave AUC values close to 0.99 for both notMNIST and
CGCL (despite the much smaller size and quality of CGCL in comparison to
notMNIST) under condition of the high lossy data augmentation. CGCL dataset was
published to be available for the data science community as an open source
resource.Comment: 11 pages, 9 figures, accepted for 25th International Conference on
Neural Information Processing (ICONIP 2018), 14-16 December, 2018 (Siem Reap,
Cambodia
Deep Learning for Fatigue Estimation on the Basis of Multimodal Human-Machine Interactions
The new method is proposed to monitor the level of current physical load and
accumulated fatigue by several objective and subjective characteristics. It was
applied to the dataset targeted to estimate the physical load and fatigue by
several statistical and machine learning methods. The data from peripheral
sensors (accelerometer, GPS, gyroscope, magnetometer) and brain-computing
interface (electroencephalography) were collected, integrated, and analyzed by
several statistical and machine learning methods (moment analysis, cluster
analysis, principal component analysis, etc.). The hypothesis 1 was presented
and proved that physical activity can be classified not only by objective
parameters, but by subjective parameters also. The hypothesis 2 (experienced
physical load and subsequent restoration as fatigue level can be estimated
quantitatively and distinctive patterns can be recognized) was presented and
some ways to prove it were demonstrated. Several "physical load" and "fatigue"
metrics were proposed. The results presented allow to extend application of the
machine learning methods for characterization of complex human activity
patterns (for example, to estimate their actual physical load and fatigue, and
give cautions and advice).Comment: 12 pages, 10 figures, 1 table; presented at XXIX IUPAP Conference in
Computational Physics (CCP2017) July 9-13, 2017, Paris, University Pierre et
Marie Curie - Sorbonne (https://ccp2017.sciencesconf.org/program