473,716 research outputs found
Data management for production quality deep learning models: Challenges and solutions
Deep learning (DL) based software systems are difficult to develop and maintain in industrial settings due to several challenges. Data management is one of the most prominent challenges which complicates DL in industrial deployments. DL models are data-hungry and require high-quality data. Therefore, the volume, variety, velocity, and quality of data cannot be compromised. This study aims to explore the data management challenges encountered by practitioners developing systems with DL components, identify the potential solutions from the literature and validate the solutions through a multiple case study. We identified 20 data management challenges experienced by DL practitioners through a multiple interpretive case study. Further, we identified 48 articles through a systematic literature review that discuss the solutions for the data management challenges. With the second round of multiple case study, we show that many of these solutions have limitations and are not used in practice due to a combination of four factors: high cost, lack of skill-set and infrastructure, inability to solve the problem completely, and incompatibility with certain DL use cases. Thus, data management for data-intensive DL models in production is complicated. Although the DL technology has achieved very promising results, there is still a significant need for further research in the field of data management to build high-quality datasets and streams that can be used for building production-ready DL systems. Furthermore, we have classified the data management challenges into four categories based on the availability of the solutions.(c) 2022 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Deep Learning for Network Traffic Monitoring and Analysis (NTMA): A Survey
Modern communication systems and networks, e.g., Internet of Things (IoT) and cellular networks, generate a massive and heterogeneous amount of traffic data. In such networks, the traditional network management techniques for monitoring and data analytics face some challenges and issues, e.g., accuracy, and effective processing of big data in a real-time fashion. Moreover, the pattern of network traffic, especially in cellular networks, shows very complex behavior because of various factors, such as device mobility and network heterogeneity. Deep learning has been efficiently employed to facilitate analytics and knowledge discovery in big data systems to recognize hidden and complex patterns. Motivated by these successes, researchers in the field of networking apply deep learning models for Network Traffic Monitoring and Analysis (NTMA) applications, e.g., traffic classification and prediction. This paper provides a comprehensive review on applications of deep learning in NTMA. We first provide fundamental background relevant to our review. Then, we give an insight into the confluence of deep learning and NTMA, and review deep learning techniques proposed for NTMA applications. Finally, we discuss key challenges, open issues, and future research directions for using deep learning in NTMA applications.publishedVersio
BigDL: A Distributed Deep Learning Framework for Big Data
This paper presents BigDL (a distributed deep learning framework for Apache
Spark), which has been used by a variety of users in the industry for building
deep learning applications on production big data platforms. It allows deep
learning applications to run on the Apache Hadoop/Spark cluster so as to
directly process the production data, and as a part of the end-to-end data
analysis pipeline for deployment and management. Unlike existing deep learning
frameworks, BigDL implements distributed, data parallel training directly on
top of the functional compute model (with copy-on-write and coarse-grained
operations) of Spark. We also share real-world experience and "war stories" of
users that have adopted BigDL to address their challenges(i.e., how to easily
build end-to-end data analysis and deep learning pipelines for their production
data).Comment: In ACM Symposium of Cloud Computing conference (SoCC) 201
The Dreaming Variational Autoencoder for Reinforcement Learning Environments
Reinforcement learning has shown great potential in generalizing over raw
sensory data using only a single neural network for value optimization. There
are several challenges in the current state-of-the-art reinforcement learning
algorithms that prevent them from converging towards the global optima. It is
likely that the solution to these problems lies in short- and long-term
planning, exploration and memory management for reinforcement learning
algorithms. Games are often used to benchmark reinforcement learning algorithms
as they provide a flexible, reproducible, and easy to control environment.
Regardless, few games feature a state-space where results in exploration,
memory, and planning are easily perceived. This paper presents The Dreaming
Variational Autoencoder (DVAE), a neural network based generative modeling
architecture for exploration in environments with sparse feedback. We further
present Deep Maze, a novel and flexible maze engine that challenges DVAE in
partial and fully-observable state-spaces, long-horizon tasks, and
deterministic and stochastic problems. We show initial findings and encourage
further work in reinforcement learning driven by generative exploration.Comment: Best Student Paper Award, Proceedings of the 38th SGAI International
Conference on Artificial Intelligence, Cambridge, UK, 2018, Artificial
Intelligence XXXV, 201
Deep Learning Approaches in Pavement Distress Identification: A Review
This paper presents a comprehensive review of recent advancements in image
processing and deep learning techniques for pavement distress detection and
classification, a critical aspect in modern pavement management systems. The
conventional manual inspection process conducted by human experts is gradually
being superseded by automated solutions, leveraging machine learning and deep
learning algorithms to enhance efficiency and accuracy. The ability of these
algorithms to discern patterns and make predictions based on extensive datasets
has revolutionized the domain of pavement distress identification. The paper
investigates the integration of unmanned aerial vehicles (UAVs) for data
collection, offering unique advantages such as aerial perspectives and
efficient coverage of large areas. By capturing high-resolution images, UAVs
provide valuable data that can be processed using deep learning algorithms to
detect and classify various pavement distresses effectively. While the primary
focus is on 2D image processing, the paper also acknowledges the challenges
associated with 3D images, such as sensor limitations and computational
requirements. Understanding these challenges is crucial for further
advancements in the field. The findings of this review significantly contribute
to the evolution of pavement distress detection, fostering the development of
efficient pavement management systems. As automated approaches continue to
mature, the implementation of deep learning techniques holds great promise in
ensuring safer and more durable road infrastructure for the benefit of society
- …