73,792 research outputs found
Regional Data Archiving and Management for Northeast Illinois
This project studies the feasibility and implementation options for establishing a regional data archiving system to help monitor
and manage traffic operations and planning for the northeastern Illinois region. It aims to provide a clear guidance to the
regional transportation agencies, from both technical and business perspectives, about building such a comprehensive
transportation information system. Several implementation alternatives are identified and analyzed. This research is carried
out in three phases.
In the first phase, existing documents related to ITS deployments in the broader Chicago area are summarized, and a
thorough review is conducted of similar systems across the country. Various stakeholders are interviewed to collect
information on all data elements that they store, including the format, system, and granularity. Their perception of a data
archive system, such as potential benefits and costs, is also surveyed. In the second phase, a conceptual design of the
database is developed. This conceptual design includes system architecture, functional modules, user interfaces, and
examples of usage. In the last phase, the possible business models for the archive system to sustain itself are reviewed. We
estimate initial capital and recurring operational/maintenance costs for the system based on realistic information on the
hardware, software, labor, and resource requirements. We also identify possible revenue opportunities.
A few implementation options for the archive system are summarized in this report; namely:
1. System hosted by a partnering agency
2. System contracted to a university
3. System contracted to a national laboratory
4. System outsourced to a service provider
The costs, advantages and disadvantages for each of these recommended options are also provided.ICT-R27-22published or submitted for publicationis peer reviewe
Data locality in Hadoop
Current market tendencies show the need of storing and processing rapidly
growing amounts of data. Therefore, it implies the demand for distributed
storage and data processing systems. The Apache Hadoop is an open-source
framework for managing such computing clusters in an effective, fault-tolerant
way.
Dealing with large volumes of data, Hadoop, and its storage system HDFS
(Hadoop Distributed File System), face challenges to keep the high efficiency
with computing in a reasonable time. The typical Hadoop implementation
transfers computation to the data, rather than shipping data across the cluster.
Otherwise, moving the big quantities of data through the network could significantly
delay data processing tasks. However, while a task is already running,
Hadoop favours local data access and chooses blocks from the nearest nodes.
Next, the necessary blocks are moved just when they are needed in the given
ask.
For supporting the Hadoop’s data locality preferences, in this thesis, we propose
adding an innovative functionality to its distributed file system (HDFS), that
enables moving data blocks on request. In-advance shipping of data makes it
possible to forcedly redistribute data between nodes in order to easily adapt it to
the given processing tasks. New functionality enables the instructed movement
of data blocks within the cluster. Data can be shifted either by user running
the proper HDFS shell command or programmatically by other module like an
appropriate scheduler.
In order to develop such functionality, the detailed analysis of Apache Hadoop
source code and its components (specifically HDFS) was conducted. Research
resulted in a deep understanding of internal architecture, what made it possible
to compare the possible approaches to achieve the desired solution, and develop
the chosen one
EOS Data and Information System (EOSDIS)
In the past decade, science and technology have reached levels that permit assessments of global environmental change. Scientific success in understanding global environmental change depends on integration and management of numerous data sources. The Global Change Data and Information System (GCDIS) must provide for the management of data, information dissemination, and technology transfer. The Earth Observing System Data and Information System (EOSDIS) is NASA's portion of this global change information system
Integrating an agent-based wireless sensor network within an existing multi-agent condition monitoring system
The use of wireless sensor networks for condition monitoring is gaining ground across all sectors of industry, and while their use for power engineering applications has yet been limited, they represent a viable platform for next-generation substation condition monitoring systems. For engineers to fully benefit from this new approach to condition monitoring, new sensor data must be incorporated into a single integrated system. This paper proposes the integration of an agent-based wireless sensor network with an existing agent-based condition monitoring system. It demonstrates that multi-agent systems can be extended down to the sensor level while considering the reduced energy availability of low-power embedded devices. A novel agent-based approach to data translation is presented, which is demonstrated through two case studies: a lab-based temperature and vibration monitoring system, and a proposal to integrate a wireless sensor network to an existing technology demonstrator deployed in a substation in the UK
ORAC-DR: A generic data reduction pipeline infrastructure
ORAC-DR is a general purpose data reduction pipeline system designed to be
instrument and observatory agnostic. The pipeline works with instruments as
varied as infrared integral field units, imaging arrays and spectrographs, and
sub-millimeter heterodyne arrays & continuum cameras. This paper describes the
architecture of the pipeline system and the implementation of the core
infrastructure. We finish by discussing the lessons learned since the initial
deployment of the pipeline system in the late 1990s.Comment: 11 pages, 1 figure, accepted for publication in Astronomy and
Computin
Digital Preservation Services : State of the Art Analysis
Research report funded by the DC-NET project.An overview of the state of the art in service provision for digital preservation and curation. Its focus is on the areas where bridging the gaps is needed between e-Infrastructures and efficient and forward-looking digital preservation services. Based on a desktop study and a rapid analysis of some 190 currently available tools and services for digital preservation, the deliverable provides a high-level view on the range of instruments currently on offer to support various functions within a preservation system.European Commission, FP7peer-reviewe
A distributed architecture to implement a prognostic function for complex systems
The proactivity in maintenance management is improved by the implementation of CBM (Condition-Based Maintenance) principles and of PHM (Prognostic and Health Management). These implementations use data about the health status of the systems. Among them, prognostic data make it possible to evaluate the future health of the systems. The Remaining Useful Lifetimes (RULs) of the components is frequently required to prognose systems. However, the availability of complex systems for productive tasks is often expressed in terms of RULs of functions and/or subsystems; those RULs have to bring information about the components. Indeed, the maintenance operators must know what components need maintenance actions in order to increase the RULs of the functions or subsystems, and consequently the availability of the complex systems for longer tasks or more productive tasks. This paper aims at defining a generic prognostic function of complex systems aiming at prognosing its functions and at enabling the isolation of components that needs maintenance actions. The proposed function requires knowledge about the system to be prognosed. The corresponding models are detailed. The proposed prognostic function contains graph traversal so its distribution is proposed to speed it up. It is carried out by generic agents
- …