A Study of Realtime Summarization Metrics

Unexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for quickly transforming this overabundance of text into clear, useful information for end-users including affected individuals, crisis responders, and interested third parties. Despite the importance of real-time summarization systems, their evaluation is not well understood as classic methods for text summarization are inappropriate for real-time and streaming conditions. The TREC 2013-2015 Temporal Summarization (TREC-TS) track was one of the first evaluation campaigns to tackle the challenges of real-time summarization evaluation, introducing new metrics, ground-truth generation methodology and dataset. In this paper, we present a study of TREC-TS track evaluation methodology, with the aim of documenting its design, analyzing its effectiveness, as well as identifying improvements and best practices for the evaluation of temporal summarization systems

An Experimental Methodology to Evaluate Concept Generation Procedures Based on Quantitative Lifecycle Performance

This study presents an experimental methodology to measure how concept generation procedures can affect the anticipated lifecycle performance of engineering systems design concepts. The methodology is based on objective and quantitative measurements of anticipated lifecycle performance of the design concepts. It merges cognitive and computer-aided techniques from the fields of collaboration engineering, creativity, and engineering design. It complements the body of existing techniques relying on subjective expert assessments, and other objective metrics not explicitly measuring anticipated lifecycle performance (e.g. development time and cost). Application of the methodology is demonstrated through evaluation of design procedures generating flexibility in engineering systems design. Experiments had ninety participants generate creative design alternatives to a simplified real estate development design problem. Thirty-two teams of two to three participants performed the collaborative design exercise. An online Group-Support System interface enabled efficient data collection and analysis. A computationally efficient mid-fidelity model was used to evaluate flexible design concepts quantitatively based on real options analysis techniques.Massachusetts Institute of Technology. Center for Real EstateNatural Sciences and Engineering Research Council of CanadaMassachusetts Institute of Technology. Engineering Systems DivisionSingapore University of Technology and Design. International Design Cente

SIMULATION OF A MULTIPROCESSOR COMPUTER SYSTEM

The introduction of computers and software engineering in telephone switching systems has dictated the need for powerful design aids for such complex systems. Among these design aids simulators - real-time environment simulators and flat-level simulators - have been found particularly useful in stored program controlled switching systems design and evaluation. However, both types of simulators suffer from certain disadvantages. An alternative methodology to the simulation of stored program controlled switching systems is proposed in this research. The methodology is based on the development of a process-based multilevel hierarchically structured software simulator. This methodology eliminates the disadvantages of environment and flat-level simulators. It enables the modelling of the system in a 1 to 1 transformation process retaining the sub-systems interfaces and, hence, making it easier to see the resemblance between the model and modelled system and to incorporate design modifications and/or additions in the simulator. This methodology has been applied in building a simulation package for the System X family of exchanges. The Processor Utility Sub-system used to control the exchanges is first simulated, verified and validated. The application sub-systems models are then added one level higher_, resulting in an open-ended simulator having sub-systems models at different levels of detail and capable of simulating any member of the System X family of exchanges. The viability of the methodology is demonstrated by conducting experiments to tune the real-time operating system and by simulating a particular exchange - The Digital Main Network Switching Centre - in order to determine its performance characteristics.The General Electric Company Ltd, GEC Hirst Research Cent, Wemble

A Hybrid Modelling Framework for Real-time Decision-support for Urgent and Emergency Healthcare

In healthcare, opportunities to use real-time data to support quick and effective decision-making are expanding rapidly, as data increases in volume, velocity and variety. In parallel, the need for short-term decision-support to improve system resilience is increasingly relevant, with the recent COVID-19 crisis underlining the pressure that our healthcare services are under to deliver safe, effective, quality care in the face of rapidly-shifting parameters. A real-time hybrid model (HM) which combines real-time data, predictions, and simulation, has the potential to support short-term decision-making in healthcare. Considering decision-making as a consequence of situation awareness focuses the HM on what information is needed where, when, how, and by whom with a view toward sustained implementation. However the articulation between real-time decision-support tools and a sociotechnical approach to their development and implementation is currently lacking in the literature. Having identified the need for a conceptual framework to support the development of real-time HMs for short-term decision-support, this research proposed and tested the Integrated Hybrid Analytics Framework (IHAF) through an examination of the stages of a Design Science methodology and insights from the literature examining decision-making in dynamic, sociotechnical systems, data analytics, and simulation. Informed by IHAF, a HM was developed using real-time Emergency Department data, time-series forecasting, and discrete-event simulation. The application started with patient questionnaires to support problem definition and to act as a formative evaluation, and was subsequently evaluated using staff interviews. Evaluation of the application found multiple examples where the objectives of people or sub-systems are not aligned, resulting in inefficiencies and other quality problems, which are characteristic of complex adaptive sociotechnical systems. Synthesis of the literature, the formative evaluation, and the final evaluation found significant themes which can act as antecedents or evaluation criteria for future real-time HM studies in sociotechnical systems, in particular in healthcare. The generic utility of IHAF is emphasised for supporting future applications in similar domains

Towards Effective Training for Process and Maritime Industries

AbstractWith the development of technological innovation, many industries such as process and maritime industries constitute more complex automated systems, dedicated operating conditions, multilevel interconnections, as well as human-machine and human-human interactions. The actions and decisions made by operators and seafarers affect the safety and productivity of these systems. It is well known that around 60-80% of the accidents are attributed to human error (directly or indirectly). Generally, each operator or seafarer is must undergo training before starting their real jobs. In recent decades, emphasis on operator training has increased – resulting in advance training simulators with several features like immersivity, stereoscopic sounds, hydraulics, and even use of different odors. However, the design of training methodology can have a significant impact on the skill acquisition of trainees. The growth of integration of technology in existing systems as well as newer systems is much higher than that of improvement in training methods. Unfortunately, there are few studies linking the training needs and the real-world demands on operators, revealing a significant research gap to fulfill.In socio-complex systems, it is necessary to consider various aspects in the training methodology, which can facilitate the operators/seafarers to handle normal and abnormal scenarios adequately. This paper provides a background of current training methods through analyzing the process- and maritime industries as illustrative examples, highlighting the limitations associated with each of different perspectives (technical, psychological and organizational) to propose an training syllabus that allows for learning by experience and interaction with scenarios of different complexity. It consists of a three-stage hierarchy with increasing demands concerning technical and relational complexity and time pressure. The training is centered on handling real-time operations with increasing complexity starting from basic components of the process, advancing to real-time operations, and reaching high technical and relational complexity that needs to be handled in situations with limited time and uncertainty in data. The challenges that arise in team-working tasks are also considered in the conceptualization of the training syllabus. The proposed training syllabus includes training content, objectives and performance evaluation criteria. The systematic methodology of performance evaluation will allow practitioners to obtain transparent, unbiased and consistent certification of trainees

Performance Analysis of the ARIA Adaptive Media Processing Workflows using Colored Petri Nets

AbstractMultimedia systems are one of the most complex and interesting applications that are nowadays proposed to the users. Their complexity derives mainly from the fact that multimedia systems have to process huge amounts of data, while respecting real-time deadlines. For this reason performance evaluation of the underlaying workflow is a key issue in the design process of a new Multimedia system.In this paper we consider the ARchitecture for Interactive Arts (ARIA), an adaptive media processing workflow, developed at the Arizona State University, and outline a semi-automatic procedure to translate its specification into Colored Petri Nets. We then provide guidelines on how to compute the parameters for the performance models, and apply the proposed methodology to a realistic example of a face recognition application

Performance Evaluation of a Wireless Network using a VoIP Traffic Generator on a Mobile Device

The problem of generating different patterns of traffic to emulate real user behaviour is receiving considerable attention with the construction of new and more complex network architectures. The theoretical modelling of waveforms or signals that flow through networks is valuable in a variety of scenarios including performance analysis and the design of communication systems. In the literature, many computer-based performance evaluation tools have been discussed. However, these tools lack the ability to run on affordable technologies such as mobile phones. The fundamental contribution of this work is the design of a traffic generating tool called MTGawn which is able to run on a mobile device. Design Science Research was the research methodology used for the design and deployment of a prototype of the proposed system. VoIP traffic was emulated using an implementation of well-known real time transport protocols such as RTP and cRTP, and parameterization was defined by using three codecs namely: G.711, G.723, and G.729. An evaluation was performed in a laboratory wireless network testbed and preliminary results were collected and analysed. The results of the experiments show that such a measuring instrument can be deployed on a mobile phone. More experiments are being done to ensure the accuracy of the data and also to compare the results with that of computer-based systems. Furthermore additional functionalities, similar to the functionality found on the computer-based open source tools, are being added to the mobile tool.Telkom, Cisco, Aria Technologies, THRIPDepartment of HE and Training approved lis

Product and process design optimization by quality engineering /

This research is concerned with product and process design optimization by quality engineering based on the work of Dr. Taguchi, with emphasis on the optimization of dynamic systems and tolerance design. Various quality loss functions are presented in this thesis which can be used for quality evaluation. The goal of robust design for dynamic systems is to reduce the deviations of quality characteristics for the real system from an ideal target which can change based on the requirements of the customer. The optimization can be simplified by decomposing the selection of control factors Z and the adjustment of scaling/leveling factors R into a two-step procedure. The first step is selecting levels for factors Z to maximize the signal-to-noise (SN) ratio that is supposed to be independent of the adjustment of factors R. The second step is used to adjust the real system to a desired model. A systematic approach to optimization is provided for dynamic systems. The motivation of the SN ratio is given and the validity of the SN ratio is examined for various systems. However, for the specific models where the use of the SN ratio is questionable, the necessary modification is suggested. In addition, discrete dynamic characteristics are discussed. The objective of tolerance design is to balance quality loss due to variations and cost increase due to control of variations. Based on the variation transmission equation developed in this thesis, the best tolerance levels are specified for components and subsystems. The tolerance design approach is presented for quality characteristics which may deteriorate over time. Also, a method is presented to develop the tolerances for lower-level quality characteristics based on the tolerances for higher-level quality characteristics, to reflect the voice of the customer. Illustrations are given to demonstrate the efficiency of the tolerance design methodology

SIMULATION ON DISTRIBUTED CONTROL SYSTEM TIME DELAY

One of the fastest growing market segments among control and automation technologies is the introduction and development of network control systems. Even though, fieldbus systems have been used for several years, functional and performance validation is an important issue addressed by fieldbus network design. Simulation support is needed for early functional validation and performance as an evaluation tool for the network design and operation performance analysis in early engineering stages. The delay time which is a random variable produced in the essence of the networked control system transmission process can greatly reduce the performance of control systems such as rising time and overshoot increasing, and situation where the control system become unstable. Therefore, this project briefly explains about the study on the fieldbus real time performance in term of time delay. Delay analysis on a designed cascaded control tank system with the case study of with and without controller have been performed to observe the severity of time delay on fieldbus system. The scope of study covered includes study on methodology to perform analysis on the fieldbus system of a plant using SIMULINK. From the findings, it can be observed that induced delay in complex fieldbus system is very prominent. Complex plant system has a high network delay and response time. Thus, these drawbacks can be overcome using simulation environment to forecast system behaviour and find the best suited network solutions beforehand to minimize engineering and hardware costs