Ubiquitous Integration and Temporal Synchronisation (UbilTS) framework : a solution for building complex multimodal data capture and interactive systems

Contemporary Data Capture and Interactive Systems (DCIS) systems are tied in with various technical complexities such as multimodal data types, diverse hardware and software components, time synchronisation issues and distributed deployment configurations. Building these systems is inherently difficult and requires addressing of these complexities before the intended and purposeful functionalities can be attained. The technical issues are often common and similar among diverse applications. This thesis presents the Ubiquitous Integration and Temporal Synchronisation (UbiITS) framework, a generic solution to address the technical complexities in building DCISs. The proposed solution is an abstract software framework that can be extended and customised to any application requirements. UbiITS includes all fundamental software components, techniques, system level layer abstractions and reference architecture as a collection to enable the systematic construction of complex DCISs. This work details four case studies to showcase the versatility and extensibility of UbiITS framework’s functionalities and demonstrate how it was employed to successfully solve a range of technical requirements. In each case UbiITS operated as the core element of each application. Additionally, these case studies are novel systems by themselves in each of their domains. Longstanding technical issues such as flexibly integrating and interoperating multimodal tools, precise time synchronisation, etc., were resolved in each application by employing UbiITS. The framework enabled establishing a functional system infrastructure in these cases, essentially opening up new lines of research in each discipline where these research approaches would not have been possible without the infrastructure provided by the framework. The thesis further presents a sample implementation of the framework on a device firmware exhibiting its capability to be directly implemented on a hardware platform. Summary metrics are also produced to establish the complexity, reusability, extendibility, implementation and maintainability characteristics of the framework.Engineering and Physical Sciences Research Council (EPSRC) grants - EP/F02553X/1, 114433 and 11394

Chapter From the Lab to the Real World: Affect Recognition Using Multiple Cues and Modalities

Interdisciplinary concept of dissipative soliton is unfolded in connection with ultrafast fibre lasers. The different mode-locking techniques as well as experimental realizations of dissipative soliton fibre lasers are surveyed briefly with an emphasis on their energy scalability. Basic topics of the dissipative soliton theory are elucidated in connection with concepts of energy scalability and stability. It is shown that the parametric space of dissipative soliton has reduced dimension and comparatively simple structure that simplifies the analysis and optimization of ultrafast fibre lasers. The main destabilization scenarios are described and the limits of energy scalability are connected with impact of optical turbulence and stimulated Raman scattering. The fast and slow dynamics of vector dissipative solitons are exposed

REAL-TIME LOGISTICS - Case Development of a Shipment Status Display System for a Large Manufacturing Company

As traditional heavy industry businesses transform into global solutions providers, their business models change into project-based and their supply networks expand. Project business faces greater uncertainty within the supply chain than traditional business, thus requires greater need for data exchange within the supply chain. Numerous information systems have provided the organization with a wealth of data. However project management often faces great challenges to utilize it for better visibility on project delivery status, as well as to communicate that to stakeholders. In response to the need for better usage and presentation of transactional project logistical data, a real-time shipment status display system has been developed. The complete system offers an intuitive, up-to-date, fast, and reliable display that is accessible through a wide range of devices. In this thesis, the system is customized to run on public displays. In term of development methodology, spiral axiomatic design approach is adopted to ensure maximum independence of components. The end result is a system comprising of two independent sub-systems: one is for data collection and the second one is for presentation. Modern web technologies such as ASP.NET MVC4, HTML5, and CSS3 have been used to develop the presentation sub-system. The thesis contributes a software artifact that complements information systems that are either too much focused on transactional data or unable to communicate project logistics data to stakeholders. It also demonstrates the use of axiomatic system design in developing modern web platforms.fi=Opinnäytetyö kokotekstinä PDF-muodossa.|en=Thesis fulltext in PDF format.|sv=Lärdomsprov tillgängligt som fulltext i PDF-format

The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

CHORUS Deliverable 2.2: Second report - identification of multi-disciplinary key issues for gap analysis toward EU multimedia search engines roadmap

After addressing the state-of-the-art during the first year of Chorus and establishing the existing landscape in multimedia search engines, we have identified and analyzed gaps within European research effort during our second year. In this period we focused on three directions, notably technological issues, user-centred issues and use-cases and socio- economic and legal aspects. These were assessed by two central studies: firstly, a concerted vision of functional breakdown of generic multimedia search engine, and secondly, a representative use-cases descriptions with the related discussion on requirement for technological challenges. Both studies have been carried out in cooperation and consultation with the community at large through EC concertation meetings (multimedia search engines cluster), several meetings with our Think-Tank, presentations in international conferences, and surveys addressed to EU projects coordinators as well as National initiatives coordinators. Based on the obtained feedback we identified two types of gaps, namely core technological gaps that involve research challenges, and “enablers”, which are not necessarily technical research challenges, but have impact on innovation progress. New socio-economic trends are presented as well as emerging legal challenges

Assistive systems for quality assurance by context-aware user interfaces in health care and production

Rüther S. Assistive systems for quality assurance by context-aware user interfaces in health care and production. Bielefeld: Universitätsbibliothek Bielefeld; 2014.The reprocessing of medical devices is an essential procedure to keep hospitals operational. Workers at the Central Sterilization Supply Department (CSSD) clean, disinfect and sterilize medical devices and have to obligate to the manifold of legal and hygiene prescriptions. Failures during reprocessing can endanger patients' safety and increase costs. The process of decontamination has rich sources of failures because of the complexity of hygiene, medical devices and regulatory specifications. The benefits of an assistance system helping workers in preventing failures are therefore obvious and crucial. New interaction technologies such as augmented reality can potentially help workers in the CSSD to avoid failures during the reprocessing of medical devices. Challenging requirements for the application of new interaction technology within the CSSD arise through process complexity, legislation, integration and hygiene restrictions. This thesis proposes an assistance system that supports the worker in the unclean area of a CSSD with respect to these requirements. The system provides a user interface for context-aware worker guidance and collection of process relevant data from the worker. The proposed interaction mechanism of 'virtual touches' fulfills the hygiene requirements and is realized by an adapted workspace which is equipped with a depth camera and a projected user interface. The 'business process modeling notation 2.0 (BPMN 2.0)' standard is utilized to define process models that control the workflow, coordinate the system's components and maintain a database for quality assurance and worker guidance. In addition to an in depth description of the system, an evaluation with two user studies and interviews with CSSD domain experts are conducted throughout this thesis. The results reveal a high capability for failure avoidance during the reprocessing of medical devices without delaying the process compared to today's CSSDs. Additionally, CSSD experts appraise a high practical relevance and underline the feasibility of the underlying concepts for the CSSD domain. The concepts of the process integration, the standardized modeling of the workflow and workers' tasks as well as the context-aware interface are also helpful, relevant and applicable in the domain of manual assembly processes. Thus, this thesis describes, how the system can be transfered to the domain of manual production. The presentation of a prototype at a renowned international industrial fair and the accompanying feedback from manufacturing experts underline the scalability and the portability of the proposed assistance system to the production domain, which is a result of a component based system architecture utilizing process models for the coordination of computational devices and human workers

CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Multimodal Video Analysis and Modeling

From recalling long forgotten experiences based on a familiar scent or on a piece of music, to lip reading aided conversation in noisy environments or travel sickness caused by mismatch of the signals from vision and the vestibular system, the human perception manifests countless examples of subtle and effortless joint adoption of the multiple senses provided to us by evolution. Emulating such multisensory (or multimodal, i.e., comprising multiple types of input modes or modalities) processing computationally offers tools for more effective, efficient, or robust accomplishment of many multimedia tasks using evidence from the multiple input modalities. Information from the modalities can also be analyzed for patterns and connections across them, opening up interesting applications not feasible with a single modality, such as prediction of some aspects of one modality based on another. In this dissertation, multimodal analysis techniques are applied to selected video tasks with accompanying modalities. More specifically, all the tasks involve some type of analysis of videos recorded by non-professional videographers using mobile devices.Fusion of information from multiple modalities is applied to recording environment classification from video and audio as well as to sport type classification from a set of multi-device videos, corresponding audio, and recording device motion sensor data. The environment classification combines support vector machine (SVM) classifiers trained on various global visual low-level features with audio event histogram based environment classification using k nearest neighbors (k-NN). Rule-based fusion schemes with genetic algorithm (GA)-optimized modality weights are compared to training a SVM classifier to perform the multimodal fusion. A comprehensive selection of fusion strategies is compared for the task of classifying the sport type of a set of recordings from a common event. These include fusion prior to, simultaneously with, and after classification; various approaches for using modality quality estimates; and fusing soft confidence scores as well as crisp single-class predictions. Additionally, different strategies are examined for aggregating the decisions of single videos to a collective prediction from the set of videos recorded concurrently with multiple devices. In both tasks multimodal analysis shows clear advantage over separate classification of the modalities.Another part of the work investigates cross-modal pattern analysis and audio-based video editing. This study examines the feasibility of automatically timing shot cuts of multi-camera concert recordings according to music-related cutting patterns learnt from professional concert videos. Cut timing is a crucial part of automated creation of multicamera mashups, where shots from multiple recording devices from a common event are alternated with the aim at mimicing a professionally produced video. In the framework, separate statistical models are formed for typical patterns of beat-quantized cuts in short segments, differences in beats between consecutive cuts, and relative deviation of cuts from exact beat times. Based on music meter and audio change point analysis of a new recording, the models can be used for synthesizing cut times. In a user study the proposed framework clearly outperforms a baseline automatic method with comparably advanced audio analysis and wins 48.2 % of comparisons against hand-edited videos