SiAM-dp : an open development platform for massively multimodal dialogue systems in cyber-physical environments

Cyber-physical environments enhance natural environments of daily life such as homes, factories, offices, and cars by connecting the cybernetic world of computers and communication with the real physical world. While under the keyword of Industrie 4.0, cyber-physical environments will take a relevant role in the next industrial revolution, and they will also appear in homes, offices, workshops, and numerous other areas. In this new world, classical interaction concepts where users exclusively interact with a single stationary device, PC or smartphone become less dominant and make room for new occurrences of interaction between humans and the environment itself. Furthermore, new technologies and a rising spectrum of applicable modalities broaden the possibilities for interaction designers to include more natural and intuitive non-verbal and verbal communication. The dynamic characteristic of a cyber-physical environment and the mobility of users confronts developers with the challenge of developing systems that are flexible concerning the connected and used devices and modalities. This implies new opportunities for cross-modal interaction that go beyond dual modalities interaction as is well known nowadays. This thesis addresses the support of application developers with a platform for the declarative and model based development of multimodal dialogue applications, with a focus on distributed input and output devices in cyber-physical environments. The main contributions can be divided into three parts: - Design of models and strategies for the specification of dialogue applications in a declarative development approach. This includes models for the definition of project resources, dialogue behaviour, speech recognition grammars, and graphical user interfaces and mapping rules, which convert the device specific representation of input and output description to a common representation language. - The implementation of a runtime platform that provides a flexible and extendable architecture for the easy integration of new devices and components. The platform realises concepts and strategies of multimodal human-computer interaction and is the basis for full-fledged multimodal dialogue applications for arbitrary device setups, domains, and scenarios. - A software development toolkit that is integrated in the Eclipse rich client platform and provides wizards and editors for creating and editing new multimodal dialogue applications.Cyber-physische Umgebungen (CPEs) erweitern natürliche Alltagsumgebungen wie Heim, Fabrik, Büro und Auto durch Verbindung der kybernetischen Welt der Computer und Kommunikation mit der realen, physischen Welt. Die möglichen Anwendungsgebiete hierbei sind weitreichend. Während unter dem Stichwort Industrie 4.0 cyber-physische Umgebungen eine bedeutende Rolle für die nächste industrielle Revolution spielen werden, erhalten sie ebenfalls Einzug in Heim, Büro, Werkstatt und zahlreiche weitere Bereiche. In solch einer neuen Welt geraten klassische Interaktionskonzepte, in denen Benutzer ausschließlich mit einem einzigen Gerät, PC oder Smartphone interagieren, immer weiter in den Hintergrund und machen Platz für eine neue Ausprägung der Interaktion zwischen dem Menschen und der Umgebung selbst. Darüber hinaus sorgen neue Technologien und ein wachsendes Spektrum an einsetzbaren Modalitäten dafür, dass sich im Interaktionsdesign neue Möglichkeiten für eine natürlichere und intuitivere verbale und nonverbale Kommunikation auftun. Die dynamische Natur von cyber-physischen Umgebungen und die Mobilität der Benutzer darin stellt Anwendungsentwickler vor die Herausforderung, Systeme zu entwickeln, die flexibel bezüglich der verbundenen und verwendeten Geräte und Modalitäten sind. Dies impliziert auch neue Möglichkeiten in der modalitätsübergreifenden Kommunikation, die über duale Interaktionskonzepte, wie sie heutzutage bereits üblich sind, hinausgehen. Die vorliegende Arbeit befasst sich mit der Unterstützung von Anwendungsentwicklern mit Hilfe einer Plattform zur deklarativen und modellbasierten Entwicklung von multimodalen Dialogapplikationen mit einem Fokus auf verteilte Ein- und Ausgabegeräte in cyber-physischen Umgebungen. Die bearbeiteten Aufgaben können grundlegend in drei Teile gegliedert werden: - Die Konzeption von Modellen und Strategien für die Spezifikation von Dialoganwendungen in einem deklarativen Entwicklungsansatz. Dies beinhaltet Modelle für das Definieren von Projektressourcen, Dialogverhalten, Spracherkennergrammatiken, graphischen Benutzerschnittstellen und Abbildungsregeln, die die gerätespezifische Darstellung von Ein- und Ausgabegeräten in eine gemeinsame Repräsentationssprache transformieren. - Die Implementierung einer Laufzeitumgebung, die eine flexible und erweiterbare Architektur für die einfache Integration neuer Geräte und Komponenten bietet. Die Plattform realisiert Konzepte und Strategien der multimodalen Mensch-Maschine-Interaktion und ist die Basis vollwertiger multimodaler Dialoganwendungen für beliebige Domänen, Szenarien und Gerätekonfigurationen. - Eine Softwareentwicklungsumgebung, die in die Eclipse Rich Client Plattform integriert ist und Entwicklern Assistenten und Editoren an die Hand gibt, die das Erstellen und Editieren von neuen multimodalen Dialoganwendungen unterstützen

A Knowledge Graph Based Integration Approach for Industry 4.0

The fourth industrial revolution, Industry 4.0 (I40) aims at creating smart factories employing among others Cyber-Physical Systems (CPS), Internet of Things (IoT) and Artificial Intelligence (AI). Realizing smart factories according to the I40 vision requires intelligent human-to-machine and machine-to-machine communication. To achieve this communication, CPS along with their data need to be described and interoperability conflicts arising from various representations need to be resolved. For establishing interoperability, industry communities have created standards and standardization frameworks. Standards describe main properties of entities, systems, and processes, as well as interactions among them. Standardization frameworks classify, align, and integrate industrial standards according to their purposes and features. Despite being published by official international organizations, different standards may contain divergent definitions for similar entities. Further, when utilizing the same standard for the design of a CPS, different views can generate interoperability conflicts. Albeit expressive, standardization frameworks may represent divergent categorizations of the same standard to some extent, interoperability conflicts need to be resolved to support effective and efficient communication in smart factories. To achieve interoperability, data need to be semantically integrated and existing conflicts conciliated. This problem has been extensively studied in the literature. Obtained results can be applied to general integration problems. However, current approaches fail to consider specific interoperability conflicts that occur between entities in I40 scenarios. In this thesis, we tackle the problem of semantic data integration in I40 scenarios. A knowledge graphbased approach allowing for the integration of entities in I40 while considering their semantics is presented. To achieve this integration, there are challenges to be addressed on different conceptual levels. Firstly, defining mappings between standards and standardization frameworks; secondly, representing knowledge of entities in I40 scenarios described by standards; thirdly, integrating perspectives of CPS design while solving semantic heterogeneity issues; and finally, determining real industry applications for the presented approach. We first devise a knowledge-driven approach allowing for the integration of standards and standardization frameworks into an Industry 4.0 knowledge graph (I40KG). The standards ontology is used for representing the main properties of standards and standardization frameworks, as well as relationships among them. The I40KG permits to integrate standards and standardization frameworks while solving specific semantic heterogeneity conflicts in the domain. Further, we semantically describe standards in knowledge graphs. To this end, standards of core importance for I40 scenarios are considered, i.e., the Reference Architectural Model for I40 (RAMI4.0), AutomationML, and the Supply Chain Operation Reference Model (SCOR). In addition, different perspectives of entities describing CPS are integrated into the knowledge graphs. To evaluate the proposed methods, we rely on empirical evaluations as well as on the development of concrete use cases. The attained results provide evidence that a knowledge graph approach enables the effective data integration of entities in I40 scenarios while solving semantic interoperability conflicts, thus empowering the communication in smart factories

The lives of objects: designing for meaningful things

Today’s Internet of Things (IoT) is often employed to connect material artefacts to digital identifiers and a digital record of their history and existence. This has been heralded as a coming together of our material existences and our increasingly-digital lives. Bringing each object that we create, use and cherish into the IoT, is an outwardly appealing prospect. Using material objects is an accepted part of connecting with narratives and our history, and such a technological boon already enables the storytelling opportunities that are supported by rich digital records. However, in everyday life and in the practices that occupy them, people consider and share stories about the things that they feel to be meaningful to them in complex ways which do not necessarily conform to the expectations of the designers and developers who attempt to intervene and support such practices by focusing on the material objects at hand. This thesis draws upon observations from a thorough engagement with the community of practice of the Tabletop Miniature Wargaming pastime, which involves the acknowledged craft and use of objects deemed as meaningful, to reveal that the practitioners, in reality, construct their shared records and narratives around intangible Identities, both singular and collective, which they find to be the actual ‘meaningful things’ of their activities. These findings contravene the conventional emphasis on the material objects, and pose technological and conceptual challenges. Considering these findings through a lens informed by philosophical grounding, the thesis examines the distinctions between ordinary objects and extraordinary things; how things become meaningful; and the interplay between material and abstract things. The culmination of these efforts is the Meaningful Things Framework, which aims to help disambiguate the complex ways by which practitioners create, perceive and treat the meaningful things involved in their activities, and aid designers, developers and the communities themselves in understanding and supporting their practices

Fabrike budućnosti kroz prizmu najnovije industrijske revolucije

Fabrike budućnosti - Ako se povežu digitalne tehnologije s industrijskim proizvodima i logistikom (odnosno Industrijom 4.0), dobijaju se „Pametne fabrike“ (eng. Smart Factories) koje karakteriše prilagodljivost, efikasno korišćenje resursa i ergonomija, te integracija klijenata i poslovnih partnera u poslovni proces. Predmet rada je analiza i razvoj novog pristupa Industrijske revolucije 4.0 kroz prizmu savremenih Fabrika budućnosti, sa aspekta njihove strukture, osnove za integraciju i analize rezultata primene koji ukazuju na karakteristike, koristi i probleme primene u preduzećima. Cilj rada je prikazati strateški visoko-tehnološki projekat budućnosti Industrija 4.0, koji predstavlja novu viziju četvrte Industrijske revolucije (revolucije 21. veka) i promoviše kompjuterizaciju tradicionalnih industrija, a posebno segment kao što je proizvodnja. Cilj ovoga rada je prikazivanje novog koncepta Industrijske Revolucije 4.0, Fabrika budućnosti i unapredjenja njihovih elemenata E–Proizvodnje (ERP, PLM-a itd.), kroz praktičnu primenu. Takođe, biće predstavljene vizije fabrika budućnosti (sa praktičnim primerima) vodećih svetskih kompanija u svojim oblastima : Siemens (procesna industrija), Bosch (automobilska industrija), AIRBUS (avio industrija), AARBAKKE Norveška (naftna industrija), GENERAL ELECTRIC (energetski sektor: proizvodnja i distribucija električne energije - termo elektrane), kao i primeri mikro fabrika budućnosti i nano tehnologija. Primenom novih tehnologija dolazi do znatnih poboljšanja, tako što se kroz sve veću automatizaciju drastično skraćuje period između razvoja nekog novog proizvoda i njegovog izlaska na tržište. Tehnološka osnova su sajber-fizički sistemi i internet