A review and comparison of ontology-based approaches to robot autonomy

Within the next decades, robots will need to be able to execute a large variety of tasks autonomously in a large variety of environments. To relax the resulting programming effort, a knowledge-enabled approach to robot programming can be adopted to organize information in re-usable knowledge pieces. However, for the ease of reuse, there needs to be an agreement on the meaning of terms. A common approach is to represent these terms using ontology languages that conceptualize the respective domain. In this work, we will review projects that use ontologies to support robot autonomy. We will systematically search for projects that fulfill a set of inclusion criteria and compare them with each other with respect to the scope of their ontology, what types of cognitive capabilities are supported by the use of ontologies, and which is their application domain.Peer ReviewedPostprint (author's final draft

Toward knowledge-based automatic 3D spatial topological modeling from LiDAR point clouds for urban areas

Le traitement d'un très grand nombre de données LiDAR demeure très coûteux et nécessite des approches de modélisation 3D automatisée. De plus, les nuages de points incomplets causés par l'occlusion et la densité ainsi que les incertitudes liées au traitement des données LiDAR compliquent la création automatique de modèles 3D enrichis sémantiquement. Ce travail de recherche vise à développer de nouvelles solutions pour la création automatique de modèles géométriques 3D complets avec des étiquettes sémantiques à partir de nuages de points incomplets. Un cadre intégrant la connaissance des objets à la modélisation 3D est proposé pour améliorer la complétude des modèles géométriques 3D en utilisant un raisonnement qualitatif basé sur les informations sémantiques des objets et de leurs composants, leurs relations géométriques et spatiales. De plus, nous visons à tirer parti de la connaissance qualitative des objets en reconnaissance automatique des objets et à la création de modèles géométriques 3D complets à partir de nuages de points incomplets. Pour atteindre cet objectif, plusieurs solutions sont proposées pour la segmentation automatique, l'identification des relations topologiques entre les composants de l'objet, la reconnaissance des caractéristiques et la création de modèles géométriques 3D complets. (1) Des solutions d'apprentissage automatique ont été proposées pour la segmentation sémantique automatique et la segmentation de type CAO afin de segmenter des objets aux structures complexes. (2) Nous avons proposé un algorithme pour identifier efficacement les relations topologiques entre les composants d'objet extraits des nuages de points afin d'assembler un modèle de Représentation Frontière. (3) L'intégration des connaissances sur les objets et la reconnaissance des caractéristiques a été développée pour inférer automatiquement les étiquettes sémantiques des objets et de leurs composants. Afin de traiter les informations incertitudes, une solution de raisonnement automatique incertain, basée sur des règles représentant la connaissance, a été développée pour reconnaître les composants du bâtiment à partir d'informations incertaines extraites des nuages de points. (4) Une méthode heuristique pour la création de modèles géométriques 3D complets a été conçue en utilisant les connaissances relatives aux bâtiments, les informations géométriques et topologiques des composants du bâtiment et les informations sémantiques obtenues à partir de la reconnaissance des caractéristiques. Enfin, le cadre proposé pour améliorer la modélisation 3D automatique à partir de nuages de points de zones urbaines a été validé par une étude de cas visant à créer un modèle de bâtiment 3D complet. L'expérimentation démontre que l'intégration des connaissances dans les étapes de la modélisation 3D est efficace pour créer un modèle de construction complet à partir de nuages de points incomplets.The processing of a very large set of LiDAR data is very costly and necessitates automatic 3D modeling approaches. In addition, incomplete point clouds caused by occlusion and uneven density and the uncertainties in the processing of LiDAR data make it difficult to automatic creation of semantically enriched 3D models. This research work aims at developing new solutions for the automatic creation of complete 3D geometric models with semantic labels from incomplete point clouds. A framework integrating knowledge about objects in urban scenes into 3D modeling is proposed for improving the completeness of 3D geometric models using qualitative reasoning based on semantic information of objects and their components, their geometric and spatial relations. Moreover, we aim at taking advantage of the qualitative knowledge of objects in automatic feature recognition and further in the creation of complete 3D geometric models from incomplete point clouds. To achieve this goal, several algorithms are proposed for automatic segmentation, the identification of the topological relations between object components, feature recognition and the creation of complete 3D geometric models. (1) Machine learning solutions have been proposed for automatic semantic segmentation and CAD-like segmentation to segment objects with complex structures. (2) We proposed an algorithm to efficiently identify topological relationships between object components extracted from point clouds to assemble a Boundary Representation model. (3) The integration of object knowledge and feature recognition has been developed to automatically obtain semantic labels of objects and their components. In order to deal with uncertain information, a rule-based automatic uncertain reasoning solution was developed to recognize building components from uncertain information extracted from point clouds. (4) A heuristic method for creating complete 3D geometric models was designed using building knowledge, geometric and topological relations of building components, and semantic information obtained from feature recognition. Finally, the proposed framework for improving automatic 3D modeling from point clouds of urban areas has been validated by a case study aimed at creating a complete 3D building model. Experiments demonstrate that the integration of knowledge into the steps of 3D modeling is effective in creating a complete building model from incomplete point clouds

Hybrid approaches based on computational intelligence and semantic web for distributed situation and context awareness

2011 - 2012The research work focuses on Situation Awareness and Context Awareness topics. Specifically, Situation Awareness involves being aware of what is happening in the vicinity to understand how information, events, and one’s own actions will impact goals and objectives, both immediately and in the near future. Thus, Situation Awareness is especially important in application domains where the information flow can be quite high and poor decisions making may lead to serious consequences. On the other hand Context Awareness is considered a process to support user applications to adapt interfaces, tailor the set of application-relevant data, increase the precision of information retrieval, discover services, make the user interaction implicit, or build smart environments. Despite being slightly different, Situation and Context Awareness involve common problems such as: the lack of a support for the acquisition and aggregation of dynamic environmental information from the field (i.e. sensors, cameras, etc.); the lack of formal approaches to knowledge representation (i.e. contexts, concepts, relations, situations, etc.) and processing (reasoning, classification, retrieval, discovery, etc.); the lack of automated and distributed systems, with considerable computing power, to support the reasoning on a huge quantity of knowledge, extracted by sensor data. So, the thesis researches new approaches for distributed Context and Situation Awareness and proposes to apply them in order to achieve some related research objectives such as knowledge representation, semantic reasoning, pattern recognition and information retrieval. The research work starts from the study and analysis of state of art in terms of techniques, technologies, tools and systems to support Context/Situation Awareness. The main aim is to develop a new contribution in this field by integrating techniques deriving from the fields of Semantic Web, Soft Computing and Computational Intelligence. From an architectural point of view, several frameworks are going to be defined according to the multi-agent paradigm. Furthermore, some preliminary experimental results have been obtained in some application domains such as Airport Security, Traffic Management, Smart Grids and Healthcare. Finally, future challenges is going to the following directions: Semantic Modeling of Fuzzy Control, Temporal Issues, Automatically Ontology Elicitation, Extension to other Application Domains and More Experiments. [edited by author]XI n.s

Semantic technologies for supporting KDD processes

209 p.Achieving a comfortable thermal situation within buildings with an efficient use of energy remains still an open challenge for most buildings. In this regard, IoT (Internet of Things) and KDD (Knowledge Discovery in Databases) processes may be combined to solve these problems, even though data analysts may feel overwhelmed by heterogeneity and volume of the data to be considered. Data analysts could benefit from an application assistant that supports them throughout the KDD process. This research work aims at supporting data analysts through the different KDD phases towards the achievement of energy efficiency and thermal comfort in tertiary buildings. To do so, the EEPSA (Energy Efficiency Prediction Semantic Assistant) is proposed, which aids data analysts discovering the most relevant variables for the matter at hand, and informs them about relationships among relevant data. This assistant leverages Semantic Technologies such as ontologies, ontology-driven rules and ontology-driven data access. More specifically, the EEPSA ontology is the cornerstone of the assistant. This ontology is developed on top of three ODPs (Ontology Design Patterns) and it is designed so that its customization to address similar problems in different types of buildings can be approached methodically

A Design Science Research Approach to Architecting and Developing Information Systems for Collaborative Manufacturing : A Case for Human-Robot Collaboration

Konseptointi- ja suunnitteluvaiheessa sekä valmistuksen, käytön ja kehitysprosessin aikana syntyy tietoa, jonka hyödyntämisessä on valtavaa potentiaalia liike-elämän ja tuotantoprosessien muuttamiseen. Neljännen teollisen vallankumouksen ytimessä oleva digitaalinen muutos tunnistaa tämän painottaen erityisesti tämän tiedon yhdistämistä toimintojen ja järjestelmien tukemiseksi läpi tuotteen elinkaareen, mitä kutsutaan digitaaliseksi säikeen kehykseksi (digital thread framework). Tämän väitöskirjan tavoitteena on kehittää ja käyttää yhtä tällaista viitekehystä ihmisen ja robotin yhteistoiminnan asiayhteydessä. Tämä kehys pyrkii vastaamaan merkittävään ongelmaan, joka liittyy mukautuvuuden ja joustavuuden abstrakteihin ominaisuuksiin. Nykyiset ihmisen ja robotin yhteistyöjärjestelmät (human-robot collaboration (HRC)) on rakennettu pääasiassa pysyviksi järjestelmiksi, jotka sivuuttavat ihmisten intuitiivisen toiminnan asettamalla heidän roolinsa yhteistyötehtävissä etukäteen määritellyiksi. Lisäksi järjestelmien kyky vaihtaa tuotteesta toiseen on rajoittunutta. Tämä on erityisen ongelmallista nykyisellä laajan tuotevalikoiman aikakaudella, joka johtuu asiakkaiden räätälöidyistä vaatimuksista. Tähän taustaan vastaten, tämä väitöskirja käyttää design science research methodology -menetelmää suunnitellakseen, kehittääkseen ja ottaakseen käyttöön kolme pääasiallista artefaktia ihmisen ja robotin yhteistyösolussa laboratorioympäristössä. Ensimmäinen on digitaalisen säikeen kehys (digital thread framework), joka integroi tuotesuunnitteluympäristön toimijaksi monitoimijajärjestelmään käyttäen uusimpia tietoon perustuvia suunnittelujärjestelmiä, mikä tarjoaa prosessin toimijoille pääsyn tuotesuunnittelumalleihin reaaliajassa. Toinen on lisätyn todellisuuden malli, joka tarjoaa rajapinnan kokoonpanotehtävässä yhteistyöhön osallistuvan ihmisoperaattorin ja edellä mainitun kehyksen välille. Kolmas on tukitietomalli, jota yhteistyötä tekevät toimijat käyttävät tietopohjanaan täyttääkseen yhteistyössä tapahtuvan kokoonpanon tavoitteet mukautuvasti. Näitä kehitettyjä artefakteja käytettiin kokonaisuutena tapaustutkimuksissa, jotka liittyivät aidon dieselmoottorin kokoonpanoon, ja joissa todennettiin niiden hyödyllisyys ja että ne lisäävät joustavuutta, jota varten kehys (framework) suunniteltiin. Rajauslaatikoiden näyttäminen skaalautuvana informaationa, joka hahmottaa alikokoonpanon osien geometriaa, demostroi kehitettyjen artefaktien käytettävyyttä yhteistyötä tekevien toimijoiden aikomuksia heijastavien laajennetun todellisuuden projektioiden tuottamiseksi. Yhteenvetona tämän väitöskirjan tuloksena syntyi lähestymistapa älykkään ja mukautuvan robotiikan toteuttamiseksi hyödyntäen tietovirtoja ja mallinnusta ihmisen ja robotin yhteistoiminnan kontekstissa. Teollisuuden raportoima älykkäästi mukautuvien HRC-järjestelmien puute taas toimi osaltaan motivaationa tähän väitöskirjassa tehtyyn työhön. Kun tulevaisuuden tuotteet ja tuotantojärjestelmät muuttuvat monimutkaisemmiksi, tietojärjestelmiltä odotetaan suurempaa vastuuta korvaamaan ihmisen työmuistin luontaiset rajat ja mahdollistamaan siirtyminen kohti ihmiskeskeistä valmistusta, joihin viitataan termeillä Operator 4.0 ja Industry 5.0. Näin ollen on odotettavissa, että tietojärjestelmien tutkimus, kuten tämä väitöskirja, voi auttaa ottamaan merkittäviä askeleita tähän suuntaan.Information generated from the conceptualization, design, manufacturing, and use of a product has immense potential in transforming both the business and manufacturing processes of the manufacturing enterprise. The digital transformation at the heart of the fourth industrial revolution has acknowledged this with a special emphasis on weaving a thread of this information to support functions and systems throughout the life cycle of the product with what is known as a digital thread framework. This dissertation aims to develop and use one such framework in the context of human-robot collaborative assembly. The overarching problem that the framework aims to solve can be attributed to the abstract qualities of adaptability and flexibility. The human-robot collaboration (HRC) systems of today are built predominantly as static systems and ignore the intuitive role of humans by having their roles in collaborative tasks pre-defined. Furthermore, their ability to switch between products during product changeovers is also limited. This is especially problematic in the current era of product variety, stemming from the customised requirements of customers. To this end, this dissertation employs the design science research methodology to design, develop, and deploy predominantly three artefacts in a human-robot work cell in a laboratory setting. The first is the digital thread framework that integrates the product design environment using state-of-the-art knowledge-based engineering systems, as an agent of a multi-agent system, which provide the collaborative human-robot agents with access to product design models at run time. The second is a constituent mixed-reality model that provides an interface for the foregoing framework for the human operator engaged in collaborative assembly. The third is a supporting information model that the agents use as their knowledge base to fulfil adaptively the goals of collaborative assembly. Together, these developed artefacts were employed in case studies involving a real diesel engine assembly during which they were observed to provide utility and support the cause of adaptability for which the framework was designed. The identification of bounding boxes as a scalable information construct, that approximates the part geometry of the sub-assembly components, demonstrates the utility of the developed artefacts for spatially augmenting them as projections as intentions of collaborating agents. In summary, this dissertation contributes with an approach towards realising intelligent and adaptive robotics within the realms of information flows and modelling in the context of human-robot collaboration. The lack of intelligently adaptable HRC systems reported by the industry in part motivated the work undertaken in this dissertation. As future products and production systems become more complex, information systems are expected to assume greater responsibility to compensate for the inherent limits of the human working memory and enable transition towards a human-centred manufacturing, the current likes of which are labelled as Operator 4.0 and Industry 5.0. Thus, the expectation is that information systems research, such as this dissertation, can help take significant strides forward in this direction

Inferring Complex Activities for Context-aware Systems within Smart Environments

The rising ageing population worldwide and the prevalence of age-related conditions such as physical fragility, mental impairments and chronic diseases have significantly impacted the quality of life and caused a shortage of health and care services. Over-stretched healthcare providers are leading to a paradigm shift in public healthcare provisioning. Thus, Ambient Assisted Living (AAL) using Smart Homes (SH) technologies has been rigorously investigated to help address the aforementioned problems. Human Activity Recognition (HAR) is a critical component in AAL systems which enables applications such as just-in-time assistance, behaviour analysis, anomalies detection and emergency notifications. This thesis is aimed at investigating challenges faced in accurately recognising Activities of Daily Living (ADLs) performed by single or multiple inhabitants within smart environments. Specifically, this thesis explores five complementary research challenges in HAR. The first study contributes to knowledge by developing a semantic-enabled data segmentation approach with user-preferences. The second study takes the segmented set of sensor data to investigate and recognise human ADLs at multi-granular action level; coarse- and fine-grained action level. At the coarse-grained actions level, semantic relationships between the sensor, object and ADLs are deduced, whereas, at fine-grained action level, object usage at the satisfactory threshold with the evidence fused from multimodal sensor data is leveraged to verify the intended actions. Moreover, due to imprecise/vague interpretations of multimodal sensors and data fusion challenges, fuzzy set theory and fuzzy web ontology language (fuzzy-OWL) are leveraged. The third study focuses on incorporating uncertainties caused in HAR due to factors such as technological failure, object malfunction, and human errors. Hence, existing studies uncertainty theories and approaches are analysed and based on the findings, probabilistic ontology (PR-OWL) based HAR approach is proposed. The fourth study extends the first three studies to distinguish activities conducted by more than one inhabitant in a shared smart environment with the use of discriminative sensor-based techniques and time-series pattern analysis. The final study investigates in a suitable system architecture with a real-time smart environment tailored to AAL system and proposes microservices architecture with sensor-based off-the-shelf and bespoke sensing methods. The initial semantic-enabled data segmentation study was evaluated with 100% and 97.8% accuracy to segment sensor events under single and mixed activities scenarios. However, the average classification time taken to segment each sensor events have suffered from 3971ms and 62183ms for single and mixed activities scenarios, respectively. The second study to detect fine-grained-level user actions was evaluated with 30 and 153 fuzzy rules to detect two fine-grained movements with a pre-collected dataset from the real-time smart environment. The result of the second study indicate good average accuracy of 83.33% and 100% but with the high average duration of 24648ms and 105318ms, and posing further challenges for the scalability of fusion rule creations. The third study was evaluated by incorporating PR-OWL ontology with ADL ontologies and Semantic-Sensor-Network (SSN) ontology to define four types of uncertainties presented in the kitchen-based activity. The fourth study illustrated a case study to extended single-user AR to multi-user AR by combining RFID tags and fingerprint sensors discriminative sensors to identify and associate user actions with the aid of time-series analysis. The last study responds to the computations and performance requirements for the four studies by analysing and proposing microservices-based system architecture for AAL system. A future research investigation towards adopting fog/edge computing paradigms from cloud computing is discussed for higher availability, reduced network traffic/energy, cost, and creating a decentralised system. As a result of the five studies, this thesis develops a knowledge-driven framework to estimate and recognise multi-user activities at fine-grained level user actions. This framework integrates three complementary ontologies to conceptualise factual, fuzzy and uncertainties in the environment/ADLs, time-series analysis and discriminative sensing environment. Moreover, a distributed software architecture, multimodal sensor-based hardware prototypes, and other supportive utility tools such as simulator and synthetic ADL data generator for the experimentation were developed to support the evaluation of the proposed approaches. The distributed system is platform-independent and currently supported by an Android mobile application and web-browser based client interfaces for retrieving information such as live sensor events and HAR results

Human-Intelligence and Machine-Intelligence Decision Governance Formal Ontology

Since the beginning of the human race, decision making and rational thinking played a pivotal role for mankind to either exist and succeed or fail and become extinct. Self-awareness, cognitive thinking, creativity, and emotional magnitude allowed us to advance civilization and to take further steps toward achieving previously unreachable goals. From the invention of wheels to rockets and telegraph to satellite, all technological ventures went through many upgrades and updates. Recently, increasing computer CPU power and memory capacity contributed to smarter and faster computing appliances that, in turn, have accelerated the integration into and use of artificial intelligence (AI) in organizational processes and everyday life. Artificial intelligence can now be found in a wide range of organizational systems including healthcare and medical diagnosis, automated stock trading, robotic production, telecommunications, space explorations, and homeland security. Self-driving cars and drones are just the latest extensions of AI. This thrust of AI into organizations and daily life rests on the AI community’s unstated assumption of its ability to completely replicate human learning and intelligence in AI. Unfortunately, even today the AI community is not close to completely coding and emulating human intelligence into machines. Despite the revolution of digital and technology in the applications level, there has been little to no research in addressing the question of decision making governance in human-intelligent and machine-intelligent (HI-MI) systems. There also exists no foundational, core reference, or domain ontologies for HI-MI decision governance systems. Further, in absence of an expert reference base or body of knowledge (BoK) integrated with an ontological framework, decision makers must rely on best practices or standards that differ from organization to organization and government to government, contributing to systems failure in complex mission critical situations. It is still debatable whether and when human or machine decision capacity should govern or when a joint human-intelligence and machine-intelligence (HI-MI) decision capacity is required in any given decision situation. To address this deficiency, this research establishes a formal, top level foundational ontology of HI-MI decision governance in parallel with a grounded theory based body of knowledge which forms the theoretical foundation of a systemic HI-MI decision governance framework