Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Low carbon manufacturing: Fundamentals, methodology and application case studies

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.The requirement and awareness of the carbon emissions reduction in several scales and application of sustainable manufacturing have been now critically reviewed as important manufacturing trends in the 21st century. The key requirements for carbon emissions reduction in this context are energy efficiency, resource utilization, waste minimization and even the reduction of total carbon footprint. The recent approaches tend to only analyse and evaluate carbon emission contents of interested engineering systems. However, a systematic approach based on strategic decision making has not been officially defined with no standards or guidelines further formulated yet. The above requirements demand a fundamentally new approach to future applications of sustainable low carbon manufacturing. Energy and resource efficiencies and effectiveness based low carbon manufacturing (EREEbased LCM) is thus proposed in this research. The proposed EREE-based LCM is able to provide the systematic approach for integrating three key elements (energy efficiency, resource utilization and waste minimization) and taking account of them comprehensively in a scientific manner. The proposed approach demonstrates the solution for reducing carbon emissions in manufacturing systems at both the machine and shop floor levels. An integrated framework has been developed to demonstrate the feasible approach to achieve effective EREE-based LCM at different manufacturing levels including machine, shop floor, enterprise and supply chains. The framework is established in the matrix form with appropriate tools and methodologies related to the three keys elements at each manufacturing level. The theoretical model for EREE-based LCM is also presented, which consists of three essential elements including carbon dioxide emissions evaluation, an optimization method and waste reduction methodology. The preliminary experiment and simulations are carried out to evaluate the proposed concept. The modelling of EREE-based LCM has been developed for both the machine and shop floor levels. At the machine level, the modelling consists of the simulation of energy consumption due to the effect of machining set-up, the optimization model and waste minimization related to the optimized machining set-up. The simulation is established using sugeno type fuzzy logic. The learning method uses on experimental data (cutting trials) while the optimization model is created using mamdani type fuzzy logic with grey relational grade technique. At the shop floor level, the modelling is designed dependent on the cooperation with machine level modelling. The determination of the work assignment including machining set-up depends on fuzzy integer linear programming for several objectives with the evaluation of energy consumption data from machine level modelling. The simulation method is applied as the part of shop floor level modelling in order to maximize resource utilization and minimize undesired waste. The output from the shop floor level modelling is machine production a planning with preventive plan that can minimize the total carbon footprint. The axiomatic design theory has been applied to generate the comprehensive conceptual model E-R-W-C (energy, resource, waste and carbon footprint) of EREE-based LCM as a generic perspective of the systematic modelling. The implementation of EREE-based LCM on both the machine and shop floor levels are demonstrated using MATLAB toolbox and ProModel based simulation. The proposed concept, framework and modelling have been further evaluated and validated through case studies and experimental results.This work is financially supported by The Royal Thai Government

Inferring Complex Activities for Context-aware Systems within Smart Environments

The rising ageing population worldwide and the prevalence of age-related conditions such as physical fragility, mental impairments and chronic diseases have significantly impacted the quality of life and caused a shortage of health and care services. Over-stretched healthcare providers are leading to a paradigm shift in public healthcare provisioning. Thus, Ambient Assisted Living (AAL) using Smart Homes (SH) technologies has been rigorously investigated to help address the aforementioned problems. Human Activity Recognition (HAR) is a critical component in AAL systems which enables applications such as just-in-time assistance, behaviour analysis, anomalies detection and emergency notifications. This thesis is aimed at investigating challenges faced in accurately recognising Activities of Daily Living (ADLs) performed by single or multiple inhabitants within smart environments. Specifically, this thesis explores five complementary research challenges in HAR. The first study contributes to knowledge by developing a semantic-enabled data segmentation approach with user-preferences. The second study takes the segmented set of sensor data to investigate and recognise human ADLs at multi-granular action level; coarse- and fine-grained action level. At the coarse-grained actions level, semantic relationships between the sensor, object and ADLs are deduced, whereas, at fine-grained action level, object usage at the satisfactory threshold with the evidence fused from multimodal sensor data is leveraged to verify the intended actions. Moreover, due to imprecise/vague interpretations of multimodal sensors and data fusion challenges, fuzzy set theory and fuzzy web ontology language (fuzzy-OWL) are leveraged. The third study focuses on incorporating uncertainties caused in HAR due to factors such as technological failure, object malfunction, and human errors. Hence, existing studies uncertainty theories and approaches are analysed and based on the findings, probabilistic ontology (PR-OWL) based HAR approach is proposed. The fourth study extends the first three studies to distinguish activities conducted by more than one inhabitant in a shared smart environment with the use of discriminative sensor-based techniques and time-series pattern analysis. The final study investigates in a suitable system architecture with a real-time smart environment tailored to AAL system and proposes microservices architecture with sensor-based off-the-shelf and bespoke sensing methods. The initial semantic-enabled data segmentation study was evaluated with 100% and 97.8% accuracy to segment sensor events under single and mixed activities scenarios. However, the average classification time taken to segment each sensor events have suffered from 3971ms and 62183ms for single and mixed activities scenarios, respectively. The second study to detect fine-grained-level user actions was evaluated with 30 and 153 fuzzy rules to detect two fine-grained movements with a pre-collected dataset from the real-time smart environment. The result of the second study indicate good average accuracy of 83.33% and 100% but with the high average duration of 24648ms and 105318ms, and posing further challenges for the scalability of fusion rule creations. The third study was evaluated by incorporating PR-OWL ontology with ADL ontologies and Semantic-Sensor-Network (SSN) ontology to define four types of uncertainties presented in the kitchen-based activity. The fourth study illustrated a case study to extended single-user AR to multi-user AR by combining RFID tags and fingerprint sensors discriminative sensors to identify and associate user actions with the aid of time-series analysis. The last study responds to the computations and performance requirements for the four studies by analysing and proposing microservices-based system architecture for AAL system. A future research investigation towards adopting fog/edge computing paradigms from cloud computing is discussed for higher availability, reduced network traffic/energy, cost, and creating a decentralised system. As a result of the five studies, this thesis develops a knowledge-driven framework to estimate and recognise multi-user activities at fine-grained level user actions. This framework integrates three complementary ontologies to conceptualise factual, fuzzy and uncertainties in the environment/ADLs, time-series analysis and discriminative sensing environment. Moreover, a distributed software architecture, multimodal sensor-based hardware prototypes, and other supportive utility tools such as simulator and synthetic ADL data generator for the experimentation were developed to support the evaluation of the proposed approaches. The distributed system is platform-independent and currently supported by an Android mobile application and web-browser based client interfaces for retrieving information such as live sensor events and HAR results

Task Allocation Strategies in Multi-Robot Environment

Multirobot systems (MRS) hold the promise of improved performance and increased fault tolerance for large-scale problems. A robot team can accomplish a given task more quickly than a single agent by executing them concurrently. A team can also make effective use of specialists designed for a single purpose rather than requiring that a single robot be a generalist. Multirobot coordination, however, is a complex problem. An empirical study is described in the thesis that sought general guidelines for task allocation strategies. Different strategies are identified, and demonstrated in the multi-robot environment.Robot selection is one of the critical issues in the design of robotic workcells. Robot selection for an application is generally done based on experience, intuition and at most using the kinematic considerations like workspace, manipulability, etc. This problem has become more difficult in recent years due to increasing complexity, available features, and facilities offered by different robotic products. A systematic procedure is developed for selection of robot manipulators based on their different pertinent attributes. The robot selection procedure allows rapid convergence from a very large number of candidate robots to a manageable shortlist of potentially suitable robots. Subsequently, the selection procedure proceeds to rank the alternatives in the shortlist by employing different attributes based specification methods. This is an attempt to create exhaustive procedure by identifying maximum possible number of attributes for robot manipulators.Availability of large number of robot configurations has made the robot workcell designers think over the issue of selecting the most suitable one for a given set of operations. The process of selection of the appropriate kind of robot must consider the various attributes of the robot manipulator in conjunction with the requirement of the various operations for accomplishing the task. The present work is an attempt to develop a systematic procedure for selection of robot based on an integrated model encompassing the manipulator attributes and manipulator requirements

Towards Zero Touch Next Generation Network Management

The current trend in user services places an ever-growing demand for higher data rates, near-real-time latencies, and near-perfect quality of service. To meet such demands, fundamental changes were made to the front and mid-haul and backbone networking segments servicing them. One of the main changes made was virtualizing the networking components to allow for faster deployment and reconfiguration when needed. However, adopting such technologies poses several challenges, such as improving the performance and efficiency of these systems by properly orchestrating the services to the ideal edge device. A second challenge is ensuring the backbone optical networking maximizes and maintains the throughput levels under more dynamically variant conditions. A third challenge is addressing the limitation of placement techniques in O-RAN. In this thesis, we propose using various optimization modeling and machine learning techniques in three segments of network systems towards lowering the need for human intervention targeting zero-touch networking. In particular, the first part of the thesis applies optimization modeling, heuristics, and segmentation to improve the locally driven orchestration techniques, which are used to place demands on edge devices throughput to ensure efficient and resilient placement decisions. The second part of the thesis proposes using reinforcement learning (RL) techniques on a nodal base to address the dynamic nature of demands within an optical networking paradigm. The RL techniques ensure blocking rates are kept to a minimum by tailoring the agents’ behavior based on each node\u27s demand intake throughout the day. The third part of the thesis proposes using transfer learning augmented reinforcement learning to drive a network slicing-based solution in O-RAN to address the stringent and divergent demands of 5G applications. The main contributions of the thesis consist of three broad parts. The first is developing optimal and heuristic orchestration algorithms that improve demands’ performance and reliability in an edge computing environment. The second is using reinforcement learning to determine the appropriate spectral placement for demands within isolated optical paths, ensuring lower fragmentation and better throughput utilization. The third is developing a heuristic controlled transfer learning augmented reinforcement learning network slicing in an O-RAN environment. Hence, ensuring improved reliability while maintaining lower complexity than traditional placement techniques