Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Robotic Wireless Sensor Networks

In this chapter, we present a literature survey of an emerging, cutting-edge, and multi-disciplinary field of research at the intersection of Robotics and Wireless Sensor Networks (WSN) which we refer to as Robotic Wireless Sensor Networks (RWSN). We define a RWSN as an autonomous networked multi-robot system that aims to achieve certain sensing goals while meeting and maintaining certain communication performance requirements, through cooperative control, learning and adaptation. While both of the component areas, i.e., Robotics and WSN, are very well-known and well-explored, there exist a whole set of new opportunities and research directions at the intersection of these two fields which are relatively or even completely unexplored. One such example would be the use of a set of robotic routers to set up a temporary communication path between a sender and a receiver that uses the controlled mobility to the advantage of packet routing. We find that there exist only a limited number of articles to be directly categorized as RWSN related works whereas there exist a range of articles in the robotics and the WSN literature that are also relevant to this new field of research. To connect the dots, we first identify the core problems and research trends related to RWSN such as connectivity, localization, routing, and robust flow of information. Next, we classify the existing research on RWSN as well as the relevant state-of-the-arts from robotics and WSN community according to the problems and trends identified in the first step. Lastly, we analyze what is missing in the existing literature, and identify topics that require more research attention in the future

Local Communication Protocols for Learning Complex Swarm Behaviors with Deep Reinforcement Learning

Swarm systems constitute a challenging problem for reinforcement learning (RL) as the algorithm needs to learn decentralized control policies that can cope with limited local sensing and communication abilities of the agents. While it is often difficult to directly define the behavior of the agents, simple communication protocols can be defined more easily using prior knowledge about the given task. In this paper, we propose a number of simple communication protocols that can be exploited by deep reinforcement learning to find decentralized control policies in a multi-robot swarm environment. The protocols are based on histograms that encode the local neighborhood relations of the agents and can also transmit task-specific information, such as the shortest distance and direction to a desired target. In our framework, we use an adaptation of Trust Region Policy Optimization to learn complex collaborative tasks, such as formation building and building a communication link. We evaluate our findings in a simulated 2D-physics environment, and compare the implications of different communication protocols.Comment: 13 pages, 4 figures, version 2, accepted at ANTS 201

Contextual and Human Factors in Information Fusion

Proceedings of: NATO Advanced Research Workshop on Human Systems Integration to Enhance Maritime Domain Awareness for Port/Harbour Security Systems, Opatija (Croatia), December 8-12, 2008Context and human factors may be essential to improving measurement processes for each sensor, and the particular context of each sensor could be used to obtain a global definition of context in multisensor environments. Reality may be captured by human sensorial domain based only on machine stimulus and then generate a feedback which can be used by the machine at its different processing levels, adapting its algorithms and methods accordingly. Reciprocally, human perception of the environment could also be modelled by context in the machine. In the proposed model, both machine and man take sensorial information from the environment and process it cooperatively until a decision or semantic synthesis is produced. In this work, we present a model for context representation and reasoning to be exploited by fusion systems. In the first place, the structure and representation of contextual information must be determined before being exploited by a specific application. Under complex circumstances, the use of context information and human interaction can help to improve a tracking system's performance (for instance, video-based tracking systems may fail when dealing with object interaction, occlusions, crosses, etc.).Publicad

Adoption of vehicular ad hoc networking protocols by networked robots

This paper focuses on the utilization of wireless networking in the robotics domain. Many researchers have already equipped their robots with wireless communication capabilities, stimulated by the observation that multi-robot systems tend to have several advantages over their single-robot counterparts. Typically, this integration of wireless communication is tackled in a quite pragmatic manner, only a few authors presented novel Robotic Ad Hoc Network (RANET) protocols that were designed specifically with robotic use cases in mind. This is in sharp contrast with the domain of vehicular ad hoc networks (VANET). This observation is the starting point of this paper. If the results of previous efforts focusing on VANET protocols could be reused in the RANET domain, this could lead to rapid progress in the field of networked robots. To investigate this possibility, this paper provides a thorough overview of the related work in the domain of robotic and vehicular ad hoc networks. Based on this information, an exhaustive list of requirements is defined for both types. It is concluded that the most significant difference lies in the fact that VANET protocols are oriented towards low throughput messaging, while RANET protocols have to support high throughput media streaming as well. Although not always with equal importance, all other defined requirements are valid for both protocols. This leads to the conclusion that cross-fertilization between them is an appealing approach for future RANET research. To support such developments, this paper concludes with the definition of an appropriate working plan

Sensor coverage and actors relocation in wireless sensor and actor networks (WSAN) : optimization models and approximation algorithms

"December 2010.""A Thesis presented to the Faculty of the Graduate School at the University of Missouri In Partial Fulfillment of the Requirements for the Degree Master of Science."Thesis supervisor: Dr. Esra Sisikoglu.Wireless Sensors and Actor Networks (WSAN) have a wide variety of applications such as military surveillance, object tracking and habitat monitoring. Sensors are data gathering devices. Selecting the minimum number of sensors for network coverage is crucial to reduce the cost of installation and data processing time. Actors in a WSAN are decision-making units. They need to be communicating with their fellow actors in order to respond to events. Therefore, the need to maintain a connected inter-actor network at all times is critical. In the Actor Relocation Problem (Chapter 2) of this thesis we considered the problem of finding optimal strategies to restore connectivity when inter-actor network fails. We used a mixed integer programming formulation to find the optimal relocation strategies for actors in which the total travel distance is minimized. In our formulation we used powers of the adjacency matrix to generate constraints that ensure connectivity. In the Sensor Coverage Problem (Chapter 3) we developed a mixed integer programming model to find the minimum number of sensors and their locations to cover a given area. We also developed a bi-level algorithm that runs two separate optimization algorithms iteratively to find the location of sensors such that every point in a continuous area is covered.Includes bibliographical references (pages 61-63)

A Multi-Agent System Architecture for Sensor Networks

The design of the control systems for sensor networks presents important challenges. Besides the traditional problems about how to process the sensor data to obtain the target information, engineers need to consider additional aspects such as the heterogeneity and high number of sensors, and the flexibility of these networks regarding topologies and the sensors in them. Although there are partial approaches for resolving these issues, their integration relies on ad hoc solutions requiring important development efforts. In order to provide an effective approach for this integration, this paper proposes an architecture based on the multi-agent system paradigm with a clear separation of concerns. The architecture considers sensors as devices used by an upper layer of manager agents. These agents are able to communicate and negotiate services to achieve the required functionality. Activities are organized according to roles related with the different aspects to integrate, mainly sensor management, data processing, communication and adaptation to changes in the available devices and their capabilities. This organization largely isolates and decouples the data management from the changing network, while encouraging reuse of solutions. The use of the architecture is facilitated by a specific modelling language developed through metamodelling. A case study concerning a generic distributed system for fire fighting illustrates the approach and the comparison with related work

Girt by sea: understanding Australia’s maritime domains in a networked world

This study aims to provide the background, language and context necessary for an informed understanding of the challenges and dilemmas faced by those responsible for the efficacy of Australia’s maritime domain awareness system. Abstract Against a rapidly changing region dominated by the rise of China, India and, closer to home, Indonesia, Australia’s approaches to understanding its maritime domains will be influenced by strategic factors and diplomatic judgements as well as operational imperatives.  Australia’s alliance relationship with the United States and its relationships with regional neighbours may be expected to have a profound impact on the strength of the information sharing and interoperability regimes on which so much of Australia’s maritime domain awareness depends. The purpose of this paper is twofold.  First, it seeks to explain in plain English some of the principles, concepts and terms that maritime domain awareness practitioners grapple with on a daily basis.  Second, it points to a series of challenges that governments face in deciding how to spend scarce tax dollars to deliver a maritime domain awareness system that is necessary and sufficient for the protection and promotion of Australia’s national interests