A parallel self-organizing community detection algorithm based on swarm intelligence for large scale complex networks

Community detection is a critical task for complex network analysis. It helps us to understand the properties of the system that a complex network represents and has significance to a wide range of applications. Nowadays, the challenges faced by community detection algorithms include overlapping community structure detection, large scale network analysis, dynamic changing of analyzed network topology and many more. In this paper a self-organizing community detection algorithm, based on the idea of swarm intelligence, was proposed and its parallel algorithm was designed on Giraph++ which is a semi-asynchronous parallel graph computation framework running on distributed environment. In the algorithm, a network of large size is firstly divided into a number of small sub-networks. Then, each sub-network is modeled as a self-evolving swarm intelligence sub-system, while each vertex within the sub-network acts iteratively to join into or leave from communities based on a set of predefined vertex action rules. Meanwhile, the local communities of a sub-network are sent to other sub-networks to make their members have a chance to join into, therefore connecting these self-evolving swarm intelligence sub-systems together as a whole, large and evolving, system. The vertex actions during evolution of a sub-network are sent as well to keep multiple community replicas being consistent. Thus network communication efficiency has a great impact on the algorithm’s performance. While there is no vertex changing in its belonging communities anymore, an optimal community structure of the whole network will have emerged as a result. In the algorithm it is natural that a vertex can join into multiple communities simultaneously, thus can be used for overlapping community detection. The algorithm deals with vertex and edge adding or deleting in the same way as the algorithm running, therefore inherently supports dynamic network analysis. The algorithm can be used for the analysis of large scale networks with its parallel version running on distributed environment. A variety of experiments conducted on synthesized networks have shown that the proposed algorithm can effectively detect community structures and its performance is much better than certain popular community detection algorithms

A parallel self-organizing overlapping community detection algorithm based on swarm intelligence for large scale complex networks

Community detection is a critical task for complex network analysis. It helps us to understand the properties of the system that a complex network represents and has significance to a wide range of applications. Though a large number of algorithms have been developed, the detection of overlapping communities from large scale and (or) dynamic networks still remains challenging. In this paper, a Parallel Self-organizing Overlapping Community Detection (PSOCD) algorithm ground on the idea of swarm intelligence is proposed. The PSOCD is designed based on the concept of swarm intelligence system where an analyzed network is treated as a decentralized, self-organized, and self-evolving systems, in which each vertex acts iteratively to join to or leave from communities based on a set of predefined simple vertex action rules. The algorithm is implemented on a distributed graph processing platform named Giraph++; therefore it is capable of analyzing large scale networks. The algorithm is also able to handle overlapping community detection well because a vertex can naturally joins to multiple communities simultaneously. Moreover, if some vertexes and edges are added to or deleted from the analyzed network, the algorithm only needs to adjust community assignments of affected vertexes in the same way as its ending joining communities for a vertex, i.e., it inherently supports dynamic network analysis. The proposed PSOCD is evaluated using a number of variety large scale synthesized and real world networks. Experimental results indicate that the proposed algorithm can effectively discover overlapping communities on large-scale network and the quality of its detected overlapping community structures is superior to two state-of-the-art algorithms, namely Speaker Listener Label Propagation Algorithm (SLPA) and Order Statistics Local Optimization Method (OSLOM), especially on high overlapping density networks and (or) high overlapping diversity networks

A self-organizing algorithm for community structure analysis in complex networks

Community structure analysis is a critical task for complex network analysis. It helps us to understand the properties of the system that a complex network represents, and has significance to a wide range of real applications. The Label Propagation Algorithm (LPA) is currently the most popular community structure analysis algorithm due to its near linear time complexity. However, the performance of the LPA has proven to be unstable and the correctness of community assignment of nodes is unsatisfactory. In this paper a Self-Organizing Community Detection and Analytic Algorithm (SOCDA2) based on swarm intelligence is proposed. In the algorithm, a network is modeled as a swarm intelligence system, while each node within the network acts iteratively to join or leave communities based on a set of pre-defined node action rules, in order to improve the quality of the communities. When there is not a node changing its belonging community anymore, an optimal community structure will emerge as a result. A variety of experiments conducted on both synthesized and real-world networks have shown results which indicate that the proposed algorithm can effectively detect community structures and the performance is better than that of the LPA. In addition, the algorithm can be extended for overlapping community detection and be parallelized for largescale network analysis

Network Partitioning in Distributed Agent-Based Models

Agent-Based Models (ABMs) are an emerging simulation paradigm for modeling complex systems, comprised of autonomous, possibly heterogeneous, interacting agents. The utility of ABMs lies in their ability to represent such complex systems as self-organizing networks of agents. Modeling and understanding the behavior of complex systems usually occurs at large and representative scales, and often obtaining and visualizing of simulation results in real-time is critical. The real-time requirement necessitates the use of in-memory computing, as it is difficult and challenging to handle the latency and unpredictability of disk accesses. Combining this observation with the scale requirement emphasizes the need to use parallel and distributed computing platforms, such as MPI-enabled CPU clusters. Consequently, the agent population must be partitioned across different CPUs in a cluster. Further, the typically high volume of interactions among agents can quickly become a significant bottleneck for real-time or large-scale simulations. The problem is exacerbated if the underlying ABM network is dynamic and the inter-process communication evolves over the course of the simulation. Therefore, it is critical to develop topology-aware partitioning mechanisms to support such large simulations. In this dissertation, we demonstrate that distributed agent-based model simulations benefit from the use of graph partitioning algorithms that involve a local, neighborhood-based perspective. Such methods do not rely on global accesses to the network and thus are more scalable. In addition, we propose two partitioning schemes that consider the bottom-up individual-centric nature of agent-based modeling. The First technique utilizes label-propagation community detection to partition the dynamic agent network of an ABM. We propose a latency-hiding, seamless integration of community detection in the dynamics of a distributed ABM. To achieve this integration, we exploit the similarity in the process flow patterns of a label-propagation community-detection algorithm and self-organizing ABMs. In the second partitioning scheme, we apply a combination of the Guided Local Search (GLS) and Fast Local Search (FLS) metaheuristics in the context of graph partitioning. The main driving principle of GLS is the dynamic modi?cation of the objective function to escape local optima. The algorithm augments the objective of a local search, thereby transforming the landscape structure and escaping a local optimum. FLS is a local search heuristic algorithm that is aimed at reducing the search space of the main search algorithm. It breaks down the space into sub-neighborhoods such that inactive sub-neighborhoods are removed from the search process. The combination of GLS and FLS allowed us to design a graph partitioning algorithm that is both scalable and sensitive to the inherent modularity of real-world networks

Complex Systems Science: Dreams of Universality, Reality of Interdisciplinarity

Using a large database (~ 215 000 records) of relevant articles, we empirically study the "complex systems" field and its claims to find universal principles applying to systems in general. The study of references shared by the papers allows us to obtain a global point of view on the structure of this highly interdisciplinary field. We show that its overall coherence does not arise from a universal theory but instead from computational techniques and fruitful adaptations of the idea of self-organization to specific systems. We also find that communication between different disciplines goes through specific "trading zones", ie sub-communities that create an interface around specific tools (a DNA microchip) or concepts (a network).Comment: Journal of the American Society for Information Science and Technology (2012) 10.1002/asi.2264

A General Framework for Complex Network Applications

Complex network theory has been applied to solving practical problems from different domains. In this paper, we present a general framework for complex network applications. The keys of a successful application are a thorough understanding of the real system and a correct mapping of complex network theory to practical problems in the system. Despite of certain limitations discussed in this paper, complex network theory provides a foundation on which to develop powerful tools in analyzing and optimizing large interconnected systems.Comment: 8 page