Finding Top-k Dominance on Incomplete Big Data Using Map-Reduce Framework

Incomplete data is one major kind of multi-dimensional dataset that has random-distributed missing nodes in its dimensions. It is very difficult to retrieve information from this type of dataset when it becomes huge. Finding top-k dominant values in this type of dataset is a challenging procedure. Some algorithms are present to enhance this process but are mostly efficient only when dealing with a small-size incomplete data. One of the algorithms that make the application of TKD query possible is the Bitmap Index Guided (BIG) algorithm. This algorithm strongly improves the performance for incomplete data, but it is not originally capable of finding top-k dominant values in incomplete big data, nor is it designed to do so. Several other algorithms have been proposed to find the TKD query, such as Skyband Based and Upper Bound Based algorithms, but their performance is also questionable. Algorithms developed previously were among the first attempts to apply TKD query on incomplete data; however, all these had weak performances or were not compatible with the incomplete data. This thesis proposes MapReduced Enhanced Bitmap Index Guided Algorithm (MRBIG) for dealing with the aforementioned issues. MRBIG uses the MapReduce framework to enhance the performance of applying top-k dominance queries on huge incomplete datasets. The proposed approach uses the MapReduce parallel computing approach using multiple computing nodes. The framework separates the tasks between several computing nodes that independently and simultaneously work to find the result. This method has achieved up to two times faster processing time in finding the TKD query result in comparison to previously presented algorithms

GraphCombEx: A Software Tool for Exploration of Combinatorial Optimisation Properties of Large Graphs

We present a prototype of a software tool for exploration of multiple combinatorial optimisation problems in large real-world and synthetic complex networks. Our tool, called GraphCombEx (an acronym of Graph Combinatorial Explorer), provides a unified framework for scalable computation and presentation of high-quality suboptimal solutions and bounds for a number of widely studied combinatorial optimisation problems. Efficient representation and applicability to large-scale graphs and complex networks are particularly considered in its design. The problems currently supported include maximum clique, graph colouring, maximum independent set, minimum vertex clique covering, minimum dominating set, as well as the longest simple cycle problem. Suboptimal solutions and intervals for optimal objective values are estimated using scalable heuristics. The tool is designed with extensibility in mind, with the view of further problems and both new fast and high-performance heuristics to be added in the future. GraphCombEx has already been successfully used as a support tool in a number of recent research studies using combinatorial optimisation to analyse complex networks, indicating its promise as a research software tool

Evolution of galaxy groups in the Illustris simulation

We present the first study of evolution of galaxy groups in the Illustris simulation. We focus on dynamically relaxed and unrelaxed galaxy groups representing dynamically evolved and evolving galaxy systems, respectively. The evolutionary state of a group is probed from its luminosity gap and separation between the brightest group galaxy and the center of mass of the group members. We find that the Illustris simulation, over-produces large luminosity gap galaxy systems, known as fossil systems, in comparison to observations and the probed semi-analytical predictions. However, this simulation is equally successful in recovering the correlation between luminosity gap and luminosity centroid offset, in comparison to the probed semi-analytic model. We find evolutionary tracks based on luminosity gap which indicate that a large luminosity gap group is rooted in a small luminosity gap group, regardless of the position of the brightest group galaxy within the halo. This simulation helps, for the first time, to explore the black hole mass and its accretion rate in galaxy groups. For a given stellar mass of the brightest group galaxies, the black hole mass is larger in dynamically relaxed groups with a lower rate of mass accretion. We find this consistent with the latest observational studies of the radio activities in the brightest group galaxies in fossil groups. We also find that the IGM in dynamically evolved groups is hotter for a given halo mass than that in evolving groups, again consistent with earlier observational studies.Comment: 10 pages, 10 figures. Accepted for publication in Ap

Truss Decomposition in Massive Networks

The k-truss is a type of cohesive subgraphs proposed recently for the study of networks. While the problem of computing most cohesive subgraphs is NP-hard, there exists a polynomial time algorithm for computing k-truss. Compared with k-core which is also efficient to compute, k-truss represents the "core" of a k-core that keeps the key information of, while filtering out less important information from, the k-core. However, existing algorithms for computing k-truss are inefficient for handling today's massive networks. We first improve the existing in-memory algorithm for computing k-truss in networks of moderate size. Then, we propose two I/O-efficient algorithms to handle massive networks that cannot fit in main memory. Our experiments on real datasets verify the efficiency of our algorithms and the value of k-truss.Comment: VLDB201

The redshift evolution of the distribution of star formation among dark matter halos as seen in the infrared

Recent studies revealed a strong correlation between the star formation rate (SFR) and stellar mass of star-forming galaxies, the so-called star-forming main sequence. An empirical modeling approach (2-SFM) which distinguishes between the main sequence and rarer starburst galaxies is capable of reproducing most statistical properties of infrared galaxies. In this paper, we extend this approach by establishing a connection between stellar mass and halo mass with the technique of abundance matching. Based on a few, simple assumptions and a physically motivated formalism, our model successfully predicts the (cross-)power spectra of the cosmic infrared background (CIB), the cross-correlation between CIB and cosmic microwave background (CMB) lensing, and the correlation functions of bright, resolved infrared galaxies measured by Herschel, Planck, ACT and SPT. We use this model to infer the redshift distribution these observables, as well as the level of correlation between CIB-anisotropies at different wavelengths. We also predict that more than 90% of cosmic star formation activity occurs in halos with masses between 10^11.5 and 10^13.5 Msun. Taking into account subsequent mass growth of halos, this implies that the majority of stars were initially (at z>3) formed in the progenitors of clusters, then in groups at 0.5<z<3 and finally in Milky-Way-like halos at z<0.5. At all redshifts, the dominant contribution to the star formation rate density stems from halos of mass ~10^12 Msun, in which the instantaneous star formation efficiency is maximal (~70%). The strong redshift-evolution of SFR in the galaxies that dominate the CIB is thus plausibly driven by increased accretion from the cosmic web onto halos of this characteristic mass scale

Influence of baryonic physics in galaxy simulations: a semi-analytic treatment of the molecular component

Recent work in galaxy formation has enlightened the important role of baryon physics, to solve the main problems encountered by the standard theory at the galactic scale, such as the galaxy stellar mass functions, or the missing satellites problem. The present work aims at investigating in particular the role of the cold and dense molecular phase, which could play a role of gas reservoir in the outer galaxy discs, with low star formation efficiency. Through TreeSPH simulations, implementing the cooling to low temperatures, and the inclusion of the molecular hydrogen component, several feedback efficiencies are studied, and results on the gas morphology and star formation are obtained. It is shown that molecular hydrogen allows some slow star formation (with gas depletion times of about 5 Gyr) to occur in the outer parts of the discs. This dense and quiescent phase might be a way to store a significant fraction of dark baryons, in a relatively long time-scale, in the complete baryonic cycle, connecting the galaxy discs to hot gaseous haloes and to the cosmic filaments.Comment: Accepted for publication in Astronomy and Astrophysics, 21 pages, 29 figure