We investigated the detection of galaxy clusters with their weak gravitational lensing effect. Galaxy clusters are the most massive gravitationally bound systems in the Universe, and their abundance and evolution provide important information about the structure formation in the cosmological context. Weak gravitational lensing makes use of the distortion of the background galaxy images to probe the matter distribution of the clusters. In this work, we present three methods for the weak lensing study of galaxies clusters. We first developed a program ExAM, which automatically and efficiently masks image defects contaminating the shape measurement of galaxies. The output of the program can be used to remove the problematic detections of galaxy in the image. We then present a catalog-space simulation of cluster weak lensing, Shuff, featuring the configurable intrinsic properties of galaxies, multiple triaxial cluster halos as lenses, and realizations of cosmological density perturbation field as a source of shear noise. It takes a catalog of unlensed galaxies, applies the multiple lens plane lensing effects, and outputs a lensed galaxies catalog, which can either be used to produce image of galaxies for the shape measurement calibration, or be taken as the input of cluster detection based on the galaxy ellipticities. We also developed a program for cluster detection, LensFilter, implementing the optimal filtering techniques suggested by Maturi et.al. (2004). We generalize the techniques for detection of specific clusters with configurable galaxies redshift distribution and noise properties. The filtering detection is first applied to the Shuff simulation, for an investigation of limit of cluster detection with weak lensing, and further applied to the CFHTLS data with comparison to cluster detection from X-ray observations
