The accuracy of the estimated modal damping ratios in operational modal analysis (OMA) remains an open issue and is often characterized by a large error. The modal damping ratio is considered to be a good practical parameter for structural damage detection due to its sensitivity and sufficient responsiveness to damage compared to natural frequency and mode shape. Therefore, an accurate estimate of the modal damping ratio will assist in developing an effective modal-based structural damage detection approach. The objective of this research focuses on improvements of frequency domain decomposition (FDD) and stochastic subspace identification (SSI) algorithms, particularly in estimating modal damping ratio. These methods have gained a lot of attention and interest compared to other OMA methods due to their ability in estimating modal parameters. However, FDD has a problem dealing with high damping levels, while SSI has difficulty in handling harmonic components. This will cause a large error in estimating the modal damping ratio. Difficulties also arise for automation of SSI as several predefined set parameters are compulsory at start-up for each analysis. This study introduces an iterative loop of advanced optimization to enhance the capabilities of classical FDD algorithm by optimizing the value of the modal assurance criterion (MAC) index and the selection of the correct time window on the auto-correlation function that represents the most challenging part of the algorithms. This study also presents the development of the SSI framework in automated OMA and harmonic removal method using image-based feature extraction along with the application of empirical mode decomposition. The implementation of image-based feature extraction can be used for clustering and classification of harmonic components from structural poles as well as to identify modal parameters by neglecting any calibration or user-defined parameter at start-up. The proposed approach is assessed through experimental and numerical simulation analysis. Based on the numerical simulation results, the proposed optimized FDD can estimate modal damping ratio with high accuracy and consistency by showing average percentage deviation (error) below 5.50% compared to classical FDD and benchmark approach, which is a refined FDD. Errors in classical FDD can reach an average of up to 15%, whereas for refined FDD the average is around 10%. Meanwhile, the results of the proposed approach in experimental verification show a reasonable average percentage deviation of about 5.75%, while the classical FDD algorithm is overestimated which averages about 29% in all cases. For the proposed automation of SSI, the estimated results of modal damping ratio in the numerical simulation are below 2.5% of the average error compared to other SSI methods which on average exceed 3.2%. For experimental verification, the results of the proposed approach indicate very satisfactory agreement by showing average deviation percentage below 4.20% compared to other SSI methods which on average exceeds 14%. Furthermore, the results of the proposed automated harmonic removal in SSI framework for estimating modal damping ratio using existing online experimental data sets demonstrate very high accuracy and consistent results after removing harmonic components, showing an average deviation percentage of below 7.22% compared to orthogonal projection and smoothing technique based on linear interpolation approaches where the average deviation percentage exceeds 9%
