The substitution box (S-box) plays a vital role in creating confusion during the encryption process of digital data. The quality of encryption schemes depends upon the S-box. There have been several attempts to enhance the quality of the S-box by using fractal chaotic mechanisms. However, there is still weakness in the robustness against cryptanalysis of fractal-based S-boxes. Due to their chaotic behavior, fractals are frequently employed to achieve randomness by confusion and diffusion process. A complex number-based S-box and a chaotic map diffusion are proposed to achieve high nonlinearity and low correlation. This study proposed a Mandelbrot set S-box construction based on the complex number and Chen chaotic map for resisting cryptanalytic attacks by creating diffusion in our proposed algorithm. The cryptosystem was built on the idea of substitution permutation networks (SPN). The complex nature of the proposed S-box makes it more random than other chaotic maps. The robustness of the proposed system was analyzed by different analysis properties of the S-box, such as nonlinearity, strict avalanche criterion, Bit independent criterion, and differential and linear probability. Moreover, to check the strength of the proposed S-box against differential and brute force attacks, we performed image encryption with the proposed S-box. The security analysis was performed, including statistical attack analysis and NIST analysis. The analysis results show that the proposed system achieves high-security standards than existing schemes
