Image segmentation is the key topic in computer vision and image processing with applications like robotic perception, scene understanding, video surveillance, image compression, medical image analysis, and augmented reality among many others. There are numerous algorithms are developed in the literature for image segmentation. This paper provides a broad spectrum of pioneering works for instance and semantic level segmentation with mask Region based Convolution Neural Network with Monarch butterfly Optimization (RCNN-MBO) architecture. The system is initially constructed in a Python environment with images of people and animals being input. Remove the unnecessary data from the gathered datasets during the pre-processing stage. Next, use a stochastic threshold function to segment the image. Then update the segmented images into a designed model for detecting and classifying a group of images. The main goal of the designed approach is to attain accurate prediction results also improve the performance of the designed model by attaining better results. To enhance the performance, two activation functions were used and MBO fitness is updated in the classification layer. It improves the prediction results and takes less time to detect and classify images. Finally, the experimental outcomes show the reliability of the designed approach by other conventional techniques in terms of accuracy, precision, sensitivity, specificity, F-measure, error rate, and computation time
