This paper aims to investigate the human silhouettes in video frames, which involves normalizing the silhouettes and describing the shape of the region in video frames using Shape Fourier Descriptors. Shape Fourier Descriptor describes the shape of an object by considering its boundaries, which are the shape centroid and calculated by a particular formula through all the video frames after normalized the videos. This shows the changes of the objects with various actions and can be recognized and characterized human or non-human in the video frames. Normalized Silhouette Image is significant before the videos are being processed to describe in descriptors. It focuses on the region based on the object’s ratio in images of the shape of the object and silhouette images are centred after action segmentation. This reduces the burden of the process of extract unnecessary part in whole videos. Various human action videos and animal videos are used for the training and testing in this study to make sure the system performed better