Moving person detection andtracking and detectionusing color features is the initialstep in biometric system and video surveillance. The main purposeof our system is to detect the object, track the object and identify the predominantcolor by using moving or static camera. Here we track object, detect and recognized the object using RGB layer matrix and PCA(Principal Component Analysis) algorithm. This is done by the virtueof “Matlab 2015a”software. To detect the color RGB layer matrix algorithm is used. The famous approach is used to convert RGB frame into HSV( Hue Saturation Value) and extract the pixels value and R,G,B by subtracting reference value minus threshold value. The facial realization system uses PCA. In PCA every image is a sum of products of each quantity time constantsof eigen vectors which are called as eigen faces. Eigen faces obtained from Co-variance matrix