Tracking a table tennis ball for umpiring purposes is a challenging task as, in real-match scenarios, the ball travels fast and can become occluded or merged with other background objects. This paper presents the design of a multi-view based tracking system that can overcome the challenges of tracking a ball in real match sequences. The system has been tested on a complete table tennis rally and the results are very promising. The system is able to continuously track the ball with only marginal variations in detection. Furthermore, the initialization of the multi-camera system means it is both a portable and cost-effective solution for umpiring purposes