The budding yeast Saccharomyces cerevisiae is an important model organism for cellular aging. A common metric for determining the lifespan of budding yeast cells is the replicative lifespan (RLS), how many times a mother cell divides in its lifetime. Traditionally, determining the RLS of yeast cells is a tedious manual process. To address this challenge, our long-term goal is to develop an automated RLS estimation process. Recently microfluidics-based methods have been developed, which generate time- series of images of individual cells. This work is focused on classifying these images into categories which can be used to estimate the RLS. We test three different deep learning models and found that all of the models have diverse and complementary errors, so we developed an ensemble of models that combine the best single models which led to high overall accuracy, precision and recall