The role of mathematics in cancer research has steadily increased over time. Multidisciplinary
collaboration in cancer research is essential and mathematical applications can
significantly contribute to many areas of cancer research. For example, mathematical
models can provide deeper insight and establish a framework for understanding properties
of cancer cells. Modeling the effects of radiation on cancer cells is one of the most
interesting areas in mathematical biology and a variety of models by using the Target
theory and DNA fragmentations have been applied to describe how radiation influence
tumor cells. In this study, two new mathematical frameworks are proposed to model the
population dynamics of heterogeneous tumor cells after the treatment with external beam
radiation. The first model is derived based on the Target Theory and Hit Theory. According
to these theories, the tumor population is divided into m different sub-populations
based on the different effects of ionizing radiations on human cells. This model consists
of a system of differential equations with random variable coefficients representing
the dynamics transition rates between sub-populations. The model is also describing
the heterogeneity of the cell damage and the repair mechanism between two consecutive
dose fractions. In the second model, we study the population dynamics of breast cancer
cells treated with radiotherapy by using a system of stochastic differential equations. According
to the cell cycle, each cell belongs to one of three subpopulations G, S, or M,
representing gap, synthesis, and mitosis subpopulations. Cells in the M subpopulation
are highly radio-sensitive, whereas cells in the S subpopulation are highly radio-resistant.
Therefore, in the process of radiotherapy, cell death rates of different subpopulations are not equal. In addition, since flow cytometry is unable to detect apoptotic cells accurately,
the small changes in cell death rate in each subpopulation during treatment are considered.
Therefore, a new definition for the lifespan of the tumor based on population size is
introduced. Tumor Lifespan is defined as the minimum number of dose fractions needed
to remove the whole tumor. The stability of the first model is studied by considering three
cases. For the first and second cases, we assumed that each cell has two and three targets
(m = 2 and m = 3). Applying Routh-Hurwitz criterion, it is proven that the system is
stable when the probability that one target becomes deactivated after the application of
a dose fraction (q) is greater than or equal to 0.5. Finally, the system stability for the
third case is investigated analytically when each cell assumed has m targets. By using
Gershgorin theorem, it is shown that the system is stable where q > 0:5. In the second
model, the existence and uniqueness of the solution are proven and an explicit solution
for the SDE model is presented. Moreover, the system stability is investigated via a necessary
and sufficient condition on model parameters. The transition rates are estimated
in a steady state condition. Subsequently, the model is solved numerically using Euler-
Murayama and Milstein methods and the other parameters of the model are estimated
using parametric and nonparametric simulated likelihood estimation parameter methods.
Finally, we did a number of experiments on MCF-7 breast cancer cell line. The cell cycle
analysis assay has been used to analyze experimental data. Then the obtained data is applied
and able to calibrate and verify our models
