Random effect models have become a mainstream statistical technique over the last decades; and the same can be said for response transformation models such as the Box-Cox transformation. The latter ensures that the assumptions of normality and of homoscedasticity of the response distribution are fulfilled, which are essential conditions for the use of a linear model or a linear mixed model. However, methodology for response transformation and simultaneous inclusion of random effects has been developed and implemented only scarcely, and is so far restricted to Gaussian random effects. The first aim of this thesis is to develop such methodology, thereby not requiring parametric assumptions on the distribution of the random effects. This is achieved by extending the “Nonparametric Maximum Likelihood” towards a “Nonparametric Profile Maximum Likelihood” (NPPML) technique. The implemented techniques allow to deal with overdispersion as well as two-level data scenarios in general linear models.
The second part of this thesis considers the transformation of mixed-effects logistic models, with the aim of improving model fit. In binary data, link functions other than the logit can be used to connect predictors with the response. The Box-Cox transformation is used in mixed–effects binary regression models as an alternative link function for linearization purposes. The NPPML approach is used similarly as before, with some adjustments.
The proposed approach is implemented in the R package boxcoxmix. Simulation studies and applications on real data are carried out to study the performance of this approach
