Variable selection in semiparametric mixed models for longitudinal data remains a challenge, especially in the presence of multiple correlated outcomes. In this paper, we propose a model selection procedure that simultaneously selects fixed and random effects using a maximum penalized likelihood method with the adaptive least absolute shrinkage and selection operator penalty. Through random effects selection, we determine the correlation structure among multiple outcomes and therefore address whether a joint model is necessary. Additionally, we include a bivariate nonparametric component, as approximated by tensor product splines, to accommodate the joint nonlinear effects of two independent variables. We use an adaptive group least absolute shrinkage and selection operator to determine whether the bivariate nonparametric component can be reduced to additive components. To implement the selection and estimation method, we develop a two-stage expectation-maximization procedure. The operating characteristics of the proposed method are assessed through simulation studies. Finally, the method is illustrated in a clinical study of blood pressure development in children.
Keywords: Adaptive least absolute shrinkage and selection operator; adaptive group least absolute shrinkage and selection operator; expectation-maximization algorithm; mixed effects; multivariate data.