Nonparametric estimation in a mixed-effect Ornstein–Uhlenbeck model.

Summary Two adaptive nonparametric procedures are proposed to estimate the density of the random effects in a mixed-effect Ornstein-Uhlenbeck model. First an estimator using deconvolution tools is introduced, which depends on two tuning parameters to be chosen in a data-driven way. The selection of these two parameters is achieved with a Goldenshluger and Lepski's method, extended to this particular case with a new two-dimensional penalized criterion. Then, we propose a kernel estimator of the density of the random effect, with a new bandwidth selection method. For both data driven estimators, risk bounds are provided in term of integrated $\mathbb{L}^2$-error. The estimators are evaluated on simulations and show good results. Finally, these nonparametric estimators are applied to a neuronal database of interspike intervals, and are compared with a previous parametric estimation.