Statistical modeling of medical data and theoretical analysis of estimation algorithms.

Summary In the medical field, the use of features extracted from images is more and more widespread. These measures can be real numbers (volume, cognitive score), organ meshes or the image itself. In these last two cases, a Euclidean space cannot describe the space of measures and it is necessary to place oneself on a Riemannian variety. Using this Riemannian framework and mixed effects models, it is then possible to estimate a representative object of the population as well as the inter-individual variability. In the longitudinal case (subjects observed repeatedly over time), these models allow to create an average trajectory representative of the global evolution of the population. In this thesis, we propose to generalize these models in the case of a mixed population. Each sub-population can follow different dynamics over time and their representative trajectory can be the same or differ from one time interval to another. This new model allows for example to model the onset of a disease as a deviation from normal aging.We are also interested in the detection of anomalies (e.g. tumors) in a population. With an object representing a control population, we define an anomaly as what cannot be reconstructed by diffeomorphic deformation of this representative object. Our method has the advantage of requiring neither a large dataset nor annotation by physicians and can be easily applied to any organ.Finally, we focus on different theoretical properties of the estimation algorithms used. In the context of nonlinear mixed effects models, the MCMC-SAEM algorithm is used. We will discuss two theoretical limitations. First, we will lift the geometric ergodicity assumption by replacing it with a sub-geometric ergodicity assumption. Furthermore, we will focus on a method to apply the SAEM algorithm when the joint distribution is not exponentially curved. We will show that this method introduces a bias in the estimate that we will measure. We will also propose a new algorithm to reduce it.