ALQUIER Pierre

This PhD thesis deals with variational inference and robustness in statistics and machine learning. Specifically, it focuses on the statistical properties of variational approximations and the design of efficient algorithms to compute them sequentially, and studies Maximum Mean Discrepancy based estimators as learning rules that are robust to model misspecification.In recent years, variational inference has been widely studied from a computational perspective, however, the literature has paid little attention to its theoretical properties until very recently. In this thesis, we study the consistency of variational approximations in various statistical models and the conditions that ensure their consistency. In particular, we address the case of mixture models and deep neural networks. We also justify from a theoretical point of view the use of the ELBO maximization strategy, a numerical criterion that is widely used in the VB community for model selection and whose effectiveness has already been confirmed in practice. In addition, Bayesian inference provides an attractive online learning framework for analyzing sequential data, and offers generalization guarantees that remain valid even under model misspecification and in the presence of adversaries. Unfortunately, exact Bayesian inference is rarely tractable in practice and approximation methods are usually employed, but do these methods preserve the generalization properties of Bayesian inference? In this thesis, we show that this is indeed the case for some variational inference (VI) algorithms. We propose new online tempered algorithms and derive generalization bounds. Our theoretical result relies on the convexity of the variational objective, but we argue that our result should be more general and present empirical evidence in support. Our work provides theoretical justifications for online algorithms that rely on approximate Bayesian methods.Another question of major interest in statistics that is addressed in this thesis is the design of a universal estimation procedure. This question is of major interest, especially because it leads to robust estimators, a topical issue in statistics and machine learning. We address the problem of universal estimation by using a distance minimization estimator based on Maximum Mean Discrepancy. We show that the estimator is robust to both dependence and the presence of outliers in the dataset. We also highlight the links that can exist with distance minimization estimators using the L2 distance. Finally, we present a theoretical study of the stochastic gradient descent algorithm used to compute the estimator, and we support our findings with numerical simulations. We also propose a Bayesian version of our estimator, which we study from both a theoretical and a computational point of view.

The objective of this thesis is to propose new methodologies to be applied to public transportation data. Indeed, we are surrounded more and more by sensors and computers generating huge amounts of data. In the public transport domain, contactless cards generate data every time we use them, whether for loading or for our trips. In this thesis, we use this data for two distinct purposes. First, we wanted to be able to detect groups of passengers with similar temporal patterns. To do this, we first used non-negative matrix factorization as a pre-processing tool for classification. Then we introduced the NMF-EM algorithm allowing dimension reduction and classification simultaneously for a mixture model of multinomial distributions. In a second step, we applied regression methods to these data in order to be able to provide a range of these probable validations. Similarly, we applied this methodology to the detection of anomalies on the network.

No summary available.

We first focus on the parsimonious Gaussian sequence model. An empirical Bayesian approach on the a priori Spike and Slab allows us to obtain the convergence at minimax speed of the second order moment a posteriori for Cauchy Slabs and we prove a suboptimality result for a Laplace Slab. A better choice of Slab allows us to obtain the exact constant. In the density estimation model, an a priori Polya tree such that the variables in the tree have a Spike and Slab distribution gives minimax and adaptive speed convergence for the sup norm of the a posteriori law and a nonparametric Bernstein-von Mises theorem.

This thesis deals with the subject of linear regression in different frameworks, notably related to learning. The first two chapters present the context of the work, its contributions and the mathematical tools used. The third chapter is devoted to the construction of an optimal regularization function, allowing for example to improve on the theoretical level the regularization of the LASSO estimator. The fourth chapter presents, in the field of convex sequential optimization, accelerations of a recent and promising algorithm, MetaGrad, and a conversion from a so-called "deterministic sequential" framework to a so-called "stochastic batch" framework for this algorithm. The fifth chapter focuses on successive interval forecasts, based on the aggregation of predictors, without intermediate feedback or stochastic modeling. Finally, the sixth chapter applies several aggregation methods to a petroleum dataset, resulting in short-term point forecasts and long-term forecast intervals.

The main part of this thesis focuses on developing the theoretical and algorithmic aspects of three distinct statistical procedures. The first problem is the completion of binary matrices. We propose an estimator based on a pseudo-Bayesian variational approximation using a loss function different from those used previously. We can compute non-asymptotic bounds on the integrated risk. The proposed estimator is much faster to compute than an MCMC type estimate and we show on examples that it is efficient in practice. The second problem is the study of the theoretical properties of the empirical risk minimizer for lipschitzian loss functions. We can then apply the main results on logistic regression with SLOPE penalization and on matrix completion. The third chapter develops an Expectation-Propagation approximation when the likelihood is not explicit. We then use the ABC approximation in a second step. This procedure can be applied to many models and is much more accurate and fast. It is applied as an example on a model of spatial extremes.

Quantum state tomography, an important task in quantum information processing, aims at reconstructing a state from prepared measurement data. Bayesian methods are recognized to be one of the good and reliable choice in estimating quantum states~\cite{blume2010optimal}. Several numerical works showed that Bayesian estimations are comparable to, and even better than other methods in the problem of $1$-qubit state recovery. However, the problem of choosing prior distribution in the general case of $n$ qubits is not straightforward. More importantly, the statistical performance of Bayesian type estimators have not been studied from a theoretical perspective yet. In this paper, we propose a novel prior for quantum states (density matrices), and we define pseudo-Bayesian estimators of the density matrix. Then, using PAC-Bayesian theorems, we derive rates of convergence for the posterior mean. The numerical performance of these estimators are tested on simulated and real datasets.

This paper deals with the high dimensionality problem in multivariate GARCH processes. The author proposes a new vine-GARCH dynamics for correlation processes parameterized by an undirected graph called "vine". This approach directly generates definite-positive matrices and encourages parsimony. After establishing existence and uniqueness results for stationary solutions of the vine-GARCH model, the author analyzes the asymptotic properties of the model. He then proposes a general framework of penalized M-estimators for dependent processes and focuses on the asymptotic properties of the adaptive Sparse Group Lasso estimator. The high dimension is treated by considering the case where the number of parameters diverges with the sample size. The asymptotic results are illustrated by simulated experiments. Finally in this framework the author proposes to generate the sparsity for dynamics of variance-covariance matrices. To do so, the class of multivariate ARCH models is used and the corresponding processes are estimated by penalized ordinary least squares.

No summary available.

Let $(\bX, Y)$ be a random pair taking values in $\mathbb R^p \times \mathbb R$. In the so-called single-index model, one has $Y=f^{\star}(\theta^{\star T}\bX)+\bW$, where $f^{\star}$ is an unknown univariate measurable function, $\theta^{\star}$ is an unknown vector in $\mathbb R^d$, and $W$ denotes a random noise satisfying $\mathbb E[\bW|\bX]=0$. The single-index model is known to offer a flexible way to model a variety of high-dimensional real-world phenomena. However, despite its relative simplicity, this dimension reduction scheme is faced with severe complications as soon as the underlying dimension becomes larger than the number of observations (''$p$ larger than $n$'' paradigm). To circumvent this difficulty, we consider the single-index model estimation problem from a sparsity perspective using a PAC-Bayesian approach. On the theoretical side, we offer a sharp oracle inequality, which is more powerful than the best known oracle inequalities for other common procedures of single-index recovery. The proposed method is implemented by means of the reversible jump Markov chain Monte Carlo technique and its performance is compared with that of standard procedures.

We establish rates of convergences in time series forecasting using the statistical learning approach based on oracle inequalities. A series of papers extends the oracle inequalities obtained for iid observations to time series under weak dependence conditions. Given a family of predictors and $n$ observations, oracle inequalities state that a predictor forecasts the series as well as the best predictor in the family up to a remainder term $\Delta_n$. Using the PAC-Bayesian approach, we establish under weak dependence conditions oracle inequalities with optimal rates of convergence. We extend previous results for the absolute loss function to any Lipschitz loss function with rates $\Delta_n\sim\sqrt{ c(\Theta)/ n}$ where $c(\Theta)$ measures the complexity of the model. We apply the method for quantile loss functions to forecast the french GDP. Under additional conditions on the loss functions (satisfied by the quadratic loss function) and on the time series, we refine the rates of convergence to $\Delta_n \sim c(\Theta)/n$. We achieve for the first time these fast rates for uniformly mixing processes. These rates are known to be optimal in the iid case and for individual sequences. In particular, we generalize the results of Dalalyan and Tsybakov on sparse regression estimation to the case of autoregression.

We introduce a new method to reconstruct the density matrix $\rho$ of a system of $n$-qubits and estimate its rank $d$ from data obtained by quantum state tomography measurements repeated $m$ times. The procedure consists in minimizing the risk of a linear estimator $\hat{\rho}$ of $\rho$ penalized by given rank (from 1 to $2^n$), where $\hat{\rho}$ is previously obtained by the moment method. We obtain simultaneously an estimator of the rank and the resulting density matrix associated to this rank. We establish an upper bound for the error of penalized estimator, evaluated with the Frobenius norm, which is of order $dn(4/3)^n /m$ and consistency for the estimator of the rank. The proposed methodology is computationaly efficient and is illustrated with some example states and real experimental data sets.

We introduce a new method to reconstruct the density matrix $\rho$ of a system of $n$-qubits and estimate its rank $d$ from data obtained by quantum state tomography measurements repeated $m$ times. The procedure consists in minimizing the risk of a linear estimator $\hat{\rho}$ of $\rho$ penalized by given rank (from 1 to $2^n$), where $\hat{\rho}$ is previously obtained by the moment method. We obtain simultaneously an estimator of the rank and the resulting density matrix associated to this rank. We establish an upper bound for the error of penalized estimator, evaluated with the Frobenius norm, which is of order $dn(4/3)^n /m$ and consistency for the estimator of the rank. The proposed methodology is computationaly efficient and is illustrated with some example states and real experimental data sets.

Adaptive, Inductive and Transductive Inference for Regression and Density Estimation (Pierre Alquier) The purpose of this thesis is to study the statistical properties of some learning algorithms in the case of regression and density estimation. It is divided into three parts. The first part consists in a generalization of Olivier Catoni's PAC-Bayesian theorems on classification to the case of regression with a general loss function. In the second part, we study more particularly the case of least squares regression and we propose a new variable selection algorithm. This method can be applied in particular to the case of a basis of orthonormal functions, and then leads to optimal convergence speeds, but also to the case of kernel type functions, it then leads to a variant of the so-called "support vector machines" (SVM) methods. The third part extends the results of the second part to the case of density estimation with quadratic loss.