A Markovian approach to distributional semantics.

Summary This thesis, organized in two independent parts, focuses on distributional semantics and variable selection. In the first part, we introduce a new method for learning word representations from large amounts of raw text. This method is based on a probabilistic model of the sentence, using a hidden Markov model and a dependency tree. We present an efficient algorithm for performing inference and learning in such a model, based on the online EM algorithm and approximate message propagation. We evaluate the obtained models on intrinsic tasks, such as predicting human similarity judgments or categorizing words and two extrinsic tasks~: named entity recognition and supersense labeling. In the second part, we introduce, in the context of linear models, a new penalty for variable selection in the presence of highly correlated predictors. This penalty, called Lasso trace, uses the norm trace of the selected predictors, which is a convex relaxation of their rank, as a complexity criterion. The Lasso trace interpolates the $\ell_1$ and $\ell_2$ norms. In particular, when all predictors are orthogonal, it is equal to the $\ell_1$ norm, while when all predictors are equal, it is equal to the $\ell_2$ norm. We propose two algorithms to compute the solution of the Lasso trace regularized least squares regression problem and perform experiments on synthetic data.