Variational inference, (Not so) approximate Bayesian techniques, and applications

We review a number of established and more recent variational Bayesian inference techniques, which we illustrate in particular through the Sparse Bayesian Learning (SBL) problem. SBL, which was initially proposed in the Machine Learning literature, is an efficient and well-studied framework for sparse or more generally underdetermined signal recovery. SBL uses hierarchical Bayes with a decorrelated Gaussian prior in which the variance profile is also to be estimated. This is more sparsity inducing than e.g., a Laplacian prior. However, SBL does not scale with problem dimensions due to the computational complexity associated with the matrix inversion in Linear Minimum Mean Squared Error (LMMSE) estimation. To address this issue, various low complexity approximate Bayesian inference techniques have been introduced for the LMMSE component, including Variational Bayesian (VB) inference, Space Alternating Variational Estimation (SAVE) or Message Passing (MP) algorithms such as Belief Propagation (BP), Expectation Propagation (EP) or Approximate MP (AMP). These algorithms may converge to the correct LMMSE estimate, with various posterior variance estimation qualities. In this tutorial, we provide a detailed overview of the low complexity approximate Bayesian inference techniques and their superiority (in terms of convergence, computational complexity, and robustness w.r.t measurement matrices) compared to other state of the art techniques. LMMSE bricks appear in the hierarchical Bayes approach of SBL and in the Gaussian approximations (e.g., in AMP), performed by EP, which are asymptotically exact as justified by large system analysis. Apart from the generalized linear model application, of which SBL is an instance, we also consider the bilinear model which appears in (semi)blind channel estimation as e.g., in Cell-Free Massive MIMO, and the dynamic instance of SBL, which leads to adaptive and extended Kalman filtering.