Graduate School and Research Center in Digital Sciences

A variational view on bootstrap ensembles as Bayesian inference

Milios, Dimitrios: Michiardi, Pietro; Filippone, Maurizio

Submitted to ArXiV, 8 June 2020

In this paper, we employ variational arguments to establish a connection between ensemble methods for Neural Networks and Bayesian inference. We consider an ensemble-based scheme where each model/particle corresponds to a perturbation of the data by means of parametric bootstrap and a perturbation of the prior. We derive conditions under which any optimization steps of the particles makes the associated distribution reduce its divergence to the posterior over model parameters. Such conditions do not require any particular form for the approximation and they are purely geometrical, giving insights on the behavior of the ensemble on a number of interesting models such as Neural Networks with ReLU activations. Experiments confirm that ensemble methods can be a valid alternative to approximate Bayesian inference; the theoretical developments in the paper seek to explain this behavior.

Arxiv Bibtex

Title:A variational view on bootstrap ensembles as Bayesian inference
Type:Conference
Language:English
City:
Date:
Department:Data Science
Eurecom ref:6289
Copyright: © EURECOM. Personal use of this material is permitted. The definitive version of this paper was published in Submitted to ArXiV, 8 June 2020 and is available at :
Bibtex: @inproceedings{EURECOM+6289, year = {2020}, title = {{A} variational view on bootstrap ensembles as {B}ayesian inference}, author = {{M}ilios, {D}imitrios: {M}ichiardi, {P}ietro and {F}ilippone, {M}aurizio}, booktitle = {{S}ubmitted to {A}r{X}i{V}, 8 {J}une 2020}, address = {}, month = {06}, url = {http://www.eurecom.fr/publication/6289} }
See also: