Approximate inference via variational sampling
Abstract
A new method called “variational sampling” is proposed to estimate integrals under probability distributions that can be evaluated up to a normalizing constant. The key idea is to fit the target distribution with an exponential family model by minimizing a strongly consistent empirical approximation to the Kullback-Leibler divergence computed using either deterministic or random sampling. It is shown how variational sampling differs conceptually from both quadrature and importance sampling and established that, in the case of random independence sampling, it may have much faster stochastic convergence than importance sampling under mild conditions. The variational sampling implementation presented in this paper requires a rough initial approximation to the target distribution, which may be found, e.g. using the Laplace method, and is shown to then have the potential to substantially improve over several existing approximate inference techniques to estimate moments of order up to two of nearly-Gaussian distributions, which occur frequently in Bayesian analysis. In particular, an application of variational sampling to Bayesian logistic regression in moderate dimension is presented.
About this article
References
[1] C. Andrieu, N. de Freitas, A. Doucet, and M. Jordan. An introduction to MCMC for machine learning. Machine Learning, 50:5–43, 2003.
[2] C. Andrieu and J. Thoms. A tutorial on adaptive MCMC. Statistics and Computing, 18(4):343–373, 2008.
[3] K. Bache and M. Lichman. UCI machine learning repository, 2013.
[4] M. J. Beal and Z. Ghahramani. The variational Bayesian EM algorithm for incomplete data: with application to scoring graphical model structures. In Bayesian Statistics 7. Oxford University Press, 2003.
[5] L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems, pages 161–168, 2008.
View more references (29)
[6] C. G. Bucher. Adaptive sampling – an iterative fast Monte Carlo procedure. Structural Safety, 5:119–126,1988.
[7] O. Capp´e, A. Guilin, J.-M. Marin, and C. Robert. Population Monte Carlo. Journal of Computational and Graphical Statistics, 13(4):907–929, 2004.
[8] P. Carbonetto, M. King, and F. Hamze. A Stochastic approximation method for inference in probabilistic graphical models. In Advances in Neural Information Processing Systems, pages 216–224, 2009. International Journal of Advanced Statistics and Probability 119
[9] P.-T. De Boer, D. Kroese, S. Mannor, and R. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19–67, 2005.
[10] P. Del Moral, A. Doucet, and A. Jasra. Sequential Monte Carlo samplers. Journal of the Royal Statistical Society: Series B, 68(3):411–436, 2006.
[11] A. Gelman and X.-L. Meng. Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statistical Science, 13:163–185, 1998.
[12] T. Gerstner and M. Griebel. Numerical integration using sparse grids. Numerical algorithms, 18(3-4):209–232,1998.
[13] M. Girolami and B. Calderhead. Riemann manifold Langevin and Hamiltonian Monte Carlo methods. Journal of the Royal Statistical Society: Series B, 73(2):123–214, 2011.
[14] P. Hall, J. Ormerod, and M. Wand. Theory of Gaussian variational approximation for a Poisson mixed model. Statistica Sinica, 21:369–389, 2011.
[15] T. Jaakkola and M. Jordan. A variational approach to Bayesian logistic regression models and their extensions. In Sixth International Workshop on Artificial Intelligence and Statistics, 1997.
[16] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods forgraphical models. Machine Learning, 37(2):183–233, 1999.
[17] S. J. Julier and J. K. Uhlmann. Unscented filtering and nonlinear estimation. Proceedings of the IEEE, 92(3):401–422, 2004.
[18] D. A. Knowles and T. Minka. Non-conjugate variational message passing for multinomial and binary regression. In Advances in Neural Information Processing Systems, pages 1701–1709, 2011.
[19] M. A. Little, P. E. McSharry, S. J. Roberts, D. A. Costello, and I. M. Moroz. Exploiting nonlinear recurrence and fractal scaling properties for voice disorder detection. BioMedical Engineering OnLine, 6(1):23, 2007.
[20] N. Metropolis. The beginning of the Monte Carlo method. Los Alamos Science, 15:125–130, 1987. Special Issue.
[21] T. P. Minka. Expectation Propagation for approximate Bayesian inference. In Uncertainty in Artificial Intelligence, pages 362–369. Morgan Kaufmann, 2001.
[22] T. P. Minka. Divergence measures and message passing. Technical Report MSR-TR-2005-173, Microsoft Research Ltd., Cambridge, UK, Dec. 2005.
[23] R. Neal. Annealed importance sampling. Statistics and Computing, 11(2):125–139, 2001.
[24] A. O’Hagan. Bayes-Hermite quadrature. Journal of Statistical Planning and Inference, 29:245–260, 1991.
[25] M. Opper and C. Archambeau. The variational Gaussian approximation revisited. Neural computation, 21(3):786–792, 2009.
[26] J. Ormerod and M. Wand. Gaussian variational approximate inference for generalized linear mixed models. Journal of Computational and Graphical Statistics, 21(1):2–17, 2012.
[27] A. Owen and Y. Zhou. Safe and effective importance sampling. Journal of the American Statistical Association, 95(449):135–143, 2000.
[28] C. Rasmussen and C. Williams. Gaussian processes for machine learning. MIT Press, 2006.
[29] C. E. Rasmussen and Z. Ghahramani. Bayesian Monte Carlo. In Advances in Neural Information Processing Systems, pages 489–496. MIT Press, 2003.
[30] M. W. Seeger and D. P. Wipf. Variational Bayesian inference techniques. Signal Processing Magazine, IEEE, 27(6):81–91, 2010.
[31] J. Shao. Monte Carlo approximations in Bayesian decision theory. Journal of the American Statistical Association, 84(407):727–732, 1989. 120 International Journal of Advanced Statistics and Probability
[32] J. Skilling. Nested sampling for general Bayesian computation. Bayesian Analysis, 1(4):833–860, 2006.
[33] L. Tierney and J. B. Kadane. Accurate approximations for posterior moments and marginal densities. Journal of the American Statistical Association, 81(393):82–86, 1986.
[34] G. Wei and M. A. Tanner. A Monte Carlo implementation of the EM algorithm and the poor man’s data augmentation algorithm. Journal of the American Statistical Association, 85:699–704, 1990