Document details for 'Finite-size effects in Bayesian model selection and generalization'

Authors Marion, G. and Saad, D.
Publication details Journal of Physics A 29, 5387-5404.
Keywords statistical mechanics, neural networks
Abstract We show that in supervised learning from a supplied data set Bayesian model selection, based on the evidence, does not optimise generalisation performance even for a learnable linear problem. This is demonstrated by examining the finite size effects in hyperparameter assignment from the evidence procedure and the resultant generalisation performance. Our approach demonstrates the weakness of average case and asymptotic analyses. Using simulations we corroborate our analytic results and examine an alternative model selection criterion, namely cross-validation. This numerical study shows that the cross-validation hyperparameter estimates correlate more strongly than those of the evidence with optimal performance. However, we show that for a sufficiently large input dimension the evidence procedure could provide a reliable alternative to the more computationally expensive cross-validation.
Last updated 2003-05-13
Files
  1. paper.ps.gz
  2. paper.pdf

Unless explicitly stated otherwise, all material is copyright © Biomathematics and Statistics Scotland.

Biomathematics and Statistics Scotland (BioSS) is formally part of The James Hutton Institute (JHI), a registered Scottish charity No. SC041796 and a company limited by guarantee No. SC374831. Registered Office: JHI, Invergowrie, Dundee, DD2 5DA, Scotland