Document details for 'Hyper-parameters, evidence and generalization for an unrealisable rule'

Authors Marion, G. and Saad, D.
Publication details In "Advances in Neural Information Processing Systems", 255-262. Eds. Tesauro, G., Touretzky, D. S. and Leen, T. K.. The MIT Press, Cambridge, Massachusetts.
Publisher details The MIT Press, Cambridge, Massachusetts
Keywords statistical mechanics, neural networks
Abstract Using a statistical mechanical formalism we calculate the evidence, generalisation error and consistency measure for a linear perceptron trained and tested on a set of examples generated by a non-linear teacher. The teacher is said to be unrealisable because the student can never model it without error. Our model allows us to interpolate between the known case of a linear teacher, and an unrealisable, nonlinear teacher. A comparison of the hyperparameters which maximise the evidence with those that optimise the performance measures reveals that, in the non-linear case, the evidence procedure is a misleading guide to optimising performance. Finally, we explore the extent to which the evidence procedure is unreliable and find that, despite being sub-optimal, in some circumstances it might be a useful method for fixing the hyperparameters.
ISBN 0-262-20104-6
Last updated 2008-06-11
  2. paper.pdf

Unless explicitly stated otherwise, all material is copyright © Biomathematics and Statistics Scotland.

Biomathematics and Statistics Scotland (BioSS) is formally part of The James Hutton Institute (JHI), a registered Scottish charity No. SC041796 and a company limited by guarantee No. SC374831. Registered Office: JHI, Invergowrie, Dundee, DD2 5DA, Scotland