Biomathematics & Statistics Scotland
Document details for 'Hyper-parameters, evidence and generalization for an unrealisable rule'
| Authors | Marion, G. and Saad, D. |
|---|---|
| Publication details | In "Advances in Neural Information Processing Systems", 255-262. Eds. Tesauro, G., Touretzky, D. S. and Leen, T. K.. The MIT Press, Cambridge, Massachusetts. |
| Publisher details | The MIT Press, Cambridge, Massachusetts |
| Keywords | statistical mechanics, neural networks |
| Abstract | Using a statistical mechanical formalism we calculate the evidence, generalisation error and consistency measure for a linear perceptron trained and tested on a set of examples generated by a non-linear teacher. The teacher is said to be unrealisable because the student can never model it without error. Our model allows us to interpolate between the known case of a linear teacher, and an unrealisable, nonlinear teacher. A comparison of the hyperparameters which maximise the evidence with those that optimise the performance measures reveals that, in the non-linear case, the evidence procedure is a misleading guide to optimising performance. Finally, we explore the extent to which the evidence procedure is unreliable and find that, despite being sub-optimal, in some circumstances it might be a useful method for fixing the hyperparameters. |
| Date entered | 2003-05-13 |
| Last updated | 2008-06-11 |
| Files |