Hyperparameteroptimizationisbothapracticalissueandaninterestingtheoreticalproblemintraining of deep architectures. Despite many recent advances the most commonly used methods almost universally involve training multiple and decoupled copies of the model, in effect sampling the hyperparameter space. We show that at negligable additional computational cost, results can be improved by sampling paths instead of points in hyperparameter space. To this end we interpret hyperparameters as controlling the level of correlated noise in the training, which can be mapped to an effective temperature. High variance of such noise corresponds to the high temperature and vice versa. It is well established that various forms of noise injection, whether to inputs, weights or gradients, help generalization performance. From the theory point of view, it has been shown that the effects of such methods are analogous – though not identical – to the introducing a particular types of regularization which smoothes the optimization landscape. Furthermore, an often invoked intuition is that noise allows thealgorithmtoescapenarrowminimaofthecostfunctiondefinedbythetrainingdata,therebypreventing overfitting. This is also true for non-white noise introduced indirectly, such as the one generated by the mini-batches. In what follows we will greatly benefit from an intuitive – if not always regorous – picture relating various kinds of noise to an effective hyperparameter controlling smoothness of the potential landscape, similarinspirittotemperature. Commonhyperparameters, suchas, forinstance, learningrate, dropout, batch size can be treated on the same footing.
Figure 1: Example of exchanges between replicas differing in learning rates for the ResNet20 architecture. Shaded in brown: the path in hyperparameter space taken by the optimal simulation.
The core of our approach is to endow the model with the ability to change hyperparameters during the training, and hence, to optimize the model over a subset of paths in the combined weight and hyperparameter space, as opposed to being confined to hyperplances determined by the fixed value of the hyperparameter. Inspired by a similar problemsinstatisticalphysicsweuseparalleltempering method. InthisproceduremultipleMCMCsimulations are run in parallel, at different temperatures which define level of uncertainty in the objective function. The tempratures are arranged in ascending manner forming a ladder on which the states of the replicas are swapped between neighboring values using Metropolis acceptance criteria. This ensuresthatthewholesetupsatisfiesdetailedbalance notonlyforeachchainindividually,butalsobetweenthechains. Thelowertemperaturereplicas’ergodicity is thus radically improved by possiblity of temporarily perfoming MC moves at higher temperature which allowstoescapenarrowminimaandcontinuetoexploreweightspace. Figure??showsarealizationofour approach. Each colored curve corresponds to a paricular value of learning rate for each individual replica during training. The emphasized line corresponds to the replica that achieved optimal validation error value during training. Each replica changes its hyperparameter value only with its neighboring systems. Tovalidatetheapproachweconductedaseriesofexperiments: usingsmallLeNet-likearchitectures, we investigated the effect of various types of hyperparameter-induced “noise” on weight diffusion and tested the associated idea of hyperparameter replica exchange. To investigate the applicability of our method on the industry-standard models we performed the experiments on various ResNet architectures as well. In both cases, we showed a consistent improvement of the absolute testing error, relative to the fixed hyperparameter value.