Interpolation Based Distributed Learning

Abstract

Artificial neural networks are trained with gradient descent-based algorithms in a complex parameter error space. As a result of the training, a point (local minimum) with a small error on the training set is obtained in this space. However, due to the random processes of the training algorithms, different local minima are reached even if starting from very close points using the same data set. When linear interpolation is made between these reached points, it is seen that the error values of the obtained points are high. In this study, it has been shown that this situation does not occur among models trained in parallel with different data after a common training, and that very successful results can be achieved when training continues from the best point obtained by interpolation. In other words, instead of training sequentially with all the data, more successful results could be obtained by adding a parallel process to the training process in approximately the same time. This approach, which does not require information sharing between models throughout the parallel process, differs from the literature in this respect and offers an innovative training method for artificial neural networks.