Luttrell S P, November 1991, Self-supervised training of hierarchical vector quantisers, Proceedings of 2nd International Conference on Artifical Neural Networks (Bournemouth, UK), IEE Conference Publication (, ed. ), vol. , pp. 5-9
In (Luttrell) we developed a hierarchical vector quantisation (VQ) model, and in (Luttrell) we successfully applied it to time series and image compression respectively. The goal of this paper is to derive an extension to this model, in which we backpropagate signals from higher to lower layers of the hierarchy to self-supervise the training of the VQ. We review the basic properties of our VQ model and its relationship to neural network methods. We extend the model to an ensemble of VQs, and we derive its properties in the limit of a large codebook size (i.e. the continuum limit). Finally, we demonstrate how self-supervision emerges naturally in this type of model.