Training generative models using per-pixel loss often results in poor generated images. Better perceptual quality can be achieved using feature-based loss functions which use pre-trained models as feature extractors. Unfortunately, pre-trained models usually come with some drawbacks. First, they are often pre-trained on a task that is unrelated to the main task. In addition, they are usually big and computationally heavy. Our project goal was to Investigate the need for pre-trained networks during VAE training by replacing them with a bootstrapped version of the trained model. As part of the project, the standard MSE loss which is used in the process of VAE training was replaced by a Perceptual loss based on features extracted from an EMA bootstrapped version of the VAE encoder. The code for the project was mainly written using Pytorch and the trained model was evaluated on fashion MNIST and SVHN datasets.