| #!/usr/bin/env python3 | |
| """ | |
| To use: | |
| 1. install/set-up the google cloud api and dependencies listed on https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/texttospeech/cloud-client | |
| 2. install pandoc and pypandoc, also tqdm | |
| 3. create and download a service_account.json ("Service account key") from https://console.cloud.google.com/apis/credentials | |
| 4. run GOOGLE_APPLICATION_CREDENTIALS=service_account.json python make_audiobook.py book_name.epub | |
| """ | |
| import re | |
| import sys |
Cleaned up version of https://gist.github.com/mrsteyk/74ad3ec2f6f823111ae4c90e168505ac,
which is in turn based on the public_diff_vae.ConvUNetVAE from https://github.com/openai/consistencydecoder.
Install the consistency decoder code (for the inference logic) and download the extracted weights:
So you want to generate images with neural networks. You're in luck! VAEs are here to save the day. They're simple to implement, they generate images in one inference step (unlike those awful slow autoregressive models) and (most importantly) VAEs are 🚀🎉🎂🥳 theoretically grounded 🚀🎉🎂🥳 (unlike those scary GANs - don't look at the GANs)!
The idea of VAE is so simple, even an AI chatbot could explain it:
torch.randn(1024)), interprets that noise as decisions about what to generate, and produces corresponding real-looking images. You want to train this network with nice simple image-space MSE loss against your dataset of real images.
Stable Diffusion's VAE is a neural network that encodes and decodes images into a compressed "latent" format. The encoder performs 48x lossy compression, and the decoder generates new detail to fill in the gaps.
(Calling this model a "VAE" is sort of a misnomer - it's an encoder with some very slight KL regularization, and a conditional GAN decoder)
This document is a big pile of various links with more info.