madebyollin

mysterious software bugs I encounter, like, daily

• if I add/destroy enough video tags (i.e. in periodic output display from a long-running jupyter notebook) Safari will eventually stop rendering all video tags across all pages until I reboot the browser
• if I interleave autocast and no_grad scopes in the wrong way (I think - autocast with no_grad inside followed by not-no-grad stuff?) pytorch will silently disable grad for the entire autocast scope
• if I edit colab notebooks (specific ones? on a specific connection? idk), I intermittently get "this notebook has been modified" conflict-resolution dialogs despite being the only editor of the notebook
• if I don't manually disable the collaborative extension on jupyterlab, it will intermittently teleport my cursor focus back to the first cell in the notebook

need to eventually find repro instructions and chase all of these down to fix

madebyollin

from IPython.display import HTML

def get_pred_original_sample(sched, model_output, timestep, sample):
    return sample - sched.sigmas[(sched.timesteps == timestep).nonzero().item()] * model_output

# TODO: fix awful globals
prev_img_str = None

def pil_to_html(pil_img, h=IM_HEIGHT, w=IM_WIDTH):
    global prev_img_str

madebyollin

Objective

Informal (vibes-based) evaluation of the following vision-language-model captioners:

• Florence-2-base-ft
• CogVLM2
• BLIP-2
• MoonDream2
• Share-Captioner
• Florence-2-SD3-Captioner

madebyollin

#!/usr/bin/env python3
import gradio as gr
import numpy as np
import random
import torch
from diffusers import (
    StableDiffusion3Pipeline,
    SD3Transformer2DModel,
    FlowMatchEulerDiscreteScheduler,
    AutoencoderTiny,

madebyollin

madebyollin

def add_profiling_markers(model):
    """Monkey-patch profiling markers into an nn.Module.
    
    Args:
        model: an nn.Module
    
    Effect:
        all model.named_module() forward calls get wrapped in their
        own profiling scope, making traces easier to understand.
    """

madebyollin

madebyollin

Notes / Links about Stable Diffusion VAE

Stable Diffusion's VAE is a neural network that encodes images into a compressed "latent" format and decodes them back. The encoder performs 48x lossy compression, and the decoder generates new detail to fill in the gaps.

(Calling this model a "VAE" is sort of a misnomer - it's an encoder with some very slight KL regularization, and a conditional GAN decoder)

This document is a big pile of various links with more info.

madebyollin

Consistency Decoder PyTorch Model Code

Cleaned up version of https://gist.github.com/mrsteyk/74ad3ec2f6f823111ae4c90e168505ac, which is in turn based on the public_diff_vae.ConvUNetVAE from https://github.com/openai/consistencydecoder.

Example Usage

Install the consistency decoder code (for the inference logic) and download the extracted weights:

madebyollin