Skip to content

Instantly share code, notes, and snippets.

@JacksonCakes
JacksonCakes / longlora_viz.py
Created October 8, 2023 04:25
simple visualization on the shifting pattern of LongLoRA
import numpy as np
import matplotlib.pyplot as plt
group_size = 2048
shift_amount = -group_size // 2
boundary = 1024 # The boundary where the rolling operation causes a transition
# Select a head from the last four heads (head number 7 in this case)
selected_head = 7
# initialize a 2D tensor with random values