Skip to content

Instantly share code, notes, and snippets.

View uzl's full-sized avatar

Uzzal Podder uzl

View GitHub Profile
@uzl
uzl / attn_calc_methods.py
Last active April 8, 2024 09:59
Attention Calculation Methods
import torch.nn as nn
import torch.nn.functional as F
from torch.backends.cuda import sdp_kernel, SDPBackend
import time
import torch
import torch.utils.benchmark as benchmark
# check dependency
assert torch.cuda.is_available(), 'CUDA is expected.'