Skip to content

Instantly share code, notes, and snippets.

@miladfa7
miladfa7 / gpt2_from_scratch.py
Last active April 30, 2023 06:54
Train GPT-2 from Scratch on your own language(Persain) | GPT-2 Training on non-english text
from simpletransformers.language_modeling import LanguageModelingModel
import logging
logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)
train_args = {
@leopd
leopd / almost-attention.py
Last active April 21, 2023 22:35
Explanatory (non-vectorized) code for how attention works
# This code doesn't work, and isn't intended to.
# The goal of this code is to explain how attention mechansisms work, in code.
# It is deliberately not vectorized to make it clearer.
def attention(self, X_in:List[Tensor]):
# For every token transform previous layer's out
for i in range(self.sequence_length):
query[i] = self.Q * X_in[i]
key[i] = self.K * X_in[i]
value[i] = self.V * X_in[i]
#!/usr/bin/env python3
'''
always getting the most recent frame of a camera
================================================
Usage:
------
freshest_camera_frame.py