Skip to content

Instantly share code, notes, and snippets.

@iamaziz

iamaziz/LLM-from-scratch.ipynb

Created July 23, 2023 17:02
Show Gist options
  • Star 9 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save iamaziz/171170dce60d9cd07fab221507fd1d52 to your computer and use it in GitHub Desktop.
Save iamaziz/171170dce60d9cd07fab221507fd1d52 to your computer and use it in GitHub Desktop.
Download ZIP
Building a large language model (LLM) from scratch (for learning and fun - inspired by Llama2).
Display the source blob
Display the rendered blob
Raw
LLM-from-scratch.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@AliHaider20
Copy link

AliHaider20 commented Nov 11, 2023 •
edited

@iamaziz

I ran the exact code from above.
text = big_text.lower(). Big_text is a list that's why it returns an error.

Why are you creating abstract is there any future use of it?
Also, can you explain what does sequence_length mean here? How did you decide that number?

Looking forward to hearing from you.

@zanstro
Copy link

zanstro commented Feb 12, 2024

Can this guide be used to build a 30b LLM?

@AliHaider20
Copy link

I think yes you can use this. 30B is just a model size so you can always increase the model size, but the difficult part is to train it.
I would highly recommend https://youtu.be/kCc8FmEb1nY where you'll be able to understand in and out of LLMs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment