Skip to content

Instantly share code, notes, and snippets.

View omarsar's full-sized avatar
🐙

Elvis Saravia omarsar

🐙
3.2k followers · 316 following
View GitHub Profile
@veekaybee
veekaybee / normcore-llm.md
Last active May 13, 2024 20:18
Normcore LLM Reads

Anti-hype LLM reading list

Goals: Add links that are reasonable and good explanations of how stuff works. No hype and no vendor content if possible. Practical first-hand accounts of models in prod eagerly sought.

Foundational Concepts

Screenshot 2023-12-18 at 10 40 27 PM

Pre-Transformer Models

@alonsosilvaallende
alonsosilvaallende / LLM_generating_random_numbers.ipynb
Created July 14, 2023 20:20
LLM_generating_random_numbers.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@JoaoLages
JoaoLages / RLHF.md
Last active March 26, 2024 18:51
Reinforcement Learning from Human Feedback (RLHF) - a simplified explanation

Maybe you've heard about this technique but you haven't completely understood it, especially the PPO part. This explanation might help.

We will focus on text-to-text language models 📝, such as GPT-3, BLOOM, and T5. Models like BERT, which are encoder-only, are not addressed.

Reinforcement Learning from Human Feedback (RLHF) has been successfully applied in ChatGPT, hence its major increase in popularity. 📈

RLHF is especially useful in two scenarios 🌟:

  • You can’t create a good loss function
    • Example: how do you calculate a metric to measure if the model’s output was funny?
  • You want to train with production data, but you can’t easily label your production data
@veekaybee
veekaybee / chatgpt.md
Last active April 12, 2024 20:16
Everything I understand about chatgpt

ChatGPT Resources

Context

ChatGPT appeared like an explosion on all my social media timelines in early December 2022. While I keep up with machine learning as an industry, I wasn't focused so much on this particular corner, and all the screenshots seemed like they came out of nowhere. What was this model? How did the chat prompting work? What was the context of OpenAI doing this work and collecting my prompts for training data?

I decided to do a quick investigation. Here's all the information I've found so far. I'm aggregating and synthesizing it as I go, so it's currently changing pretty frequently.

Model Architecture

@aditya-malte
aditya-malte / smallberta_pretraining.ipynb
Created February 22, 2020 13:41
smallBERTa_Pretraining.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@phillipi
phillipi / biggan_slerp
Last active October 8, 2023 01:25
Slerp through the BigGAN latent space
# to be used in conjunction with the functions defined here:
# https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/biggan_generation_with_tf_hub.ipynb
# party parrot transformation
noise_seed_A = 3 # right facing
noise_seed_B = 31 # left facing
num_interps = 14
truncation = 0.2
category = 14
@rxwei
rxwei / ad-manifesto.md
Last active November 9, 2023 09:58
First-Class Automatic Differentiation in Swift: A Manifesto
@omarsar
omarsar / data_mining_2017_fall_nthu.md
Last active October 14, 2017 02:51
Data Mining Lab Session ( 2017 Fall)

Computing Resources

  • Operating system: Preferably Linux or MacOS. If you have Windows, things may crash unexpectedly (try installing a virtual machine if you need to)
  • RAM: Minimum 8GB
  • Disk space: Mininium 8GB

Software Requirements

Here is a list of the required programs and libraries necessary for this lab session. (Please install them before coming to our lab session on Tuesday; this will save us a lot of time, plus these are the same libraries you may need for your first assignment).

  • Python 3+ (Note: lab and assignment will be done strictly using Python 3)
  • Install latest version of Python 3
@yxtay
yxtay / tensorflow_word2vec_cbow_basic.py
Last active December 21, 2023 03:25
Basic implementation of CBOW word2vec with TensorFlow. Minimal modification to the skipgram word2vec implementation in the TensorFlow tutorials.
# References
# - https://www.tensorflow.org/versions/r0.10/tutorials/word2vec/index.html
# - https://github.com/tensorflow/tensorflow/blob/r0.10/tensorflow/examples/tutorials/word2vec/word2vec_basic.py
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import collections
import math
@puf
puf / index.html
Last active January 15, 2023 19:41
Zero to App: Develop with Firebase (for Web - Google I/O 2016)
<html>
<head>
<script src="https://www.gstatic.com/firebasejs/3.0.0/firebase.js"></script>
<title>ZeroToApp</title>
<style>
#messages { width: 40em; border: 1px solid grey; min-height: 20em; }
#messages img { max-width: 240px; max-height: 160px; display: block; }
#header { position: fixed; top: 0; background-color: white; }
.push { margin-bottom: 2em; }
@keyframes yellow-fade { 0% {background: #f2f2b8;} 100% {background: none;} }