cedrickchee

Notes: State of GPT 2023

On May 2023, Andrej Karpathy delivered the "State of GPT" presentation for the Microsoft Build event.

Intro

• First part: how to train (Chat)GPT Assistant - an emerging recipe
• Second part: Use these assistant effectively for your applications

cedrickchee

Generative AI Engineering

Building applications with foundation models.

Things to consider when using proprietary models and open models.

	Proprietary Models	Open Models
Data	Have to send your data to model providers, which means your team can accidentally leak confidential information	Less checks and balances for data lineage/training data co

cedrickchee

AI Agent UX

AI + UIs

^{"Climbing the Ladder of Abstraction" talk by Amelia Wattenberger of Adept.ai}

Often when people talk about building interfaces with AI, they refer to these two distinct categories:

cedrickchee

Tradeoffs Between Long-Context LLM and RAG

From Claude 100K to Gemini 10M, we are in the era of long-context large language models (LLMs).

Retrieval-Augmented Generation (RAG) is a technique for enhancing the accuracy and reliability of generative AI models with facts fetched from external sources. [^1] [^2]

The hype surrounding RAG is largely driven by its potential to address some of the limitations of language models by enabling the development of more accurate, contextually grounded, and creative language generation systems. However, it's essential to note that RAG is still a relatively new and evolving field, and there are several challenges and limitations that need to be addressed before it can be widely adopted. For example, RAG requires large amounts of high-quality training data, and it can be challenging to integrate the retrieval and generation components in a way that produces coherent and natural-sounding language.

LLMs such as Claude 100K and Gemini 10M support long contexts spanning tens

cedrickchee

Evaluating Llama 3 on Code Tasks

To test Meta Llama 3's performance against existing models, we used the coding benchmarks: HumanEval. HumanEval tests the model's ability to complete code based on docstrings.

The benchmark tests 137 publicly available large language models (LLMs) on code tasks.

Model	Accuracy[^1]

cedrickchee

The Intuition behind Rotary Positional Embedding (RoPE)

RoPE is a position embedding method proposed by Jianlin Su et al. in 2021 (paper).

I aim to make the subject matter accessible to a broader audience. You won't find any math-heavy equations or theoretical proofs here.

---

Let me try to explain RoPE in a way that a non-technical person can understand.

cedrickchee

Meta releases Llama 3 Large Language Models (LLMs) 🦙

Llama 3 8B and 70B pretrained and instruction-tuned models available today. Based on benchmarks, 8B and 70B model is not quite GPT-4 class, but 400B+ (still in development) will reach GPT-4 level soon. Llama 3 sets a new standard for state-of-the art performance and efficiency for openly available LLMs.

Key highlights:

• 8k context length
• New capabilities: enhanced reasoning and coding
• Big change: new tokenizer that expands the vocab size to 128K (from 32K tokens in v2) for better multilingual performance
• Trained with 7x+ more data on 15 trillion tokens on two clusters with 24K GPUs

cedrickchee

Rebutting Devin: "First AI Software Engineer" Claim is Not True

A human software engineer, Carl (aka. "InternetOfBugs") looked closer and exposed Cognition Labs's Devin "First AI Software Engineer" Upwork lie. InternetOfBugs is an AI enthusiast and uses coding AI himself. InternetOfBugs is not anti-AI, but anti-hype.

The company lied and said that their video showed Devin completing and getting paid for freelance jobs on Upwork, but it didn't show that at all. On the whole that's not surprising given the current state of Generative AI, and I wouldn't be bothering to debunk it, except:

1. The company lied about what Devin could do in the video description, and
2. a lot of people uncritically parroted the lie all over the Internet, and

cedrickchee

AI Agents

In response to Dr. Andrew Ng's letter, "Four AI agent strategies that improve GPT-4 and GPT-3.5 performance".

When I read Andrew's letter, I'm imagining him as Steve Balmer, shouting "Agentic, agentic, agentic workflows!". Haha, we can hear you. No need for that.

Before we move on, let's be clear what is agent in this context. The context is, we're now in 2024 and LLMs such as GPT-4 and Llama 3 is the state-of-the-art. In early 2022, everybody in the field knew about the agent from RL, but the general public had no conception of what it was. Their narrative were still everything is a chatbot. All sorts of different things are being called agents. Chatbots are being called agents. Things that make a function call are being called agents. Now when people think agent, they actually think the right thing.

An agent is something that you can give a goal and get an end step workflow done correctly in the minimum number

cedrickchee

❤️ 3blue1brown

I love 3blue1brown videos. I have been watching for a few years now.

I've found that the "visual" intuition about a concept/topic is really great. They are simple, structured, and clear way to represent these concepts in video.

3blue1brown videos really transformed my view on math.

Recently, 3blue1brown drop two new videos in the Deep Learning series: