Skip to content

Instantly share code, notes, and snippets.

View renaud's full-sized avatar

Renaud Richardet renaud

  • Eaternity
  • Lausanne, Switzerland
View GitHub Profile

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

@pncnmnp
pncnmnp / news_corpus.txt
Last active February 5, 2024 09:59
22 years (1995 - 2017) worth of news scrapped from Wikipedia
This file has been truncated, but you can view the full file.
Current events of September 3, 1995 (1995-09-03) (Sunday) :
eBay is founded.
Current events of September 6, 1995 (1995-09-06) (Wednesday) :
NATO air strikes against Bosnian Serb forces continue, after repeated attempts at a solution to the Bosnian War fail.
Current events of September 19, 1995 (1995-09-19) (Tuesday) :
The Washington Post and The New York Times publish the Unabomber's manifesto.
Current events of September 22, 1995 (1995-09-22) (Friday) :
American millionaire Steve Forbes announces his candidacy for the 1996 Republican presidential nomination.
Current events of September 23, 1995 (1995-09-23) (Saturday) :
Argentine national Guillermo "Bill" Gaede is arrested in Phoenix, Arizona, on charges of industrial espionage. His sales to Cuba, China, North Korea and Iran are believed to have involved Intel and AMD trade secrets worth US$10–20 million.
@moskomule
moskomule / sparql_access.py
Created October 30, 2016 09:37
fetch more than 10000 contents from DBpedia etc. by sparql
from SPARQLWrapper import SPARQLWrapper, JSON
sparql = SPARQLWrapper("http://localhost:8890/sparql")
for i in range(25):
query = """
select ?slabel ?olabel
where {
?s rdfs:subClassOf ?o.
?s rdf:type owl:Class.
?o rdf:type owl:Class.
from gensim import models
sentence = models.doc2vec.LabeledSentence(
words=[u'so`bme', u'words', u'here'], tags=["SENT_0"])
sentence1 = models.doc2vec.LabeledSentence(
words=[u'here', u'we', u'go'], tags=["SENT_1"])
sentences = [sentence, sentence1]
class LabeledLineSentence(object):
@syllog1sm
syllog1sm / gist:10343947
Last active November 7, 2023 13:09
A simple Python dependency parser
"""A simple implementation of a greedy transition-based parser. Released under BSD license."""
from os import path
import os
import sys
from collections import defaultdict
import random
import time
import pickle
SHIFT = 0; RIGHT = 1; LEFT = 2;
@tylerneylon
tylerneylon / learn.lua
Last active July 25, 2024 19:50
Learn Lua quickly with this short yet comprehensive and friendly script. It's written as both an introduction and a quick reference. It's also a valid Lua script so you can verify that the code does what it says, and learn more by modifying and running this script in your Lua interpreter.
-- Two dashes start a one-line comment.
--[[
Adding two ['s and ]'s makes it a
multi-line comment.
--]]
----------------------------------------------------
-- 1. Variables and flow control.
----------------------------------------------------
@waleking
waleking / SparkGibbsLDA.scala
Last active January 31, 2020 11:15
We implement gibbs sampling for LDA by Spark. This version performs much better than alpha version, and now can handle 3196204 words, 100 topics, 1000 sample iterations on server in 161.7 minutes. To solve the long time consuming in collect() process in alpha version, we utilize the cache() method as line 261 and line 262. We also solve a pile o…
package topic
import spark.broadcast._
import spark.SparkContext
import spark.SparkContext._
import spark.RDD
import spark.storage.StorageLevel
import scala.util.Random
import scala.math.{ sqrt, log, pow, abs, exp, min, max }
import scala.collection.mutable.HashMap
@nova77
nova77 / clip_magic.py
Last active October 24, 2023 18:50
copy to clipboard ipython magic
"""
Add copy to clipboard from IPython!
To install, just copy it to your profile/startup directory, typically:
~/.ipython/profile_default/startup/
Example usage:
%clip hello world
# will store "hello world"
@fhars
fhars / PA3Test.m
Created October 7, 2012 14:18
A test suite for PA3 of the Coursera PGM couse
% A simple test suite for PA 3
%
% copy the comparedata.m file from last week's test suite or from
% http://www.mathworks.com/matlabcentral/fileexchange/1459-comparedata
% into the directory for this weeks assignment and save this file
% as PA3Test.m
%
% A test can have three different results:
% - If the test suite says "OK", your code produced the exactly the same
% output as the sample data.