yoavg

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

thomwolf

def top_k_top_p_filtering(logits, top_k=0, top_p=0.0, filter_value=-float('Inf')):
    """ Filter a distribution of logits using top-k and/or nucleus (top-p) filtering
        Args:
            logits: logits distribution shape (vocabulary size)
            top_k >0: keep only top k tokens with highest probability (top-k filtering).
            top_p >0.0: keep the top tokens with cumulative probability >= top_p (nucleus filtering).
                Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751)
    """
    assert logits.dim() == 1  # batch size 1 for now - could be updated for more but the code would be less clear
    top_k = min(top_k, logits.size(-1))  # Safety check

MatthieuBizien

# coding: utf-8

import logging
import re
from collections import Counter

import numpy as np
import torch
from sklearn.datasets import fetch_20newsgroups
from torch.autograd import Variable

apaszke

from graphviz import Digraph
import torch
from torch.autograd import Variable, Function

def iter_graph(root, callback):
    queue = [root]
    seen = set()
    while queue:
        fn = queue.pop()
        if fn in seen:

saurabhshri

Install and use pip in a local directory without root/sudo access.

By: @saurabhshri

Why?

Many users when are given server access, do not have root (or sudo) privileges and can not simply do sudo apt-get install python-pip . Here's an easy way you can install and use pip without root (or sudo) access in a local directory. Note : This works without easy_install too.

How?

miohtama

import email

def get_decoded_email_body(message_body):
    """ Decode email body.

    Detect character set if the header is not set.

    We try to get text/plain, but if there is not one then fallback to text/html.

    :param message_body: Raw 7-bit message body input e.g. from imaplib. Double encoded in quoted-printable and latin-1

MohamedAlaa

tmux shortcuts & cheatsheet

start new:

tmux

start new with session name:

tmux new -s myname

artero

Launch Sublime Text 2 from the Mac OS X Terminal

Sublime Text 2 ships with a CLI called subl (why not "sublime", go figure). This utility is hidden in the following folder (assuming you installed Sublime in /Applications like normal folk. If this following line opens Sublime Text for you, then bingo, you're ready.

open /Applications/Sublime\ Text\ 2.app/Contents/SharedSupport/bin/subl

You can find more (official) details about subl here: http://www.sublimetext.com/docs/2/osx_command_line.html

Installation