yoavg

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

tintoy

#!/usr/bin/env python3

import os
import paramiko

ssh_key_filename = os.getenv('HOME') + '/.ssh/id_rsa'

jumpbox_public_addr = '168.128.52.199'
jumpbox_private_addr = '10.0.5.10'
target_addr = '10.0.5.20'

sixy6e

A simple MPI logging and compute example

The simple logging example can be run via:

• mpiexec -n 4 python mpi_logger.py

For the compute and logging example:

• mpiexec -n 4 python mock_execution.py

xylcbd

#coding: utf-8
#demo of beam search for seq2seq model
import numpy as np
import random

vocab = {
    0: 'a',
    1: 'b',
    2: 'c',
    3: 'd',

waichee

# Clone the Tensorflow Serving source
git clone https://github.com/tensorflow/serving
cd serving && git checkout <commit_hash>

# Build the docker image (time to go get yourself a coffee, maybe a meal as well, this will take a while.)
docker build -t some_user_namespace/tensorflow-serving:latest -f ./serving/tensorflow_serving/tools/docker/Dockerfile.devel .

# Run up the Docker container in terminal
docker run -ti some_user_namespace/tensorflow-serving:latest

waichee

mkdir -p /work/

# Clone the source from Github
cd /work/ && git clone — recurse-submodules https://github.com/tensorflow/serving

# Pin the version of Tensorflow Serving and its submodule
TENSOR_SERVING_COMMIT_HASH=85db9d3
TENSORFLOW_COMMIT_HASH=dbe5e17
cd /work/serving && git checkout $TENSOR_SERVING_COMMIT_HASH

0xnurl

Installing OpenFst Native Python Extension on MacOS

Starting from version 1.5, OpenFst has offered a native Python module, making the use of external wrappers like PyFst unnecessary. This has been greatly helpful since PyFst doesn't support Python 3.

1. Install OpenFst

• Download OpenFst 1.5.4 or above from http://openfst.cs.nyu.edu/twiki/bin/view/FST/FstDownload
• Unzip, untar: $ tar xzvf openfst-1.5.4.tar.gz
• $ ./configure --enable-python --enable-far

krivonogov

################
#   CPROFILE   #
#############################################################################

# 1 - Profile myfunc() from ipython
import cProfile
filename = 'filename.prof'
cProfile.run('myfunc()', filename)

# 2 - Convert your file to a usable kcachegrind file in your shell

wojteklu

Code is clean if it can be understood easily – by everyone on the team. Clean code can be read and enhanced by a developer other than its original author. With understandability comes readability, changeability, extensibility and maintainability.

---

General rules

1. Follow standard conventions.
2. Keep it simple stupid. Simpler is always better. Reduce complexity as much as possible.
3. Boy scout rule. Leave the campground cleaner than you found it.
4. Always find root cause. Always look for the root cause of a problem.

Design rules

igormq

import tensorflow as tf

def beam_decoder(decoder_inputs, initial_state, cell, loop_function, scope=None,
                 beam_size=7, done_token=0
                ):
    """
    Beam search decoder
    
    Args:
      decoder_inputs: A list of 2D Tensors [batch_size x input_size].