yoavg

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

petewarden

git clone https://github.com/tensorflow/tensorflow tensorflow
cd tensorflow
docker build -t "tf-pi" -f tensorflow/tools/ci_build/Dockerfile.cpu tensorflow/tools/ci_build/
docker run -it tf-pi
# Commands after this are run on the Docker instance.
dpkg --add-architecture armhf
echo 'deb [arch=armhf] http://ports.ubuntu.com/ trusty main restricted universe multiverse' >> /etc/apt/sources.list.d/armhf.list
echo 'deb [arch=armhf] http://ports.ubuntu.com/ trusty-updates main restricted universe multiverse' >> /etc/apt/sources.list.d/armhf.list
echo 'deb [arch=armhf] http://ports.ubuntu.com/ trusty-security main restricted universe multiverse' >> /etc/apt/sources.list.d/armhf.list
echo 'deb [arch=armhf] http://ports.ubuntu.com/ trusty-backports main restricted universe multiverse' >> /etc/apt/sources.list.d/armhf.list

jodersky

object Main extends App {

  type Succ[A] <: Shape

  type _0 <: Shape
  type _1 = Succ[_0]
  type _2 = Succ[_1]
  type _3 = Succ[_2]

  type Shape

csswizardry

Vim without NERD tree or CtrlP

I used to use NERD tree for quite a while, then switched to CtrlP for something a little more lightweight. My setup now includes zero file browser or tree view, and instead uses native Vim fuzzy search and auto-directory switching.

Fuzzy Search

There is a super sweet feature in Vim whereby you can fuzzy find your files using **/*, e.g.:

:vs **/*<partial file name><Tab>

anziem

#msgs_scroller_div::-webkit-scrollbar-track, #client_body::before, .client_container, 
#search_terms, #client_body, #footer, ts-message, .channel_header, ts-jumper ts-jumper-container,
ts-jumper input[type="text"] {
  background: #002B36 !important;
}

#client_body::before {
  border-bottom: 1px solid #268BD2 !important;
}

bstriner

from keras.optimizers import Adam
from keras import backend as K
from keras.datasets import mnist
from keras.utils.np_utils import to_categorical
from keras.metrics import categorical_accuracy
from keras.initializations import glorot_uniform, zero
import numpy as np

# inputs and targets are placeholders
input_dim = 28*28

BMeu

Installing Python 3.5 on Raspbian

As of October 2016, Raspbian does not yet include the latest Python release, Python 3.5. This means we will have to build it ourselves, and here is how to do it.

1. Install the required build-tools (some might already be installed on your system).

deroneriksson

Apache Committer Notes

Deploying to Apache Snapshot Repository

To deploy project artifacts to the Apache Snapshot Repository takes a little set-up. Here are the steps that I followed.

Maven Password Encryption

Follow the instructions here: https://maven.apache.org/guides/mini/guide-encryption.html

DrewML

Theming Slack for OSX

So, you love Slack, but you hate applications with large white backgrounds? Why not use Dark Mode!

Unfortunately, Slack does not have a Dark Mode, although it's on their list of possibilities.

But, don't fret - there is a solution! Because the slack native desktop apps are just wrappers around a web app, we can inject our own CSS to customize the application to our liking.

How to (OSX Only)

kukuruza

from math import sqrt

def put_kernels_on_grid (kernel, pad = 1):

  '''Visualize conv. filters as an image (mostly for the 1st layer).
  Arranges filters into a grid, with some paddings between adjacent filters.

  Args:
    kernel:            tensor of shape [Y, X, NumChannels, NumKernels]
    pad:               number of black pixels around each filter (between them)