yoavg

Some remarks on Large Language Models

Yoav Goldberg, January 2023

Audience: I assume you heard of chatGPT, maybe played with it a little, and was imressed by it (or tried very hard not to be). And that you also heard that it is "a large language model". And maybe that it "solved natural language understanding". Here is a short personal perspective of my thoughts of this (and similar) models, and where we stand with respect to language understanding.

Intro

Around 2014-2017, right within the rise of neural-network based methods for NLP, I was giving a semi-academic-semi-popsci lecture, revolving around the story that achieving perfect language modeling is equivalent to being as intelligent as a human. Somewhere around the same time I was also asked in an academic panel "what would you do if you were given infinite compute and no need to worry about labour costs" to which I cockily responded "I would train a really huge language model, just to show that it doesn't solve everything!". We

djspiewak

Quick Tips for Fast Code on the JVM

I was talking to a coworker recently about general techniques that almost always form the core of any effort to write very fast, down-to-the-metal hot path code on the JVM, and they pointed out that there really isn't a particularly good place to go for this information. It occurred to me that, really, I had more or less picked up all of it by word of mouth and experience, and there just aren't any good reference sources on the topic. So… here's my word of mouth.

This is by no means a comprehensive gist. It's also important to understand that the techniques that I outline in here are not 100% absolute either. Performance on the JVM is an incredibly complicated subject, and while there are rules that almost always hold true, the "almost" remains very salient. Also, for many or even most applications, there will be other techniques that I'm not mentioning which will have a greater impact. JMH, Java Flight Recorder, and a good profiler are your very best friend! Mea

BaseCase

Deconstruct 2017 Bibliography

Here are all of the resources mentioned by Deconstruct 2017 speakers, along with who recommended what. Please post a comment if I missed something or have an error!

DC 2017 Speakers' Choice Gold Medalist

• Seeing Like a State by James Scott

Books

• Public Opinion by Walter Lippmann (Evan Czaplicki)
• A Pattern Language by Christopher Alexander (Brian Marick)
• Domain Driven Design by Eric Evans (Brian Marick)

dwbuiten

Problems & Solutions for Interaction Between C and Go

At Vimeo, on the transcoding team, we work a lot with Go, and a lot with C, for various tasks such as media ingest. This means we use CGO quite extensively, and consequently, have run into bits that are perhaps not very well documented, if at all. Below is my effort to document some of the problems we've run into, and how we fixed or worked around them.

Many of these are obviously wrong in retrospect, but hindsight is 20/20, and these problems do exist in many codebases currently.

Some are definitely ugly, and I much welcome better solutions! Tweet me at @daemon404 if you have any, or have your own CGO story/tips, please! I'd love to learn of them.

Table of Contents

jonhoo

Distributed Read-Write Mutex in Go

This project has moved to https://github.com/jonhoo/drwmutex so it can be imported into Go applications.

lavalamp

There are three easy to make mistakes in go. I present them here in the way they are often found in the wild, not in the way that is easiest to understand.

All three of these mistakes have been made in Kubernetes code, getting past code review at least once each that I know of.

1. Loop variables are scoped outside the loop.

What do these lines do? Make predictions and then scroll down.

func print(pi *int) { fmt.Println(*pi) }

chanks

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

pozorvlak

Here's where I understand the state of the art to be:

• In this INRIA tech report, Shapiro, Preguiça, Baquero and Zawirski (SPBZ) prove, amongst other things, that a sufficient condition for CRDTs to achieve eventual consistency on networks which may reorder and duplicate packets (which I'll call flaky networks, henceforth) is that
  1. the underlying datatype forms a semilattice,
  2. messages are full states,
  3. incoming messages are combined with the node's current state using the least-upper-bound operation in the semilattice.
• It's possible to relax condition 2 and still achieve eventual consistency over flaky networks by fragmenting the state into independent parts and transmitting updates to each part separately. For instance, in the G-Set CRDT (an add-only bitset) one can transmit only the index of the element to be added.
• In [these slides from a talk at Dagstuhl](http://www.dagstuhl.de/mat/Files/13/13081/13081.BaqueroCarlos.Sl

adamwiggins

Make it real

Ideas are cheap. Make a prototype, sketch a CLI session, draw a wireframe. Discuss around concrete examples, not hand-waving abstractions. Don't say you did something, provide a URL that proves it.

Ship it

Nothing is real until it's being used by a real user. This doesn't mean you make a prototype in the morning and blog about it in the evening. It means you find one person you believe your product will help and try to get them to use it.

Do it with style

jcs

Update (2019-05-06): The Broadcom wireless card in the MacBook Pro works and can be crammed into the Air.

Update (2015-12-04): This document used to be very lengthy as there were many manual steps required to get OpenBSD and Mac OS X working together through Boot Camp Assistant (BCA), which created a hybrid MBR and enabled a legacy BIOS emulation mode which older versions of Windows (and OpenBSD) required. Newer Macbooks stopped supporting older versions of Windows through BCA and now only support Windows 10 since it uses GPT and UEFI. However, now that newer versions of OpenBSD support GPT and UEFI, Boot Camp Assistant is no longer needed at all to boot OpenBSD.

macOS FileVault encryption and OpenBSD encrypted softraid on a Macbook Air/Pro

OpenBSD works pretty well on at least the Mid-2011 Macbook Air (A1370, SandyBridge) and Mid-2013 Macbook Air (Haswell). The new KMS code in 5.4 brings up the MBA's eDP display in 1366x768 with backlight