yoavg

Reinforcement Learning for Language Models

Yoav Goldberg, April 2023.

Why RL?

With the release of the ChatGPT model and followup large language models (LLMs), there was a lot of discussion of the importance of "RLHF training", that is, "reinforcement learning from human feedback". I was puzzled for a while as to why RL (Reinforcement Learning) is better than learning from demonstrations (a.k.a supervised learning) for training language models. Shouldn't learning from demonstrations (or, in language model terminology "instruction fine tuning", learning to immitate human written answers) be sufficient? I came up with a theoretical argument that was somewhat convincing. But I came to realize there is an additional argumment which not only supports the case of RL training, but also requires it, in particular for models like ChatGPT. This additional argument is spelled out in (the first half of) a talk by John Schulman from OpenAI. This post pretty much

mshafrir

{
    "AL": "Alabama",
    "AK": "Alaska",
    "AS": "American Samoa",
    "AZ": "Arizona",
    "AR": "Arkansas",
    "CA": "California",
    "CO": "Colorado",
    "CT": "Connecticut",
    "DE": "Delaware",

bsweger

Useful Pandas Snippets

A personal diary of DataFrame munging over the years.

Data Types and Conversion

Convert Series datatype to numeric (will error if column has non-numeric values)
(h/t @makmanalp)

robulouski

#!/usr/bin/env python
#
# Very basic example of using Python and IMAP to iterate over emails in a
# gmail folder/label.  This code is released into the public domain.
#
# RKI July 2013
# http://www.voidynullness.net/blog/2013/07/25/gmail-email-with-python-via-imap/
#
import sys
import imaplib

0XDE57

ABOUT

about:config settings to harden the Firefox browser. Privacy and performance enhancements.
To change these settings type 'about:config' in the url bar. Then search the setting you would like to change and modify the value. Some settings may break certain websites from functioning and rendering normally. Some settings may also make firefox unstable. I am not liable for any damages/loss of data.

Not all these changes are necessary and will be dependent upon your usage and hardware. Do some research on settings if you don't understand what they do. These settings are best combined with your standard privacy extensions (HTTPS Everywhere No longer required: Enable HTTPS-Only Mode, NoScript/Request Policy, uBlock origin, agent spoofing, Privacy Badger etc), and all plugins set to "Ask To Activate".

malarkey

When times get tough and people get nasty, you’ll need more than a killer smile. You’ll need a killer contract.

Used by 1000s of designers and developers Clarify what’s expected on both sides Helps build great relationships between you and your clients Plain and simple, no legal jargon Customisable to suit your business Used on countless web projects since 2008

…………………………

dideler

#!/usr/bin/python

"""
Playing around with slightly various ways to simulate uniq in Python.
The different strategies are timed.
Only m1() and m2() do not change the order of the data.
`in` is the input file, `out*` are output files.
"""

infile = 'in'  # Change filename to suit your needs.

kapkaev

$ redis-cli
> config set stop-writes-on-bgsave-error no

tijptjik

BretFisher

sudo: required #is required to use docker service in travis

language: php #can be any language, just php for example

services:
  - docker # required, but travis uses older version of docker :(

install:
  - echo "install nothing!" # put your normal pre-testing installs here