yoavg

A criticism of "On the Dangers of Stochastic Parrots: Can Languae Models be Too Big"

Yoav Goldberg, Jan 23, 2021.

The FAccT paper "On the Dangers of Stochastic Parrots: Can Languae Models be Too Big" by Bender, Gebru, McMillan-Major and Shmitchell has been the center of a controversary recently. The final version is now out, and, owing a lot to this controversary, would undoubtly become very widely read. I read an earlier draft of the paper, and I think that the new and updated final version is much improved in many ways: kudos for the authors for this upgrade. I also agree with and endorse most of the content. This is important stuff, you should read it.

However, I do find some aspects of the paper (and the resulting discourse around it and around technology) to be problematic. These weren't clear to me when initially reading the first draft several months ago, but they became very clear to me now. These points are for the most part

junyann

from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch
from torch.nn import CrossEntropyLoss
from tqdm import trange


max_length = 24
batch_size = 200

yuchenlin

import torch
from transformers import OpenAIGPTTokenizer, OpenAIGPTLMHeadModel
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import numpy as np
from scipy.special import softmax
 
def model_init(model_string, cuda):
    if model_string.startswith("gpt2"):
        tokenizer = GPT2Tokenizer.from_pretrained(model_string)
        model = GPT2LMHeadModel.from_pretrained(model_string)

yuchenlin

import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM
import logging
logging.basicConfig(level=logging.INFO)# OPTIONAL



tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()

chuanconggao

from collections import defaultdict

def frequent_rec(patt, mdb):
    results.append((len(mdb), patt))

    occurs = defaultdict(list)
    for (i, startpos) in mdb:
        seq = db[i]
        for j in range(startpos + 1, len(seq)):
            l = occurs[seq[j]]

peterjc123

# Prerequisites
# 1. MSVC 2017 C++ Build Tools
# 2. CMAKE 3.0 or up
# 3. 64 bits of Windows
# 4. Anaconda / MiniConda 64 bits

# Prerequisites for CUDA
# 1. CUDA 8.0 or up
# 2. NVTX( in CUDA as Visual Studio Integration. if fail to install, you can extract
#     the CUDA installer exe and found the NVTX installer under the CUDAVisualStudioIntegration)

abhishekcs10

#!/bin/bash

# this script installs GCC 5.4.0 
# to use it navigate to your home directory and type:
# sh install-gcc-5.4.0.sh

# download and install gcc 4.9.3
wget https://github.com/gcc-mirror/gcc/archive/gcc-5_4_0-release.tar.gz
tar xzf gcc-5_4_0-release.tar.gz
cd gcc-5_4_0-release

Tushar-N

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

seqs = ['gigantic_string','tiny_str','medium_str']

# make <pad> idx 0
vocab = ['<pad>'] + sorted(set(''.join(seqs)))

# make model

WeiTang114

#!/bin/bash
# Show username after each process in nvidia-smi
# like:
# ...
# +------------------------------------------------------+
# | Processes:                                GPU Memory |
# |  GPU       PID  Type  Process name        Usage      |
# |======================================================|
# |    0    150752    C   python                  830MiB | User: user1
# |    1      2185    C   /usr/bin/python        1090MiB | User: user2

thousandlemons

This setup guide has been moved to my blog:

https://nathaniel.blog/tutorials/shadowsocks/