mapmeld

# All I'm looking for on an ML example:
# ! pip install name_of_library

from name_of_library import model, other_stuff

tdata = load_data_from_file() # not a built-in datasets source where I'd need to write python to add data
tdata.apply(changes) # whose dataset is so perfect we don't edit it

model.train(tdata, **explained_params)

mapmeld

Patching Models with New Words, People, and Events

May 6 - June 15, 2021

Scope

Once a large pre-trained language model is published, it is a snapshot of language when its corpus was collected. What are ways to update models to support new or newly-frequent terms (BIPOC), phrasing (social distancing), or people and events (Fyre Festival)? What are reliable, low-cost ways to test and benchmark these methods of updating?

Current status

mapmeld

/*
  Generally, don't run random JS in your browser console, especially on financial sites, but here we are
  By default this sorts by Percent Change. If you uncomment the next line it sorts by myDelta (price x your shares)
  Caveats:
  - I'm not affiliated with Vanguard or any licensed financial advisor or tax preparer. I don't have a clue what's going on with your finances.
  - The script assumes you did NOT trade today; it uses today's change and current shares
  - Delta-sort does not handle penny stocks as well because the UI says 0.01 and we reverse-engineer from current balance
*/

let sortRule = 'pct';

mapmeld

t5.data.TaskRegistry.add(
      "byt5_ex",
      t5.data.TextLineTask,
      split_to_filepattern={
            "train": "gs://BUCKET/train_lines.txt",
            "validation": "gs://BUCKET/validation_lines.txt",
        },
      text_preprocessor=[
        functools.partial(
          t5.data.preprocessors.parse_tsv,

mapmeld

Code: https://colab.research.google.com/drive/1vltPI81atzRvlALv4eCvEB0KdFoEaCOb?usp=sharing

Can these scores be improved? YES!

Rerunning with more training data, more epochs of training, or using other libraries to set a learning rate / other hyperparameters before training.

• Experimenting with epochs - when I doubled the number of epochs, MuRIL improves only slightly (69.5->69.7 on one task)

The point of a benchmark is to run these models through a reasonable and identical process; you can tweak hyperparameters on any model to improve results.

mapmeld

Measuring Gender Bias in Spanish Language Models

Presenter

Nick Doiron, Tufts University / Independent Research

GitHub: https://github.com/mapmeld ; LinkedIn: https://www.linkedin.com/in/nickdoiron/

Context

mapmeld

from simpletransformers.classification import ClassificationModel

# set use_cuda=False on CPU-only platforms
model = ClassificationModel('bert', 'monsoon-nlp/dv-wave', num_labels=8, use_cuda=True, args={
    'reprocess_input_data': True,
    'use_cached_eval_features': False,
    'overwrite_output_dir': True,
    'num_train_epochs': 3,
    'silent': True
})

mapmeld

# pip install gdal
import json
from osgeo import ogr

# depends on your shapefile
target_shapefile = 'tl_2010_sample_shapefile.shp'
fips_id = 'GEOID10'

saveblocks = json.loads(open('savefile.json', 'r').read())

mapmeld

# pip install requests
import time, json
import requests

api_key = "API_KEY_STRING"

# look up FIPS for state and county:
# https://www.nrcs.usda.gov/wps/portal/nrcs/detail/national/home/?cid=nrcs143_013697
state = '12'
county_fips = ['086']

mapmeld

Related Research

Towards Debiasing Sentence Representations (Liang et al., 2020) English BERT and ELMO

• https://cs.cmu.edu/~pliang/papers/acl2020_debiasing.pdf
• https://github.com/pliang279/sent_debias

Measuring Bias in Contextualized Word Representations (Kurita et al., 2019) analyzes English BERT with word-association https://aclweb.org/anthology/W19-3823/