Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
twiml-lightning-share

Measuring Gender Bias in Spanish Language Models

Presenter

Nick Doiron, Tufts University / Independent Research

GitHub: https://github.com/mapmeld ; LinkedIn: https://www.linkedin.com/in/nickdoiron/

Context

Neural networks in natural language processing (NLP) are frequently based on large pre-trained models of word embeddings. It's understood that proximity of words' vectors can create a gender bias in the final model (e.g. assuming male doctors). Researchers have developed tools to measure bias in English pretrained models, but how can they be adapted for Spanish, a language with grammatical gender?

Description

I translated the word lists for the Word Embedding Association Test (WEAT) and created parallel columns for masculine and feminine words where appropriate (gendered professions, family relationships, and adjectives). This made it easier to determine bias of mBERT and a monolingual Spanish model (BETO).

Later I used spaCy and the BETO word vectors to "flip" gender of long sentences. This allows us to train a model on less biased examples, or to test if a model would have a different outcome with a different gender.

longer form: https://medium.com/ai-in-plain-english/gender-bias-in-spanish-bert-1f4d76780617

Code

WEAT-ES https://colab.research.google.com/drive/1Yicr3qSkh0reKBEYwohygM3RAQdEirxf

Counterfactuals https://colab.research.google.com/drive/1zVOIyWjQqLeittQIy3HHdKhjgMqGpiSU

Results, Outcomes, Conclusions

  • Compared results on BETO (monolingual Spanish model) and mBERT (100 language model)
  • Generated counterfactual sentences from Wikipedia articles
  • Data augmentation on a Spanish movie reviews dataset greatly improved accuracy

Python libraries used

NumPy, HuggingFace/Transformers, spaCy, WEAT

Open questions/issues

  • Could a seq2seq model replace the programmatic, spaCy-based counterfactual generator?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.