Skip to content

Instantly share code, notes, and snippets.

@flavienbwk

flavienbwk/README.md

Last active October 27, 2023 00:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save flavienbwk/a0d028f9a6ea5553d3e819521204c046 to your computer and use it in GitHub Desktop.
Save flavienbwk/a0d028f9a6ea5553d3e819521204c046 to your computer and use it in GitHub Desktop.
Download ZIP
LLMs translation benchmark comparison (Tatoeba fra to eng with 1000 sentences).
Raw
README.md

Values in bold represent the best value.

Metrics Google Translate Azure Translate Deepl Translate OpenAI GPT3.5-t OpenAI GPT4
Duration (s) 79.12 113.00 124.43 1820.75 1278.52
BLEU score 0.65 0.69 0.68 0.77 0.72
SacreBLEU score 53.36 55.21 53.08 58.59 57.27
Price* ($) 0.96 0.48 0.96 0.04 2.17
BLEU/price 0.68 1.44 0.71 19.25 0.332
Duration/price 82.42 235.42 182.98 2364.61 588.94

* Google Translate, Azure Translate and Deepl prices are considered outside of their free tiers (10$-20$/1M characters).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment