Skip to content

Instantly share code, notes, and snippets.

@Aschen
Last active July 30, 2023 12:48
Show Gist options
  • Save Aschen/cd73b13ae09e5dad5f50e7485bad4d1a to your computer and use it in GitHub Desktop.
Save Aschen/cd73b13ae09e5dad5f50e7485bad4d1a to your computer and use it in GitHub Desktop.
Performance differences for classification (BART) between Transformers.js and Transformers (Python)

Performance differences for classification (BART) between Transformers.js and Transformers (Python)

This is a GIST for this issue in Transformer.js repository

Trying to classify the following texts as street address:

  • '1 rue de la paix',
  • 'Karlstraße 12, 80333 Munich, DE',
  • 'Startups da América Latina demonstraram capacidade de gerar valor econômico diz fundo alemão',
  • "But what if one day, you come up with a great idea to combine those, for instance to use some Python libraries in your application, but you just do not have any idea how to integrate it with your Node.js application. Of course you can always build API on top of Python backend(Flack, etc), but in that case you need to build, host and manage one more application, when you just need to run a single Python script. That's why I want to give you step by step instructions on how to achieve this.\n",
$ python bart.py
facebook/bart-large-mnli
0.9975677132606506
0.9981901049613953
0.004343739710748196
0.022633692249655724

$ node bart.js
Xenova/bart-large-mnli
0.4556246405304541
0.5631336436601322
0.6358004417358224
0.6758611902199643

Run yourself

Javascript

  • Install: npm i @xenova/transformers
  • Run: node bart.js

Python

  • Install: pip install transformers
  • Run: python bart.py
import { pipeline, env } from '@xenova/transformers';
const modelName = 'Xenova/bart-large-mnli';
const labels = ['a street address'];
const texts = [
'1 rue de la paix',
'Karlstraße 12, 80333 Munich, DE',
'Startups da América Latina demonstraram capacidade de gerar valor econômico diz fundo alemão\n',
"But what if one day, you come up with a great idea to combine those, for instance to use some Python libraries in your application, but you just do not have any idea how to integrate it with your Node.js application. Of course you can always build API on top of Python backend(Flack, etc), but in that case you need to build, host and manage one more application, when you just need to run a single Python script. That's why I want to give you step by step instructions on how to achieve this.\n",
];
env.cacheDir = './.cache';
console.log(modelName);
let pipe = await pipeline('zero-shot-classification', modelName);
for (const text of texts) {
try {
const result = await pipe(text, {
labels,
});
console.log(result.scores[0]);
} catch (e) {
console.error(e);
}
}
from transformers import pipeline
model_name = "facebook/bart-large-mnli"
candidate_labels = ["street address"]
texts = [
'1 rue de la paix',
'Karlstraße 12, 80333 Munich, DE',
'Startups da América Latina demonstraram capacidade de gerar valor econômico diz fundo alemão\n',
"But what if one day, you come up with a great idea to combine those, for instance to use some Python libraries in your application, but you just do not have any idea how to integrate it with your Node.js application. Of course you can always build API on top of Python backend(Flack, etc), but in that case you need to build, host and manage one more application, when you just need to run a single Python script. That's why I want to give you step by step instructions on how to achieve this.\n",
]
model = pipeline('zero-shot-classification', model=model_name)
print(model_name)
for text in texts:
result = model(text, candidate_labels)
top_score = result["scores"][0]
print(top_score)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment