Performance differences for classification (BART) between Transformers.js and Transformers (Python)

README.md

Performance differences for classification (BART) between Transformers.js and Transformers (Python)

This is a GIST for this issue in Transformer.js repository

Trying to classify the following texts as street address:

• '1 rue de la paix',
• 'Karlstraße 12, 80333 Munich, DE',
• 'Startups da América Latina demonstraram capacidade de gerar valor econômico diz fundo alemão',
• "But what if one day, you come up with a great idea to combine those, for instance to use some Python libraries in your application, but you just do not have any idea how to integrate it with your Node.js application. Of course you can always build API on top of Python backend(Flack, etc), but in that case you need to build, host and manage one more application, when you just need to run a single Python script. That's why I want to give you step by step instructions on how to achieve this.\n",

$ python bart.py
facebook/bart-large-mnli
0.9975677132606506
0.9981901049613953
0.004343739710748196
0.022633692249655724

$ node bart.js
Xenova/bart-large-mnli
0.4556246405304541
0.5631336436601322
0.6358004417358224
0.6758611902199643

Run yourself

Javascript

• Install: npm i @xenova/transformers
• Run: node bart.js

Python

• Install: pip install transformers
• Run: python bart.py

bart.js

import { pipeline, env } from '@xenova/transformers';

const modelName = 'Xenova/bart-large-mnli';
const labels = ['a street address'];
const texts = [
  '1 rue de la paix',
  'Karlstraße 12, 80333 Munich, DE',
  'Startups da América Latina demonstraram capacidade de gerar valor econômico diz fundo alemão\n',
  "But what if one day, you come up with a great idea to combine those, for instance to use some Python libraries in your application, but you just do not have any idea how to integrate it with your Node.js application. Of course you can always build API on top of Python backend(Flack, etc), but in that case you need to build, host and manage one more application, when you just need to run a single Python script. That's why I want to give you step by step instructions on how to achieve this.\n",
];

env.cacheDir = './.cache';
console.log(modelName);

let pipe = await pipeline('zero-shot-classification', modelName);

for (const text of texts) {
  try {
    const result = await pipe(text, {
      labels,
    });
    console.log(result.scores[0]);
  } catch (e) {
    console.error(e);
  }
}

bart.py

from transformers import pipeline

model_name = "facebook/bart-large-mnli"
candidate_labels = ["street address"]
texts = [
  '1 rue de la paix',
  'Karlstraße 12, 80333 Munich, DE',
  'Startups da América Latina demonstraram capacidade de gerar valor econômico diz fundo alemão\n',
  "But what if one day, you come up with a great idea to combine those, for instance to use some Python libraries in your application, but you just do not have any idea how to integrate it with your Node.js application. Of course you can always build API on top of Python backend(Flack, etc), but in that case you need to build, host and manage one more application, when you just need to run a single Python script. That's why I want to give you step by step instructions on how to achieve this.\n",
]

model = pipeline('zero-shot-classification', model=model_name)

print(model_name)

for text in texts:
    result = model(text, candidate_labels)
    top_score = result["scores"][0]
    print(top_score)