Benchmarking OpenAI Tokenizers for Node.js

README.md

Benchmarking OpenAI Tokenizers for Node.js

Context

Developers often curios about how Node.js ports of OpenAI's tiktoken perform. This question is usually prompted by Vercel's Edge Runtime and AWS Lambda requiring more work to make ports with Python binding and WASM to work compared to the purely JavaScript js-tiktoken.

Benchmark results

$ npm start

> tiktoken-benchmark@1.0.0 start
> node index.js

Fastest is @dqbd/tiktoken with 1992 ops/sec
Slowest is js-tiktoken with 1494 ops/sec (75%)

With short completion content:

Fastest is js-tiktoken with 31334 ops/sec
Slowest is @dqbd/tiktoken with 5460 ops/sec (17%)

Content:

Once upon a time, in a small village nestled among rolling green hills, lived a young girl named Lily. She possessed an insatiable curiosity and an adventurous spirit that often led her to explore the world beyond her home.

Libraries

• @dqbd/tiktoken uses WASM
• js-tiktoken is pure JavaScript

Environment

MB Air Apple M1 Node.js v18.18.0

Related blog post: https://openmeter.io/blog/token-usage-with-openai-streams-and-nextjs

index.js

const { encodingForModel } = require("js-tiktoken");
const { encoding_for_model } = require("@dqbd/tiktoken");
const { Suite } = require("benchmark");

const model = "gpt-3.5-turbo";
const enc1 = encodingForModel(model);
const enc2 = encoding_for_model(model);

// Input: Write me a one page long novel
const completion = `Once upon a time, in a small village nestled among rolling green hills, lived a young girl named Lily. She possessed an insatiable curiosity and an adventurous spirit that often led her to explore the world beyond her home.
One sunny morning, as she skipped along the path towards the nearby forest, Lily noticed something peculiar glimmering behind an ancient oak tree. Curiosity piqued, she rushed over to investigate and discovered a key lying on the ground. It was no ordinary key; it had intricate carvings embellishing its handle and seemed to emit a faint glow.
Lily's imagination ran wild with possibilities of where this mysterious key could lead her. With excitement bubbling within her heart, she set off on an adventure like no other.
Following the guidance of intuition alone, Lily found herself standing before an ornate door hidden deep within the heart of the forest. The door appeared weathered yet grandiose—a portal into another realm waiting for someone brave enough to unlock its secrets.
Taking a deep breath filled with anticipation, Lily inserted the enigmatic key into the lock and turned it slowly. To her astonishment, as soon as she did so, vines unfurled from either side of the doorframe until they completely obscured everything around them—creating an impenetrable wall blocking any glimpse of what lay beyond.
Undeterred by this unforeseen obstacle but filled with trepidation nonetheless, Lily reached out tentatively for one vine-covered doorknob. As if sensing her determination and pure-heartedness, both knobs sprung open simultaneously without resistance—an invitation to enter this mystical realm.
As soon as Lily stepped through that threshold into unknown territory—a place suspended between reality and dreams—she felt a wave of enchantment wash over her senses. A vibrant landscape unfolded before her eyes: towering trees adorned with glowing flowers stretched toward azure skies while cascading waterfalls whispered their melodies in harmony with nature's symphony.
With each step, Lily discovered something extraordinary—a magical creature hiding behind a tree trunk or a shimmering pond revealing its mesmerizing inhabitants. She encountered fairies dancing on beams of sunlight and unicorns grazing in meadows of iridescent flowers.
As time passed, Lily explored every nook and cranny, embracing the beauty surrounding her with wide-eyed wonder. But deep within her heart, she knew that this realm was not meant for her to stay forever—its magic belonged to another world.`;

new Suite()
  .add("js-tiktoken", function () {
    enc1.encode(completion);
  })
  .add("@dqbd/tiktoken", function () {
    enc2.encode(completion);
  })
  .on("complete", function () {
    enc2.free();

    const fastest = this.filter("fastest")[0];
    const slowest = this.filter("slowest")[0];
    const percentage = Math.round((slowest.hz / fastest.hz) * 100);

    console.log(`Fastest is ${fastest.name} with ${Math.round(fastest.hz)} ops/sec`);
    console.log(
      `Slowest is ${slowest.name} with ${Math.round(slowest.hz)} ops/sec (${percentage}%)`,
    );
  })
  // run async
  .run({ async: true });

package.json

{
  "name": "tiktoken-benchmark",
  "version": "1.0.0",
  "description": "Benchmarking multiple tiktoken libraries",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "author": "OpenMeter",
  "license": "MIT",
  "dependencies": {
    "@dqbd/tiktoken": "^1.0.7",
    "benchmark": "^2.1.4",
    "js-tiktoken": "^1.0.7"
  }
}

Moving the initializer inside the test has a big impact:

  .add("js-tiktoken", function () {
    const enc1 = encodingForModel(model);
    enc1.encode(completion);
  })
  .add("@dqbd/tiktoken", function () {
    const enc2 = encoding_for_model(model);
    enc2.encode(completion);
    enc2.free();
  })

Fastest is @dqbd/tiktoken with 20 ops/sec
Slowest is js-tiktoken with 10 ops/sec (52%)