Streaming ChatGPT API Responses in Node.js using async generators

Stream ChatGPT API.md

Streaming ChatGPT API Responses in Node.js

This repository contains a reusable function leveraging async generators to stream real-time responses from OpenAI's ChatGPT API, word by word, in a Node.js environment. Instead of waiting for the entire response to be generated, this function allows for an interactive experience where the AI's thoughts are streamed as they are produced.

Prerequisites

1. Node.js 18.14 or higher (since we're using nodejs native fetch)
2. A valid API key for OpenAI's ChatGPT

Usage

First, make sure to set the CHAT_GPT_API_KEY constant in the script to your ChatGPT API key.

Here's a quick example of how to use the function:

import { streamChatgptApi } from "./streamChatgptApi.js";

for await (const responsePart of streamChatgptApi("Hello AI, I am a human.")) {
  if (responsePart.finish_reason) {
    console.log("finished execution with reason:", responsePart.finish_reason);
  } else {
    process.stdout.write(responsePart.delta.content);
  }
}

How It Works

The function streamChatgptApi makes a POST request to ChatGPT with the stream parameter set to true. It then reads the incoming stream and yields the message parts as they arrive.

This makes it possible to process each chunk of the response in real-time, as illustrated in the usage example above.

For a deep dive into the intricacies of this approach, including how server-sent events work, and the significance of streaming via POST, check out my detailed blog post:

Stream Real-time Feedback with ChatGPT: SSE via Fetch in Node.js

streamChatgptApi.js

import assert from "node:assert";

const CHAT_GPT_API_KEY = "<YOUR_API_KEY>"; // or from env, e.g. process.env.CHAT_GPT_API_KEY

/**
 * Example usage:
 *
 * for await (const responsePart of streamChatgptApi("Hello World.")) {
 *   if (responsePart.finish_reason) {
 *     console.log("finished execution with reason:", responsePart.finish_reason);
 *   } else {
 *     process.stdout.write(responsePart.delta.content);
 *   }
 * }
 *
 * @param {string} userMessage
 * @return {AsyncGenerator<Choice, void, *>}
 */
export async function* streamChatgptApi(userMessage) {
    /** @link https://platform.openai.com/docs/api-reference/chat/create */
    const body = {
        model: "gpt-3.5-turbo-0613",
        stream: true,
        messages: [{ role: "user", content: userMessage }],
    };

    const response = await fetch("https://api.openai.com/v1/chat/completions", {
        method: "POST",
        headers: {
            Authorization: `Bearer ${CHAT_GPT_API_KEY}`,
            "Content-Type": "application/json",
        },
        body: JSON.stringify(body),
    });

    const reader = response?.body?.getReader();
    // make sure the reader exists
    assert.ok(reader);

    const textDecoder = new TextDecoder("utf-8");
    while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        const str = textDecoder.decode(value);

        // get all data objects from the string which start with "data: " and remove the first one because it must be empty (since it started with DATA)
        const dataObjects = str.split("data: ").slice(1);

        for (const dataString of dataObjects) {
            if (dataString.trim() === "[DONE]") break;

            /** @type {ChatCompletionChunk} */
            const data = JSON.parse(dataString);
            yield data.choices[0]; // this will be returned on each iteration
        }
    }
}


/**
 * Represents a chunk completion of chat.
 * @typedef {Object} ChatCompletionChunk
 * @property {string} id - The ID of the chunk.
 * @property {Choice[]} choices - The choices for the chunk.
 * @property {number} created - The timestamp of creation.
 * @property {string} model - The model used.
 * @property {string} object - The type of object.
 */

/**
 * Represents a choice.
 * @typedef {Object} Choice
 * @property {Delta} delta - The delta of the choice.
 * @property {("stop" | "length" | "function_call" | null)} finish_reason - The reason for finishing.
 * @property {number} index - The index of the choice.
 */

/**
 * Represents a delta.
 * @typedef {Object} Delta
 * @property {string|null} [content] - The contents of the chunk message.
 * @property {("system" | "user" | "assistant" | "function")} [role] - The role of the author of this message.
 */