Store more data than the 1GB free limit in Firestore database by compressing text content as a Blob using Unishox

Store-More-In-Firestore-with-Unishox.js

// This assumes you have installed firebase and unishox using:
// npm i firebase
// npm i unishox2.siara.cc
var firebase = require("firebase");
var usx = require("unishox2.siara.cc");

// Get this from your firebase console
// Open/Create Project, click Project Settings, Click Add App,
// Click </>, provide a Web App Name, click Register App,
// Copy paste firebaseConfig from <script> to below:
var firebaseConfig = {
    apiKey: "***************************************",
    authDomain: "************.firebaseapp.com",
    projectId: "*******",
    storageBucket: "************.appspot.com",
    messagingSenderId: "************",
    appId: "*:************:web:**********************",
    measurementId: "************"
};

// Initialize Firebase
firebase.initializeApp(firebaseConfig);

var my_str = "The quick brown fox jumped over the lazy dog";
var out_buf = new Uint8Array(100); // A buffer with arbitrary length
var out_len = usx.unishox2_compress_simple(my_str, my_str.length, out_buf);
firebase.firestore().collection("/my-collection").add({
  key: "my-data",
  myCompressedBlob: firebase.firestore.Blob.fromUint8Array(out_buf.slice(0, out_len))
});

Store-More-In-Firestore-with-Unishox.md

Store more data than the 1GB free limit in Firestore database

Unishox can be used to store more data than the 1GB free limit in Firestore database by compressing text content and storing as a Blob.

Since the size of compressed content will be smaller than the input text, more than 1GB data can be compressed and stored into Firestore. The compression ratio could be between 30 to 60 %, depending on the type of text that is being compressed.

Unishox was developed to individually compress and decompress small strings. In general compression utilities such as zip, gzip do not compress short strings well and often expand them. For more details see here:

Unishox - Guaranteed compression for short strings

Unishox can co-exist on the blob with conventional compressed data like gzip, zip, bzip2 etc., since the first byte of Unishox is always greater than 127, and so can be distinguished from the other formats. So by using two methods, it is possible to achieve best of both worlds.

See here for Unishox specification.

Text search

This method is ideal when you have a lot of text in your document such as comments or remarks.

If you wish to search inside text, Firestore does not support full-text search even for uncompressed text. We will have to depend on third party tools or implement full-text search inside firestore ourselves. See Firestore documentation here.