Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Using the Google Apps Script Cache Service for objects above 100Kb
function ChunkyCache(cache, chunkSize){
return {
put: function (key, value, timeout) {
var json = JSON.stringify(value);
var cSize = Math.floor(chunkSize / 2);
var chunks = [];
var index = 0;
while (index < json.length){
cKey = key + "_" + index;
chunks.push(cKey);
cache.put(cKey, json.substr(index, cSize), timeout+5);
index += cSize;
}
var superBlk = {
chunkSize: chunkSize,
chunks: chunks,
length: json.length
};
cache.put(key, JSON.stringify(superBlk), timeout);
},
get: function (key) {
var superBlkCache = cache.get(key);
if (superBlkCache != null) {
var superBlk = JSON.parse(superBlkCache);
chunks = superBlk.chunks.map(function (cKey){
return cache.get(cKey);
});
if (chunks.every(function (c) { return c != null; })){
return JSON.parse(chunks.join(''));
}
}
}
};
};
function testGetCacheFrom(){
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Data');
var data = sheet.getDataRange().getValues();
var chunky = ChunkyCache(CacheService.getDocumentCache(), 1024*90);
chunky.put('Data', data, 120);
var check = chunky.get('Data');
var sheetPut = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Out');
for (c in check) {
sheetPut.appendRow(check[c]);
}
}
@pilbot

This comment has been minimized.

Copy link
Owner Author

@pilbot pilbot commented Jan 6, 2016

The ChunkyCache allows storage of objects that are larger than 100Kb by splitting them on a configurable number of bytes. It uses JSON.stringify to export the object so some types will be coalesced.

The test function will copy all data in the spreadsheet Data to the spreadsheet Out going through the cache.

@rajeshruhil

This comment has been minimized.

Copy link

@rajeshruhil rajeshruhil commented Jul 6, 2017

What is the maximum time, we can keep data in cache any why is there +5 with timeout.

@patchypt

This comment has been minimized.

Copy link

@patchypt patchypt commented Jul 19, 2017

Thanks for the code.
It helps me a lot in my project.

@nielsfogt

This comment has been minimized.

Copy link

@nielsfogt nielsfogt commented Oct 6, 2017

LIFE SAVER -- THANK YOU SO MUCH!!!

@ajdrake

This comment has been minimized.

Copy link

@ajdrake ajdrake commented Nov 1, 2019

this is excellent, thank you. This should be included in google script cache service

@Max-Makhrov

This comment has been minimized.

Copy link

@Max-Makhrov Max-Makhrov commented Mar 11, 2021

Hi, still great code for use, and it's 5 years now.

The limit is still same, see the reference:

The maximum amount of data that can be stored per key is 100KB

I wondered if it works fine with larger "byte" size as your code gives the char size, and 1 char is not always 1 byte,

Here's the code for check:

  var chunky = ChunkyCache(CacheService.getDocumentCache(), 1024*200);
  var s = '🧞‍♂️'.repeat(1024 * 400); // 🧞‍♂️
  chunky.put('Data', s, 120);
  var check = chunky.get('Data');
  // console.log(check);
  // console.log(Utilities.newBlob(s).getBytes().length / 1024);

The final row of sample code ↑ checks real K-byte syze. Appeared, Google Quota checks NOT size of bytes, but simple text length. In this case your code is perfect.

Your code also devides chunk size by 2:
var cSize = Math.floor(chunkSize / 2);
It is really interesting because it gives axtra room for total size. Real size would be twice lower in this case. I though of the other limitation:

The cap for cached items is 1,000

It should be better to consider that number of parts (cached items) is limited. So there's another upper limit, and we better make a chunk bigger if we have to store more than 1,000 parts (chunks) of cahce data.

@contributorpw

This comment has been minimized.

Copy link

@contributorpw contributorpw commented Jul 16, 2021

I wondered if it works fine with larger "byte" size as your code gives the char size, and 1 char is not always 1 byte,

This may depend on the specifics of the coding language. https://262.ecma-international.org/5.1/#sec-4.3.16

4.3.16 String value
primitive value that is a finite ordered sequence of zero or more 16-bit unsigned integer
NOTE A String value is a member of the String type. Each integer value in the sequence usually represents a single 16-bit unit of UTF-16 text. However, ECMAScript does not place any restrictions or requirements on the values except that they must be 16-bit unsigned integers.

It is really interesting because it gives axtra room for total size. Real size would be twice lower in this case.

Check the @ChrisBaker97's fork https://gist.github.com/ChrisBaker97/c19ff5ee1a43a1d68178c74f39962516

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment