Concurrent processing in App Scripts

readme.md

Concurrent processing in App Scripts

This demonstrates how to implement a function defined at global scope executed with more than one instance, concurrently in the background, and then processed when all instances have completed.

How

This technique uses two methods available in the Apps Scripts stack: UrlFetchApp.fetchAll and the scripts.run portion of the Apps Scripts API. The former interacts with the latter via its API conventions.

UrlFetchApp.fetchAll() reaches out to external resources (including APIs) asyncronously. The requests that are passed through are not guaranteed to be executed in the order they appear. However, the responses do come back in the order they appear, since the function hands it back to us that way.

Since the App Script API (formally known as execution API) can be interacted with the same as an external resource, we can use that to call our own function. All we have to do is process the results.

Requirements

As this is an advanced topic, there are some special considerations to take into account, such as setup and limitations:

• You'll need to enable the Apps Scripts API on your project
• You'll need to Publish —> Deploy as executable API
• You'll need to edit your manifest to include one of the scopes listed in "Authorization Scopes" in the documentation i.e. "oauthScopes": ["https://www.googleapis.com/auth/script.external_request"]
• You'll need to make sure doSomething function is available as a function on the project (cannot end with underscore)

Common pitfalls

If the script logs instructions to go to the Google Platform Project and turn on Apps Scripts API, copy the link and do that, as it is required to work.

If the script log outputs "Requested entity was not found" this could be either because you haven't defined scope, or because the apps scripts api was turned on recently and systems haven't propagated. If you have done all those, then I believe you have run into the same bug I have. (When I change devMode to true it works.) Solution seems to edit the source, and save again.

Implications & Limitations

Subject to quota limitations, depending on the kind of domain in which the scripts are deployed. Please refer to Current quotas section for reference. Note that the quota counts for .fetchAll is incremented according to how many resquests are passed to it. In other words, 10 requests passed to UrlFetchApp.fetchAll counts for use of ten towards the quota.

Any project that uses this method will have to manage its manifest and scopes manually. This is because we have defined one scope manually, and thus the stack is no longer able to automanage it (by detecting what methods you are using and adding them automagically).

The Apps Scripts API at the time of writing was not compatible with the use of service accounts. This is not relevant to our example.

Example

Copy the code, run the myFunction function. In the end it outputs the following:

[[row, of, students], [another row, of, students], [row, of, teachers], [another row, of, teachers], [row, of, parents], [another row, of, parents], [row, of, classes], [another row, of, classes]]

Since there are four doSomething functions running asyncronously, it only takes 5 seconds, and not 15.

concurrency.gs

/**
 * Pretends to take a long time to return two rows of data
 *
 * @param {string} endpoint 
 * @return {ResponseObject} 
 */ 
function doSomething (endpoint) {
  Utilities.sleep(5 * 1000);  // sleep for 5 seconds
  return {
    numberOfRows: 2,
    rows: [
      ['row of ' + endpoint],
      ['another row of ' + endpoint]
    ]
  };
}

function main() {
  
  // set up the four requests
  var url, parameterList, requests = [];
  url = 'https://script.googleapis.com/v1/scripts/' + ScriptApp.getScriptId() + ':run'; 
  parameterList = [
    ['students'],
    ['teachers'],
    ['parents'],
    ['classes'],
  ];

  parameterList.forEach(function (parameters) {
    requests.push({
      url: url,
      method: 'post',
      headers: {
        "Authorization": "Bearer " + ScriptApp.getOAuthToken(),
        'Content-Type': 'application/json'
      },
      payload: JSON.stringify({
        "devMode": true,
        "function": 'doSomething',
        "parameters": parameters
      }),
      muteHttpExceptions: true  // you can put "advanced parameters" on this object directly
    });
  });

  // Now execute the instances with a call to the app script execution API endpoint
  var responses;
  responses = UrlFetchApp.fetchAll(requests);
    
  // process the results with Array.reduce
  // demonstrates how to unpack the responses
  var result;
  result = responses.reduce(function (acc, response, index) {
    var request, json, returnedValues, params;
    json = JSON.parse(response);
    if (json.error) { 
      // early exits since we got error, output to Logger.log
      Logger.log(json.error.message); 
      return acc;
    }
    responseObject = json.response.result;  // value of return statement, for us an ResponseObject
    request = requests[index];  // the original request, for reference
  
    Array.prototype.push.apply(acc, apiResponseObject.rows);  // appends the rows property to the end of acc
    return acc;
  }, []);

  Logger.log(result);
}

/**
 *  Calls a function, reports how long it took.
 *
 *  @param {object} np named parameters 
 *  @param {function} np.func The function to time, required
 *  @param {array} np.args List of parameters to pass to np.func
 *  @param {string} np.name The moniker, default is the name of the passed function
 *  @param {any} np.ctx What the this is supposed to refer to when passed functione executes, default is np.func
 *  @param {bool} np.useStackDriver Use console.log to log result, otherwise use Logger.log. Default is true
 *  @return {any} Returns the result of passed function
 *
 */
function timeit_(np) {
  var now, then, args, ret;
  np = np || {};
  if (!np.func) throw Error("Call signature requires func");
  np.name = np.name || np.func.name;
  np.ctx = np.ctx || np.func;
  np.args = np.args || [];
  if (typeof np.useStackDriver === 'undefined') np.useStackDriver = true;

  now = Date.now();
  var ret = np.func.apply(np.ctx, np.args);
  then = Date.now();
  var outputText = 'Time for ' + np.name + ': ' + ((then - now) / 1000).toString() + ' seconds.';
  if (np.useStackDriver) console.log(outputText);
  else Logger.log(outputText);
  return ret;
}  


/**
 * RUN THIS
 */
function myFunction () {
  timeit_({func: main, useStackDriver: false});
}

Making requests to the url with the script id and devMode: true did not work for me.

What worked however was

• deploying the script as executable api
• making the request to deployment URL (note that it changes in every deployment)
• setting devMode to false