Skip to content

Instantly share code, notes, and snippets.



Last active Dec 14, 2015
What would you like to do?

Using API's on Google App Engine

Demo and code

Missing presentable (non-internal) implementation atm. Maybe on later Pyvo's:

>>> from __future__ import demo
... ImportError: n/a atm


  1. Run sync on background. Use either:
  • Task Push Queues

    • Use deferred Tasks for simple implementation to start with.

    • Runs up to 10 minutes, then raises google.appengine.runtime import DeadlineExceededError.

      • Wrap the task, catch the exception and reschedule it with current state (fast).
      • Keep separated from built-in on-error retries.
  • Dynamic Backends

  1. Raise urlfetch default deadline (5s):
  • On GAE instance decorate google.appengine.api.urlfetch.fetch()

    • insert custom deadline either as a 6th positional argument, or a keyword one
  • Alternate (official) path would be to hook urlfetch calls,

    apiproxy_stub_map.apiproxy.GetPreCallHooks().Append('urlfetch default deadline', hook_method, 'urlfetch')

    but request.set_deadline() has no effect here at GAE instances (works on GAE SDK dev server).

  1. Avoid application instances being killed and restarted because of RAM exhaustion.
  • Disable ndb caching, if you use ndb to store the retrieved objects:

    ndb.put_multi(changes, use_cache=False, use_memcache=False)
  • Disable Google Data lib debug logs:

    client.http_client.debug = False
  • Limit batch sizes if you receive errors:

    • Happened on 128MB instances with syncing batches of 999 document changes, lowering to 500 solved that, we even pushed that down to 250 on live site.

    • It can even help you to optimize the performance. Do the tests.

      • Size: 999 items, Request: 7-10 sec, Storing: 30-45 sec
      • Size: 250 items, Request: 1 sec, Storing: 5 sec
  1. Limits:
  • Urlfetch free quota: 657k / day

    • Witch batch size 250 it's 164m synced items daily.
    • When performing at 40 items per second and one task instance, you need to set the queue to process 47 tasks in parallel to run at max troughput.
    • With paid apps you get default limit: 45m / day
  1. Use ETags if synchronizing possibly updated objects, like Google Apps user profiles via Contacts API
  • Store ETag with saved objects. On synchronization add new objects and update those with newer ETag.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment