api-ga-gae.md

Using API's on Google App Engine

Demo and code

Missing presentable (non-internal) implementation atm. Maybe on later Pyvo's:

>>> from __future__ import demo
... ImportError: n/a atm

Hints

1. Run sync on background. Use either:

• Task Push Queues

  • Use deferred Tasks for simple implementation to start with.

  • Runs up to 10 minutes, then raises google.appengine.runtime import DeadlineExceededError.

    • Wrap the task, catch the exception and reschedule it with current state (fast).
    • Keep separated from built-in on-error retries.

• Dynamic Backends

  • May run indefinitelly.
  • Tasks added to Task Push Queues can be specified to run on particular Backends

2. Raise urlfetch default deadline (5s):

• On GAE instance decorate google.appengine.api.urlfetch.fetch()

  • insert custom deadline either as a 6th positional argument, or a keyword one

• Alternate (official) path would be to hook urlfetch calls,

  apiproxy_stub_map.apiproxy.GetPreCallHooks().Append('urlfetch default deadline', hook_method, 'urlfetch')
  

  but request.set_deadline() has no effect here at GAE instances (works on GAE SDK dev server).

3. Avoid application instances being killed and restarted because of RAM exhaustion.

• Disable ndb caching, if you use ndb to store the retrieved objects:

  ndb.put_multi(changes, use_cache=False, use_memcache=False)
  

• Disable Google Data lib debug logs:

  client.http_client.debug = False
  

• Limit batch sizes if you receive errors:

  • Happened on 128MB instances with syncing batches of 999 document changes, lowering to 500 solved that, we even pushed that down to 250 on live site.

  • It can even help you to optimize the performance. Do the tests.

    • Size: 999 items, Request: 7-10 sec, Storing: 30-45 sec
    • Size: 250 items, Request: 1 sec, Storing: 5 sec

4. Limits:

• Urlfetch free quota: 657k / day

  • Witch batch size 250 it's 164m synced items daily.
  • When performing at 40 items per second and one task instance, you need to set the queue to process 47 tasks in parallel to run at max troughput.
  • With paid apps you get default limit: 45m / day

5. Use ETags if synchronizing possibly updated objects, like Google Apps user profiles via Contacts API

• Store ETag with saved objects. On synchronization add new objects and update those with newer ETag.