Using API's on Google App Engine
Demo and code
Missing presentable (non-internal) implementation atm. Maybe on later Pyvo's:
>>> from __future__ import demo ... ImportError: n/a atm
- Run sync on background. Use either:
Use deferred Tasks for simple implementation to start with.
Runs up to 10 minutes, then raises google.appengine.runtime import DeadlineExceededError.
- Wrap the task, catch the exception and reschedule it with current state (fast).
- Keep separated from built-in on-error retries.
- Raise urlfetch default deadline (5s):
On GAE instance decorate google.appengine.api.urlfetch.fetch()
- insert custom deadline either as a 6th positional argument, or a keyword one
Alternate (official) path would be to hook urlfetch calls,
apiproxy_stub_map.apiproxy.GetPreCallHooks().Append('urlfetch default deadline', hook_method, 'urlfetch')
but request.set_deadline() has no effect here at GAE instances (works on GAE SDK dev server).
- Avoid application instances being killed and restarted because of RAM exhaustion.
Disable ndb caching, if you use ndb to store the retrieved objects:
ndb.put_multi(changes, use_cache=False, use_memcache=False)
Disable Google Data lib debug logs:
client.http_client.debug = False
Limit batch sizes if you receive errors:
Happened on 128MB instances with syncing batches of 999 document changes, lowering to 500 solved that, we even pushed that down to 250 on live site.
It can even help you to optimize the performance. Do the tests.
- Size: 999 items, Request: 7-10 sec, Storing: 30-45 sec
- Size: 250 items, Request: 1 sec, Storing: 5 sec
Urlfetch free quota: 657k / day
- Witch batch size 250 it's 164m synced items daily.
- When performing at 40 items per second and one task instance, you need to set the queue to process 47 tasks in parallel to run at max troughput.
- With paid apps you get default limit: 45m / day
- Use ETags if synchronizing possibly updated objects, like Google Apps user profiles via Contacts API
- Store ETag with saved objects. On synchronization add new objects and update those with newer ETag.