Missing presentable (non-internal) implementation atm. Maybe on later Pyvo's:
>>> from __future__ import demo
... ImportError: n/a atm
- Run sync on background. Use either:
-
-
Use deferred Tasks for simple implementation to start with.
-
Runs up to 10 minutes, then raises google.appengine.runtime import DeadlineExceededError.
- Wrap the task, catch the exception and reschedule it with current state (fast).
- Keep separated from built-in on-error retries.
-
-
Dynamic Backends
- May run indefinitelly.
- Tasks added to Task Push Queues can be specified to run on particular Backends
- Raise urlfetch default deadline (5s):
-
On GAE instance decorate google.appengine.api.urlfetch.fetch()
- insert custom deadline either as a 6th positional argument, or a keyword one
-
Alternate (official) path would be to hook urlfetch calls,
apiproxy_stub_map.apiproxy.GetPreCallHooks().Append('urlfetch default deadline', hook_method, 'urlfetch')
but request.set_deadline() has no effect here at GAE instances (works on GAE SDK dev server).
- Avoid application instances being killed and restarted because of RAM exhaustion.
-
Disable ndb caching, if you use ndb to store the retrieved objects:
ndb.put_multi(changes, use_cache=False, use_memcache=False)
-
Disable Google Data lib debug logs:
client.http_client.debug = False
-
Limit batch sizes if you receive errors:
-
Happened on 128MB instances with syncing batches of 999 document changes, lowering to 500 solved that, we even pushed that down to 250 on live site.
-
It can even help you to optimize the performance. Do the tests.
- Size: 999 items, Request: 7-10 sec, Storing: 30-45 sec
- Size: 250 items, Request: 1 sec, Storing: 5 sec
-
- Limits:
-
Urlfetch free quota: 657k / day
- Witch batch size 250 it's 164m synced items daily.
- When performing at 40 items per second and one task instance, you need to set the queue to process 47 tasks in parallel to run at max troughput.
- With paid apps you get default limit: 45m / day
- Use ETags if synchronizing possibly updated objects, like Google Apps user profiles via Contacts API
- Store ETag with saved objects. On synchronization add new objects and update those with newer ETag.