Celery / Redis / Kombu Upgrade Notes

readme.md

Celery / Redis / Kombu

Django Autocompleter Changes

1. Revert this commit: https://github.com/ycharts/django-autocompleter/commit/ca9881e24da522302c449f36be0a574d36c5332e
2. Revert this commit: https://github.com/ycharts/django-autocompleter/commit/73326273d175424a0af2202134e9dfa1a21de724
3. Bump version to latest dot release ( at the time of writing this would be 0.8.4)

YCharts Repo Changes

1. hiredis (hiredis-py) is upgrade to 1.0.0 from 0.2.0
2. Redis (redis-py) upgrade to 3.4.1 from 2.10.5
   1. Read the migration guide for reference: https://github.com/andymccurdy/redis-py#upgrading-from-redis-py-2x-to-30
      • We use the Redis client instead of StrictRedis which is now no longer supported since StrictRedis. So we need to make sure that the differing implementaitons that Redis client had for zadd and pipeline do not require changes in our code. We use both zadd and pipeline in our code.
   2. No changes for django-redis-sessions
   3. No changes for redlock
3. Upgrade Celery to 4.4.x from 4.2.0
4. Upgrade Kombu to the latest version (4.6.8 from 4.3.0)
   • this issue is closed: celery/kombu#1019 -- CLOSED
   • this issue is closed: celery/kombu#1122 -- MERGED (included in 4.6.7)
   • this issue is closed: celery/kombu#1114 & celery/kombu#1145 -- MERGRED (included in 4.6.8)
     • We need this so we can configure a timeout that is sensible for our workload per Andy’s comment
5. Revert this commit: https://github.com/ycharts/ycharts/pull/6599/files#diff-9eb78032baccc38f3f987c4fc851e30e
   • Specifically change that code related to catching the exception since it shouldn't happen in the latest version.
6. Add back these reverted changes in this PR: https://github.com/ycharts/ycharts/pull/5782/files
   1. Make sure you get the django-autocompleter version correct though after you update the version in Django Autocompleter
   2. Specifically audit all uses of zadd since they need to be updated.
7. Determine if we need to change the health_check_interval in BROKER_TRANSPORT_OPTIONS

@jls83 @kdabiedeen couple things to point out post these changes:

1. Should we be using ConnectionPool? https://github.com/andymccurdy/redis-py#connection-pools
   • I feel like this would be something we should probably do especially when it comes to our web machines. Can you take a deeper look and propose a way forward or why we shouldn't?
2. If use a connection pool is that when we would really need to use health_check_interval config option?
3. Can we update all uses of .pipeline() to be used a context manager?
4. Should we be setting retry_on_timeout? Why not?
5. Can you read through all changes and make sure there isn't anything else to consider while we are deep in redis land?