Cm_Cache_Backend_Redis SUNION Chunking

Cm_Cache_Backend_Redis-patch1.diff

diff --git a/htdocs/app/code/core/Enterprise/PageCache/Model/Observer/Index.php b/htdocs/app/code/core/Enterprise/PageCache/Model/Observer/Index.php
index 57ece57..f0ad2a6 100644
--- a/htdocs/app/code/core/Enterprise/PageCache/Model/Observer/Index.php
+++ b/htdocs/app/code/core/Enterprise/PageCache/Model/Observer/Index.php
@@ -46,6 +46,7 @@ class Enterprise_PageCache_Model_Observer_Index
             $entity->setId($entityId);
             $cacheTags = array_merge($cacheTags, $entity->getCacheIdTags());
         }
+        $cacheTags = array_unique($cacheTags);
         if (!empty($cacheTags)) {
             Enterprise_PageCache_Model_Cache::getCacheInstance()->clean($cacheTags);
         }
diff --git a/htdocs/lib/Cm/Cache/Backend/Redis.php b/htdocs/lib/Cm/Cache/Backend/Redis.php
index 8352a80..961ccb8 100644
--- a/htdocs/lib/Cm/Cache/Backend/Redis.php
+++ b/htdocs/lib/Cm/Cache/Backend/Redis.php
@@ -55,6 +55,8 @@ class Cm_Cache_Backend_Redis extends Zend_Cache_Backend implements Zend_Cache_Ba
     const DEFAULT_CONNECT_TIMEOUT = 2.5;
     const DEFAULT_CONNECT_RETRIES = 1;
 
+    const SUNION_CHUNK_SIZE = 500;
+
     /** @var Credis_Client */
     protected $_redis;
 
@@ -583,10 +586,14 @@ class Cm_Cache_Backend_Redis extends Zend_Cache_Backend implements Zend_Cache_Ba
      */
     public function getIdsMatchingAnyTags($tags = array())
     {
+        $result = array();
         if ($tags) {
-            return (array) $this->_redis->sUnion( $this->_preprocessTagIds($tags));
+            $chunks = array_chunk($tags, self::SUNION_CHUNK_SIZE);
+            foreach ($chunks as $chunk) {
+                $result = array_merge($result, (array) $this->_redis->sUnion( $this->_preprocessTagIds($chunk)));
+            }
         }
-        return array();
+        return $result;
     }
 
     /**

The chunking should also be performed in _removeByMatchingAnyTags so that it can also apply to lua mode. Not sure what ideal chunk size would be, maybe allow it to be configured?

@colinmollenhour lua mode? _removeByMatchingAnyTags calls through to getIdsMatchingAnyTags.

Are you using latest version? Lua mode does SUNION inside lua script and does not call getIdsMatchingAnyTags so would still be affected by lack of chunking.

Actually looking at it now the lua_max_c_stack option would effectively perform sunion chunking already. But optimally lua_max_c_stack could still be high (5000) while sunion_chunk_size would be low (500).

Using what shipped with EE 1.14.x. Doesn't look like it's got lua support. I'll have to take a look.

Alright, not super familiar with Lua here. Looks like it would be simple enough to implement. Probably would want sunion_chunk_size to default to lua_max_c_stack if it was unspecified. Just not sure of the simplest way to test that change would be since I'm not familiar with Lua in general etc.

The patch to Cm_Cache_Backend_Redis has been submitted as a PR: colinmollenhour/Cm_Cache_Backend_Redis#89 with some modifications to make the setting configurable, and to de-duplicate the result.