Reindexing Oak Async Indexes on a Clone AEM Instance

ClonedReindexingInstructions.md

How to Reindex AEM on a Clone Instance and Sync over the Changes

1. Install Oak 1.0.23 hotfix via the AEM package manager (Download from here https://files.acrobat.com/a/preview/40ddf9c0-83de-48b3-837b-602388e68b06)
2. Validate that the hotfix is installed by going to /system/console/bundles UI and validate that oak-core version is now at 1.0.23. Also go to /crx/de/index.jsp and see the Oak version on the right side.
3. Clone your AEM instance to another VM for reindexing
4. Download these oak-run jars:

• https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/1.1.8/oak-run-1.1.8.jar
• https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/1.0.23/oak-run-1.0.23.jar

4. Upload the oak-run jars to the clone server
5. Stop AEM (stop all AEM instances if a cluster or cold standby)
6. Run this command (using oak-run-1.0.23.jar)

• java -Xmx4096m -jar oak-run-1.0.23.jar checkpoints /path/to/repository/segmentstore

7. Run this command (using oak-run-1.0.23.jar)

• java -Xmx4096m -jar oak-run-1.0.23.jar checkpoints /path/to/repository/segmentstore rm-all

8. Run this command to perform offline compaction (using oak-run-1.0.23.jar)

• nohup java -Dtar.memoryMapped=true -Dupdate.limit=5000000 -Dcompaction-progress-log=1500000 -Dcompress-interval=10000000 -Doffline-compaction=true -Xmx10g -jar oak-run-1.0.23.jar compact crx-quickstart/repository/segmentstore > tarcompaction.log 2>&1 &

9. Using your browser, download this file https://gist.githubusercontent.com/stillalex/43c49af065e3dd1fd5bf/raw/9e726a59f75b46e7b474f7ac763b0888d5a3f0c3/rmNode.groovy
10. Upload that file to the server to the same folder as the oak-run jar
11. Run this command using oak-run-1.1.8.jar version (downloaded in step 2 above) to open the console tool:

• java -Xmx4096m -jar oak-run-1.1.8.jar console crx-quickstart/repository/segmentstore

12. In the console tool shell, run these commands:

• :load rmNode.groovy
• rmNode(session, "/:async")
• rmNode(session, "/oak:index/lucene")
• :exit

13. Start AEM (on startup all OOTB async indexes would be recreated from scratch and fully indexed)
14. Monitor the indexing process by looking for log messages like these.

These show that the indexing started:

17.11.2015 16:22:18.481 *INFO* [FelixStartLevel] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing would be performed for following indexes [/oak:index/cqCloudServiceConfig, /oak:index/slingeventEventId, /oak:index/cqPayloadPath]
17.11.2015 16:22:26.624 *INFO* [pool-70-thread-5] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing would be performed for following indexes [/oak:index/damFileSize, /oak:index/cqLastModified]
17.11.2015 16:30:38.060 *INFO* [pool-9-thread-5] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Found a new index node [damFileSize]. Reindexing would be requested
17.11.2015 16:30:38.074 *INFO* [pool-9-thread-5] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Found a new index node [cqLastModified]. Reindexing would be requested
17.11.2015 16:30:38.074 *INFO* [pool-9-thread-5] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing would be performed for following indexes [/oak:index/damFileSize, /oak:index/lucene, /oak:index/cqLastModified]

These messages show indexing progress:

17.11.2015 16:31:04.293 *INFO* [pool-9-thread-5] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing Traversed #220000 /libs/granite/i18n/pt-br/Last build_
17.11.2015 16:31:05.603 *INFO* [pool-9-thread-5] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing Traversed #230000 /libs/granite/security/content/groupEditor/items/page/items

If you see this log message then it completed successfully:

17.11.2015 16:31:08.294 *INFO* [pool-9-thread-5] org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate Reindexing (async) completed for indexes: [/oak:index
/damFileSize*(93), /oak:index/lucene*(237527), /oak:index/cqLastModified*(1525)] in 30.24 s

If you never saw a "completed" log message and this log message shows up again (or similar "Reindexing would be ..." message) then it means indexing failed and has gone in a loop:

17.11.2015 16:30:38.074 *INFO* [pool-9-thread-5] org.apache.jackrabbit.oak.plugins.index.IndexUpdate Reindexing would be performed for following indexes [/oak:index/damFileSize, /oak:index/lucene, /oak:index/cqLastModified]

If it failed then go to /system/console/slinglog UI and enable Debug level logging for org.apache.jackrabbit.oak.plugins.index. The next time the indexing loops (showing "Reindexing would be performed" you would see a log message before the error showing an error indicating why indexing failed. Contact Adobe AEM Customer Care and provide them that log file. 15. Copy over changes that occurred on the production instance during the reindexing on the copy

1. Install this tool to the production live (source) instance:
   http://adobe-consulting-services.github.io/acs-aem-commons/features/query-packager.html
2. Create package using a search for pages that changed since the copy was done. Use this query for pages (but change the date):
   • //element(*,cq:PageContent)[@cq:lastModified >= xs:dateTime('2015-09-16T00:00:00.000-05:00')]
3. Create package using a search for assets that changed since the copy was done. Use this query for assets (but change the date):
   • //element(*, dam:AssetContent)[@jcr:lastModified > xs:dateTime('2015-09-16T00:00:00.000-05:00')]
4. Create package using a search for tags that changed since the copy was done. Use this query for tags (but change the date):
   • //element(*, cq:Tag)[@jcr:created > xs:dateTime('2015-09-16T00:00:00.000-05:00')]
5. Install all the packages to the (clone) AEM instance (install the tags first, then the assets then pages)