Note that offline compaction requires a long duration of downtime from 30 minutes to 7 hours (and in extreme cases more time).
Offline Oak compaction with debug logging:
- Stop Oak / AEM
- Install the latest Oak hotfix to your AEM instance:
- AEM 6.0 - https://helpx.adobe.com/experience-manager/kb/aem6-available-hotfixes.html
- AEM 6.1 - https://helpx.adobe.com/experience-manager/kb/aem61-available-hotfixes.html
- AEM 6.2 - https://helpx.adobe.com/experience-manager/kb/aem62-available-hotfixes.html
- Later - https://helpx.adobe.com/experience-manager/aem-releases-updates.html
- Download the oak-run version matching your oak version: https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/
- AEM6.0 - use oak-run 1.0.x - if you have Oak 1.0.11 or earlier installed in AEM then use oak-run 1.0.11 otherwise use the version that matches the installed on in AEM.
- AEM6.1 - use latest oak-run 1.2.x version
- AEM6.2 - use latest oak-run 1.4.x version
- AEM6.3 - use latest oak-run 1.6.x version
- AEM6.4 - use latest oak-run 1.8.x version
- AEM6.5 - use latest oak-run 1.10.x version
- Upload the oak-run jar file to your server
- Create a file logback-compaction.xml (on the server in the same folder as the oak-run jar)
<configuration>
<appender name="STDERR" class="ch.qos.logback.core.ConsoleAppender">
<target>System.out</target>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.jackrabbit.oak.plugins.segment.Compactor" level="INFO"/>
<root level="warn">
<appender-ref ref="STDERR" />
</root>
</configuration>
Note: Debug logging can be enabled by setting level="DEBUG" 6. Run this command from the same folder as logback-compaction.xml (change the paths and jar file name to match your system’s paths, and tune -Xmx10g to not exceed your server's memory while leaving room for the OS memory):
Linux, Unix, Mac:
nohup sudo -u crx nohup /usr/java/latest/bin/java -server -Xmx30g -Dtar.memoryMapped=true -Doak.compaction.eagerFlush=true -Dcompaction-progress-log=5000000 -Dupdate.limit=2500000 -Dlogback.configurationFile=logback-compaction.xml -Dcompress-interval=150000000 -jar oak-run-*.jar compact /path/to/segmentstore >> oak-tar-compact-a0.log 2>>oak-tar-error-a0.log &
Windows (note that memoryMapped setting is removed):
java -server -Xmx30g -Dtar.memoryMapped=false -Doak.compaction.eagerFlush=true -Dcompaction-progress-log=5000000 -Dupdate.limit=2500000 -Dlogback.configurationFile=logback-compaction.xml -Dcompress-interval=150000000 -jar oak-run-*.jar compact C:\path\to\segmentstore >> oak-tar-compact-a0.log
- Once the process completes then start Oak / AEM
Andrew,
We're pushing the limits compacting a very large 6.0 repository (oak 1.0.34) and we haven't been successful lately (after it grew). I've run it with the following:
/usr/bin/java -jar -Xms54g -Xmx54g -XX:MaxDirectMemorySize=6g -XX:MaxMetaspaceSize=1g -XX:+UseCompressedOops -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -Xloggc:/tmp/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=7 -XX:GCLogFileSize=128M -Djava.io.tmpdir=/opt/aem/crx-quickstart/tmp -XX:+CMSScavengeBeforeRemark -XX:+UseCompressedOops -Djava.io.tmpdir=/opt/aem/crx-quickstart/tmp -Dtar.memoryMapped=true -Dupdate.limit=2500000 -Dcompaction-progress-log=1500000 -Dcompress-interval=5000000 -Doffline-compaction=true /opt/aem/oak-run-1.0.34.jar compact /opt/aem/crx-quickstart/repository/segmentstore
We've tried with a couple variations on the settings (lower xmx with higher MaxDirect... lower update.limit and compress-interval).
There is 64G RAM on this box yet we keep running out of heap!
Do you have any suggestions on parameters to add/remove or increase in sizing?
Thank you,
Bryce