gistfile1.txt

02:55:32,526 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-scala-common-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,627 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@6f8e8894 - URL [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml] is not of type file
02:55:33,372 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
02:55:33,420 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
02:55:33,511 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
aether:magda-config alex$ kubectl logs indexer-7579488d5c-4tn4w | head -n20
02:55:32,526 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-scala-common-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,627 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@6f8e8894 - URL [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml] is not of type file
02:55:33,372 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
02:55:33,420 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
02:55:33,511 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
02:55:34,420 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
02:55:34,427 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ASYNC]
02:55:34,427 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to ch.qos.logback.classic.AsyncAppender[ASYNC]
02:55:34,427 |-INFO in ch.qos.logback.classic.AsyncAppender[ASYNC] - Attaching appender named [STDOUT] to AsyncAppender.
02:55:34,428 |-INFO in ch.qos.logback.classic.AsyncAppender[ASYNC] - Setting discardingThreshold to 51
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [ch.qos.logback] to ERROR
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.sksamuel] to ERROR
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
02:55:34,437 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ASYNC] to Logger[ROOT]
02:55:34,437 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
aether:magda-config alex$ kubectl logs indexer-7579488d5c-4tn4w | head -n40
02:55:32,526 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-scala-common-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,627 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@6f8e8894 - URL [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml] is not of type file
02:55:33,372 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
02:55:33,420 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
02:55:33,511 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
02:55:34,420 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
02:55:34,427 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ASYNC]
02:55:34,427 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to ch.qos.logback.classic.AsyncAppender[ASYNC]
02:55:34,427 |-INFO in ch.qos.logback.classic.AsyncAppender[ASYNC] - Attaching appender named [STDOUT] to AsyncAppender.
02:55:34,428 |-INFO in ch.qos.logback.classic.AsyncAppender[ASYNC] - Setting discardingThreshold to 51
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [ch.qos.logback] to ERROR
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.sksamuel] to ERROR
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
02:55:34,437 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ASYNC] to Logger[ROOT]
02:55:34,437 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
02:55:34,439 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@3cfdd820 - Registering current configuration as safe fallback point

[INFO] [10/15/2018 02:55:34.524] [main] [IndexerApp$(akka://indexer)] Starting Indexer
[INFO] [10/15/2018 02:55:34.528] [main] [IndexerApp$(akka://indexer)] Log level is INFO
[INFO] [10/15/2018 02:55:39.848] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client server Url: elasticsearch://elasticsearch:9200?cluster.name=myesdb
[INFO] [10/15/2018 02:55:39.850] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client maxRetryTimeout: 30000
[INFO] [10/15/2018 02:55:40.625] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client connectTimeout: 30000
[INFO] [10/15/2018 02:55:40.708] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client socketTimeout: 600000
[INFO] [10/15/2018 02:55:43.109] [main] [IndexerApp$(akka://indexer)] Listening on 0.0.0.0:80
[INFO] [10/15/2018 02:55:48.122] [indexer-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(indexer)] Successfully made initial contact with the ES client (this doesn't mean we're fully connected yet!)
[INFO] [10/15/2018 02:56:05.376] [main] [RegisterWebhook$(akka://indexer)] Looking up existing webhook with id indexer
[INFO] [10/15/2018 02:56:06.474] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] datasets index version 36 already exists
[INFO] [10/15/2018 02:56:06.481] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] regions index version 21 already exists
[INFO] [10/15/2018 02:56:06.481] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] publishers index version 2 already exists
[INFO] [10/15/2018 02:56:06.482] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] formats index version 1 already exists
[INFO] [10/15/2018 02:56:13.679] [indexer-akka.actor.default-dispatcher-5] [RegisterWebhook$(akka://indexer)] Hook already exists, updating...
[INFO] [10/15/2018 02:56:15.080] [indexer-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(indexer)] Successfully added webhook
[INFO] [10/15/2018 02:56:15.081] [indexer-akka.actor.default-dispatcher-6] [RegisterWebhook$(akka://indexer)] Updated, attempting to resume...
[INFO] [10/15/2018 02:56:15.179] [indexer-akka.actor.default-dispatcher-5] [RegisterWebhook$(akka://indexer)] Successfully resumed webhook
[INFO] [10/15/2018 02:56:15.185] [indexer-akka.actor.default-dispatcher-5] [IndexerApp$(akka://indexer)] Checking to see if index is empty
aether:magda-config alex$ kubectl logs indexer-7579488d5c-4tn4w | head -n50
02:55:32,526 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
02:55:32,528 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-scala-common-0.0.49-RC3.jar!/logback.xml]
02:55:32,529 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml]
02:55:32,627 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@6f8e8894 - URL [jar:file:/app/lib/au.csiro.data61.magda-indexer-0.0.49-RC3.jar!/logback.xml] is not of type file
02:55:33,372 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
02:55:33,420 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
02:55:33,511 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
02:55:34,420 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.classic.AsyncAppender]
02:55:34,427 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ASYNC]
02:55:34,427 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to ch.qos.logback.classic.AsyncAppender[ASYNC]
02:55:34,427 |-INFO in ch.qos.logback.classic.AsyncAppender[ASYNC] - Attaching appender named [STDOUT] to AsyncAppender.
02:55:34,428 |-INFO in ch.qos.logback.classic.AsyncAppender[ASYNC] - Setting discardingThreshold to 51
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [ch.qos.logback] to ERROR
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.sksamuel] to ERROR
02:55:34,436 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
02:55:34,437 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ASYNC] to Logger[ROOT]
02:55:34,437 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
02:55:34,439 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@3cfdd820 - Registering current configuration as safe fallback point

[INFO] [10/15/2018 02:55:34.524] [main] [IndexerApp$(akka://indexer)] Starting Indexer
[INFO] [10/15/2018 02:55:34.528] [main] [IndexerApp$(akka://indexer)] Log level is INFO
[INFO] [10/15/2018 02:55:39.848] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client server Url: elasticsearch://elasticsearch:9200?cluster.name=myesdb
[INFO] [10/15/2018 02:55:39.850] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client maxRetryTimeout: 30000
[INFO] [10/15/2018 02:55:40.625] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client connectTimeout: 30000
[INFO] [10/15/2018 02:55:40.708] [indexer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(indexer)] Elastic Client socketTimeout: 600000
[INFO] [10/15/2018 02:55:43.109] [main] [IndexerApp$(akka://indexer)] Listening on 0.0.0.0:80
[INFO] [10/15/2018 02:55:48.122] [indexer-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(indexer)] Successfully made initial contact with the ES client (this doesn't mean we're fully connected yet!)
[INFO] [10/15/2018 02:56:05.376] [main] [RegisterWebhook$(akka://indexer)] Looking up existing webhook with id indexer
[INFO] [10/15/2018 02:56:06.474] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] datasets index version 36 already exists
[INFO] [10/15/2018 02:56:06.481] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] regions index version 21 already exists
[INFO] [10/15/2018 02:56:06.481] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] publishers index version 2 already exists
[INFO] [10/15/2018 02:56:06.482] [indexer-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(indexer)] formats index version 1 already exists
[INFO] [10/15/2018 02:56:13.679] [indexer-akka.actor.default-dispatcher-5] [RegisterWebhook$(akka://indexer)] Hook already exists, updating...
[INFO] [10/15/2018 02:56:15.080] [indexer-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(indexer)] Successfully added webhook
[INFO] [10/15/2018 02:56:15.081] [indexer-akka.actor.default-dispatcher-6] [RegisterWebhook$(akka://indexer)] Updated, attempting to resume...
[INFO] [10/15/2018 02:56:15.179] [indexer-akka.actor.default-dispatcher-5] [RegisterWebhook$(akka://indexer)] Successfully resumed webhook
[INFO] [10/15/2018 02:56:15.185] [indexer-akka.actor.default-dispatcher-5] [IndexerApp$(akka://indexer)] Checking to see if index is empty
[INFO] [10/15/2018 02:56:24.073] [indexer-akka.actor.default-dispatcher-5] [IndexerApp$(akka://indexer)] Datasets index is empty, recrawling
[INFO] [10/15/2018 02:56:24.080] [indexer-akka.actor.default-dispatcher-5] [RegistryCrawler(akka://indexer)] Crawling registry at /v0
[WARN] [10/15/2018 02:56:27.777] [indexer-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(indexer)] Failure when indexing ds-dga-559708e5-480e-4f94-8429-c49571e82761: Some(BulkError(cluster_block_exception,blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];,null,0,null))