Skip to content

Instantly share code, notes, and snippets.

@atombender

atombender/vespa.log Secret

Created October 4, 2017 15:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save atombender/f5e996207e925f9d667d53d6aeaa8978 to your computer and use it in GitHub Desktop.
Save atombender/f5e996207e925f9d667d53d6aeaa8978 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
vespa.log
1507131004.958 vespa-0.vespa.default.svc.cluster.local 283/156 configserver Container.com.yahoo.vespa.defaults.Defaults info VESPA_WEB_SERVICE_PORT not set, using 8080
1507131006.582 vespa-0.vespa.default.svc.cluster.local 283/156 configserver Container.com.yahoo.vespa.config.server.http.v2.SessionPrepareHandler info Session 2 prepared successfully.
1507131006.896 vespa-0.vespa.default.svc.cluster.local 283/156 configserver Container.com.yahoo.vespa.config.server.deploy.Deployment info Session 2 activated successfully using no host provisioner. Config generation 2
1507131008.452752 vespa-0.vespa.default.svc.cluster.local 2243/34642 config-sentinel config-sentinel.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131008.452866 vespa-0.vespa.default.svc.cluster.local 2243/34642 config-sentinel config-sentinel.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:30:58 GMT
1507131008.452895 vespa-0.vespa.default.svc.cluster.local 2243/34642 config-sentinel config-sentinel.config.frt.frtconfigagent info Error response or no response from config server (key: name=sentinel,namespace=cloud.config,configId=hosts/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 10000 milliseconds
1507131018.365224 vespa-0.vespa.default.svc.cluster.local 2243/64029 config-sentinel config-sentinel warning Timout getting config, please check your setup. Will exit and restart: Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/vespa-0.vespa.default.svc.cluster.local'
1507131018.365642 vespa-0.vespa.default.svc.cluster.local 2243/64029 config-sentinel config-sentinel event stopping/1 name="config-sentinel" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.sentinel', configid 'hosts/vespa-0.vespa.default.svc.cluster.local'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f66d29a4a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f66d29974b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f66d29a827b]\n sbin/vespa-config-sentinel(std::unique_ptr<config::ConfigHandle<cloud::config::internal::InternalSentinelType const>, std::default_delete<config::ConfigHandle<cloud::config::internal::InternalSentinelType const> > > config::ConfigSubscriber::subscribe<cloud::config::internal::InternalSentinelType const>(std::string const&, unsigned long)+0xca) [0x40bc9a]\n sbin/vespa-config-sentinel(config::sentinel::ConfigHandler::subscribe(std::string const&, unsigned long)+0x25) [0x4090a5]\n sbin/vespa-config-sentinel(main+0x145) [0x406025]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f66cfe88c05]\n sbin/vespa-config-sentinel() [0x40677d]"
1507131018.366815 vespa-0.vespa.default.svc.cluster.local 608/35910 config-sentinel runserver event stopped/1 name="sbin/vespa-config-sentinel -c hosts/vespa-0.vespa.default.svc.cluster.local (pid 2243)" pid=2243 exitcode=1
1507131018.366861 vespa-0.vespa.default.svc.cluster.local 608/35910 config-sentinel runserver info will restart in 0 seconds
1507131018.367062 vespa-0.vespa.default.svc.cluster.local 608/35910 config-sentinel runserver event starting/1 name="sbin/vespa-config-sentinel -c hosts/vespa-0.vespa.default.svc.cluster.local (pid 2301)"
1507131018.371818 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel event started/1 name="config-sentinel"
1507131018.384211 vespa-0.vespa.default.svc.cluster.local 2307/1079 config-sentinel config-sentinel.service event starting/1 name="container"
1507131018.385391 vespa-0.vespa.default.svc.cluster.local 2313/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131018.385725 vespa-0.vespa.default.svc.cluster.local 2308/1079 config-sentinel config-sentinel.service event starting/1 name="logserver"
1507131018.386405 vespa-0.vespa.default.svc.cluster.local 2309/1079 config-sentinel config-sentinel.service event starting/1 name="searchnode"
1507131018.386697 vespa-0.vespa.default.svc.cluster.local 2310/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131018.386955 vespa-0.vespa.default.svc.cluster.local 2311/1079 config-sentinel config-sentinel.service event starting/1 name="container-clustercontroller"
1507131018.387050 vespa-0.vespa.default.svc.cluster.local 2314/1079 config-sentinel config-sentinel.service event starting/1 name="slobrok"
1507131018.387212 vespa-0.vespa.default.svc.cluster.local 2312/1079 config-sentinel config-sentinel.service event starting/1 name="docprocservice"
1507131018.387367 vespa-0.vespa.default.svc.cluster.local 2315/1079 config-sentinel config-sentinel.service event starting/1 name="logd"
1507131018.387987 vespa-0.vespa.default.svc.cluster.local 2316/1079 config-sentinel config-sentinel.service event starting/1 name="filedistributorservice"
1507131018.421807 vespa-0.vespa.default.svc.cluster.local 2315/11234 logd logdemon event started/1 name="logdemon"
1507131018.495114 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.sbenv config listening on port 19099
1507131018.495226 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.sbenv event started/1 name="slobrok"
1507131018.495245 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="heartbeats_failed" value=0
1507131018.495250 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="register_reqs" value=0
1507131018.495254 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="mirror_reqs" value=0
1507131018.495258 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="wantadd_reqs" value=0
1507131018.495271 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="doadd_reqs" value=0
1507131018.495275 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="doremove_reqs" value=0
1507131018.495279 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="admin_reqs" value=0
1507131018.495283 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="other_reqs" value=0
1507131018.751917 vespa-0.vespa.default.svc.cluster.local 2316/3580 filedistributorservice filedistributor event started/1 name="filedistributor"
1507131018.905518 vespa-0.vespa.default.svc.cluster.local 2313/10914 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131019.253 vespa-0.vespa.default.svc.cluster.local 283/176 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131019.254 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131019.526 vespa-0.vespa.default.svc.cluster.local 2308/1 logserver ADM.com.yahoo.logserver.handlers.HandlerThread config logserver.queue.size=200
1507131019.550 vespa-0.vespa.default.svc.cluster.local 2308/1 logserver ADM.com.yahoo.logserver.handlers.replicator.Replicator config port=19083
1507131019.555 vespa-0.vespa.default.svc.cluster.local 2308/1 logserver ADM.com.yahoo.logserver.handlers.HandlerThread config logserver.queue.size=200
1507131019.557 vespa-0.vespa.default.svc.cluster.local 2308/1 logserver ADM.com.yahoo.logserver.handlers.HandlerThread config logserver.queue.size=200
1507131019.559 vespa-0.vespa.default.svc.cluster.local 2308/1 logserver ADM.com.yahoo.logserver.handlers.lasterrorsholder.LastErrorsHolder config port=19082
1507131019.559 vespa-0.vespa.default.svc.cluster.local 2308/1 logserver ADM.com.yahoo.logserver.handlers.HandlerThread config logserver.queue.size=200
1507131019.561 vespa-0.vespa.default.svc.cluster.local 2308/22 logserver ADM.com.yahoo.logserver.Server config logserver.listenport=19081
1507131019.563 vespa-0.vespa.default.svc.cluster.local 2308/22 logserver ADM.com.yahoo.logserver.Server event started/1 name=logserver
1507131022.582 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=container/container.0 within timeout, will retry
1507131023.460811 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131023.460859 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=0
1507131023.460869 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=0
1507131023.700 vespa-0.vespa.default.svc.cluster.local 283/180 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131023.702 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,16705,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131028.959549 vespa-0.vespa.default.svc.cluster.local 2307 container stderr warning error 103: (RPC) Invocation timed out
1507131028.980759 vespa-0.vespa.default.svc.cluster.local 2307 container stdout debug starting container for container/container.0
1507131028.980784 vespa-0.vespa.default.svc.cluster.local 2307 container stdout debug without numactl (no permission or not available)
1507131028.980788 vespa-0.vespa.default.svc.cluster.local 2307 container stdout debug numactlcmd:
1507131028.980792 vespa-0.vespa.default.svc.cluster.local 2307 container stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131029.158715 vespa-0.vespa.default.svc.cluster.local 2311 container-clustercontroller stderr warning error 103: (RPC) Invocation timed out
1507131029.158745 vespa-0.vespa.default.svc.cluster.local 2312 docprocservice stderr warning error 103: (RPC) Invocation timed out
1507131029.186862 vespa-0.vespa.default.svc.cluster.local 2311 container-clustercontroller stdout debug starting container-clustercontroller for admin/cluster-controllers/0
1507131029.186883 vespa-0.vespa.default.svc.cluster.local 2311 container-clustercontroller stdout debug without numactl (no permission or not available)
1507131029.186889 vespa-0.vespa.default.svc.cluster.local 2311 container-clustercontroller stdout debug numactlcmd:
1507131029.186893 vespa-0.vespa.default.svc.cluster.local 2311 container-clustercontroller stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131029.186909 vespa-0.vespa.default.svc.cluster.local 2312 docprocservice stdout debug starting docprocservice for docproc/cluster.music.indexing/0
1507131029.201062 vespa-0.vespa.default.svc.cluster.local 2312 docprocservice stdout debug without numactl (no permission or not available)
1507131029.201081 vespa-0.vespa.default.svc.cluster.local 2312 docprocservice stdout debug numactlcmd:
1507131029.201087 vespa-0.vespa.default.svc.cluster.local 2312 docprocservice stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131030.71328 vespa-0.vespa.default.svc.cluster.local 2307 container stdout debug Initializing application without privileges.
1507131030.71374 vespa-0.vespa.default.svc.cluster.local 2312 docprocservice stdout debug Initializing application without privileges.
1507131030.205544 vespa-0.vespa.default.svc.cluster.local 2311 container-clustercontroller stdout debug Initializing application without privileges.
1507131030.426 vespa-0.vespa.default.svc.cluster.local 283/183 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131030.588 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,17393,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131038.105 vespa-0.vespa.default.svc.cluster.local 283/184 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131038.108 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,18541,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131038.890491 vespa-0.vespa.default.svc.cluster.local 2316/31472 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131038.894380 vespa-0.vespa.default.svc.cluster.local 2316/31472 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:30:48 GMT
1507131038.895843 vespa-0.vespa.default.svc.cluster.local 2316/31472 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=filedistributor,namespace=cloud.config.filedistribution,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131038.929074 vespa-0.vespa.default.svc.cluster.local 2313/38806 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131038.929942 vespa-0.vespa.default.svc.cluster.local 2313/38806 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:30:48 GMT
1507131038.930048 vespa-0.vespa.default.svc.cluster.local 2313/38806 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131039.161204 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131039.161381 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:30:49 GMT
1507131039.161447 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=documenttypes,namespace=document,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131039.269554 vespa-0.vespa.default.svc.cluster.local 2310/60208 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131039.269812 vespa-0.vespa.default.svc.cluster.local 2310/60208 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:30:49 GMT
1507131039.269911 vespa-0.vespa.default.svc.cluster.local 2310/60208 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131044.212 vespa-0.vespa.default.svc.cluster.local 283/185 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131044.214 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,20837,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131052.739 vespa-0.vespa.default.svc.cluster.local 283/186 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131052.740 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,21420,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131060.000 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.requests value=17
1507131060.002 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.failedRequests value=6
1507131060.002 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=procTime value=168
1507131060.147 vespa-0.vespa.default.svc.cluster.local 283/187 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131060.149 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,17936,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131063.253 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=container/container.0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131063.373 vespa-0.vespa.default.svc.cluster.local 283/175 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131063.374 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131066.582 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0 within timeout, will retry
1507131068.812 vespa-0.vespa.default.svc.cluster.local 283/179 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131068.816 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,23379,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131071.727 vespa-0.vespa.default.svc.cluster.local 283/180 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131073.937237 vespa-0.vespa.default.svc.cluster.local 2313/10914 topleveldispatch fdispatch error getting config: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.search.core.fdispatchrc', configid 'music/search/cluster.music/tlds/container.0.tld.0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7ff3dca44a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7ff3dca374b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7ff3dca4827b]\n /opt/vespa/sbin/vespa-dispatch-bin(std::unique_ptr<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const>, std::default_delete<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const> > > config::ConfigSubscriber::subscribe<vespa::config::search::core::internal::InternalFdispatchrcType const>(std::string const&, unsigned long)+0xca) [0x41f22a]\n /opt/vespa/sbin/vespa-dispatch-bin(fdispatch::Fdispatch::Fdispatch(config::ConfigUri const&)+0x270) [0x41e580]\n /opt/vespa/sbin/vespa-dispatch-bin(FastS_FDispatchApp::Main()+0xa9) [0x41c6e9]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7ff3dbc670f1]\n /opt/vespa/sbin/vespa-dispatch-bin(main+0x45) [0x41ad55]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff3d79e9c05]\n /opt/vespa/sbin/vespa-dispatch-bin() [0x41c219]
1507131073.937292 vespa-0.vespa.default.svc.cluster.local 2313/10914 topleveldispatch fdispatch event stopping/1 name="fdispatch" why="error getting config"
1507131074.005339 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="topleveldispatch" pid=2313 exitcode=1
1507131074.005839 vespa-0.vespa.default.svc.cluster.local 2866/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131074.162 vespa-0.vespa.default.svc.cluster.local 283/181 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131074.163 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,20596,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131074.245148 vespa-0.vespa.default.svc.cluster.local 2866/16887 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131074.265561 vespa-0.vespa.default.svc.cluster.local 2310/32316 distributor vds.application error Caught exception when starting: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.content.core.stor-server', configid 'music/distributor/0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f645c922a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f645c9154b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f645c92627b]\n /opt/vespa/sbin/vespa-distributord-bin(std::unique_ptr<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const>, std::default_delete<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const> > > config::ConfigSubscriber::subscribe<vespa::config::content::core::internal::InternalStorServerType const>(std::string const&, unsigned long)+0xca) [0x40cfea]\n /opt/vespa/sbin/vespa-distributord-bin(config::ConfigGetter<vespa::config::content::core::internal::InternalStorServerType const>::getConfig(long&, std::string const&, std::shared_ptr<config::IConfigContext> const&, unsigned long)+0x48) [0x40d128]\n /opt/vespa/sbin/vespa-distributord-bin(storage::StorageApp::Main()+0xee) [0x40c31e]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f645beba0f1]\n /opt/vespa/sbin/vespa-distributord-bin(main+0x3b) [0x40b64b]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f645a3dec05]\n /opt/vespa/sbin/vespa-distributord-bin() [0x40b7dd]
1507131074.296829 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="distributor" pid=2310 exitcode=1
1507131074.297161 vespa-0.vespa.default.svc.cluster.local 2904/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131075.164796 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131075.164847 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:31:35 GMT
1507131075.164865 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=documenttypes,namespace=document,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 7000 milliseconds
1507131080.786 vespa-0.vespa.default.svc.cluster.local 283/182 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131080.787 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,22832,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131083.301472 vespa-0.vespa.default.svc.cluster.local 2316/31472 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131083.301548 vespa-0.vespa.default.svc.cluster.local 2316/31472 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:31:43 GMT
1507131083.301567 vespa-0.vespa.default.svc.cluster.local 2316/31472 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=filedistributorrpc,namespace=cloud.config.filedistribution,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131083.427342 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131083.427391 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=2
1507131083.427398 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=2
1507131087.745 vespa-0.vespa.default.svc.cluster.local 283/183 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131087.746 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,17305,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131089.755 vespa-0.vespa.default.svc.cluster.local 2311/1 container-clustercontroller Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131089.863 vespa-0.vespa.default.svc.cluster.local 2311/1 container-clustercontroller Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131089.755 vespa-0.vespa.default.svc.cluster.local 2312/1 docprocservice Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131089.894 vespa-0.vespa.default.svc.cluster.local 2312/1 docprocservice Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131089.992828 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container-clustercontroller" pid=2311 exitcode=6
1507131089.993133 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="docprocservice" pid=2312 exitcode=6
1507131089.993215 vespa-0.vespa.default.svc.cluster.local 2945/1079 config-sentinel config-sentinel.service event starting/1 name="container-clustercontroller"
1507131089.993667 vespa-0.vespa.default.svc.cluster.local 2946/1079 config-sentinel config-sentinel.service event starting/1 name="docprocservice"
1507131089.981 vespa-0.vespa.default.svc.cluster.local 2307/1 container Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131090.001 vespa-0.vespa.default.svc.cluster.local 2307/1 container Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131090.100242 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container" pid=2307 exitcode=6
1507131090.100614 vespa-0.vespa.default.svc.cluster.local 3083/1079 config-sentinel config-sentinel.service event starting/1 name="container"
1507131094.260159 vespa-0.vespa.default.svc.cluster.local 2866/44779 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131094.260851 vespa-0.vespa.default.svc.cluster.local 2866/44779 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:31:44 GMT
1507131094.260943 vespa-0.vespa.default.svc.cluster.local 2866/44779 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131094.397184 vespa-0.vespa.default.svc.cluster.local 2904/14200 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131094.397425 vespa-0.vespa.default.svc.cluster.local 2904/14200 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:31:44 GMT
1507131094.397536 vespa-0.vespa.default.svc.cluster.local 2904/14200 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131097.193 vespa-0.vespa.default.svc.cluster.local 283/184 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131097.194 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,22521,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131100.124855 vespa-0.vespa.default.svc.cluster.local 2945 container-clustercontroller stderr warning error 103: (RPC) Invocation timed out
1507131100.124882 vespa-0.vespa.default.svc.cluster.local 2946 docprocservice stderr warning error 103: (RPC) Invocation timed out
1507131100.142246 vespa-0.vespa.default.svc.cluster.local 2945 container-clustercontroller stdout debug starting container-clustercontroller for admin/cluster-controllers/0
1507131100.142273 vespa-0.vespa.default.svc.cluster.local 2945 container-clustercontroller stdout debug without numactl (no permission or not available)
1507131100.142278 vespa-0.vespa.default.svc.cluster.local 2945 container-clustercontroller stdout debug numactlcmd:
1507131100.142282 vespa-0.vespa.default.svc.cluster.local 2945 container-clustercontroller stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131100.142289 vespa-0.vespa.default.svc.cluster.local 2946 docprocservice stdout debug starting docprocservice for docproc/cluster.music.indexing/0
1507131100.142294 vespa-0.vespa.default.svc.cluster.local 2946 docprocservice stdout debug without numactl (no permission or not available)
1507131100.142298 vespa-0.vespa.default.svc.cluster.local 2946 docprocservice stdout debug numactlcmd:
1507131100.142307 vespa-0.vespa.default.svc.cluster.local 2946 docprocservice stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131100.179223 vespa-0.vespa.default.svc.cluster.local 3083 container stdout debug starting container for container/container.0
1507131100.179246 vespa-0.vespa.default.svc.cluster.local 3083 container stdout debug without numactl (no permission or not available)
1507131100.179250 vespa-0.vespa.default.svc.cluster.local 3083 container stdout debug numactlcmd:
1507131100.179254 vespa-0.vespa.default.svc.cluster.local 3083 container stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131100.179260 vespa-0.vespa.default.svc.cluster.local 3083 container stderr warning error 103: (RPC) Invocation timed out
1507131101.226171 vespa-0.vespa.default.svc.cluster.local 2945 container-clustercontroller stdout debug Initializing application without privileges.
1507131101.226202 vespa-0.vespa.default.svc.cluster.local 3083 container stdout debug Initializing application without privileges.
1507131101.289662 vespa-0.vespa.default.svc.cluster.local 2946 docprocservice stdout debug Initializing application without privileges.
1507131105.892 vespa-0.vespa.default.svc.cluster.local 283/185 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131105.893 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,23225,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131107.377 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131107.392 vespa-0.vespa.default.svc.cluster.local 283/186 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131107.393 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131110.582 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0 within timeout, will retry
1507131112.184338 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131112.184391 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:32:22 GMT
1507131112.184417 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=documenttypes,namespace=document,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 8000 milliseconds
1507131115.655 vespa-0.vespa.default.svc.cluster.local 283/187 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131118.305021 vespa-0.vespa.default.svc.cluster.local 2316/3580 filedistributorservice filedistributor event stopping/1 name="filedistributor" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.filedistribution.filedistributorrpc', configid 'filedistribution/vespa-0.vespa.default.svc.cluster.local'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f9cd2e12a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f9cd2e054b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f9cd2e1627b]\n /opt/vespa/sbin/vespa-filedistributor-bin(std::unique_ptr<config::ConfigHandle<cloud::config::filedistribution::internal::InternalFiledistributorrpcType const>, std::default_delete<config::ConfigHandle<cloud::config::filedistribution::internal::InternalFiledistributorrpcType const> > > config::ConfigSubscriber::subscribe<cloud::config::filedistribution::internal::InternalFiledistributorrpcType const>(std::string const&, unsigned long)+0xca) [0x43457a]\n /opt/vespa/sbin/vespa-filedistributor-bin(void config::ConfigPoller::subscribe<cloud::config::filedistribution::internal::InternalFiledistributorrpcType const>(std::string const&, config::IFetcherCallback<cloud::config::filedistribution::internal::InternalFiledistributorrpcType const>*, unsigned long)+0x27) [0x435047]\n /opt/vespa/sbin/vespa-filedistributor-bin(FileDistributorApplication::Main()+0x157) [0x42f5a7]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f9cd23aa0f1]\n /opt/vespa/sbin/vespa-filedistributor-bin(executeApplication(int, char**)+0xa0e) [0x42f12e]\n /opt/vespa/sbin/vespa-filedistributor-bin(main+0x50) [0x42cd20]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9cd0d26c05]\n /opt/vespa/sbin/vespa-filedistributor-bin() [0x42e021]"
1507131118.320051 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="filedistributorservice" pid=2316 exitcode=4
1507131118.320393 vespa-0.vespa.default.svc.cluster.local 3271/1079 config-sentinel config-sentinel.service event starting/1 name="filedistributorservice"
1507131118.406546 vespa-0.vespa.default.svc.cluster.local 3271/47134 filedistributorservice filedistributor event started/1 name="filedistributor"
1507131119.220 vespa-0.vespa.default.svc.cluster.local 283/188 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131119.221 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,20043,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131120.000 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=procTime value=240
1507131120.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.requests value=35
1507131120.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.failedRequests value=18
1507131129.283656 vespa-0.vespa.default.svc.cluster.local 2866/16887 topleveldispatch fdispatch error getting config: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.search.core.fdispatchrc', configid 'music/search/cluster.music/tlds/container.0.tld.0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f785474ea65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f78547414b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f785475227b]\n /opt/vespa/sbin/vespa-dispatch-bin(std::unique_ptr<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const>, std::default_delete<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const> > > config::ConfigSubscriber::subscribe<vespa::config::search::core::internal::InternalFdispatchrcType const>(std::string const&, unsigned long)+0xca) [0x41f22a]\n /opt/vespa/sbin/vespa-dispatch-bin(fdispatch::Fdispatch::Fdispatch(config::ConfigUri const&)+0x270) [0x41e580]\n /opt/vespa/sbin/vespa-dispatch-bin(FastS_FDispatchApp::Main()+0xa9) [0x41c6e9]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f78539710f1]\n /opt/vespa/sbin/vespa-dispatch-bin(main+0x45) [0x41ad55]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f784f6f3c05]\n /opt/vespa/sbin/vespa-dispatch-bin() [0x41c219]
1507131129.283707 vespa-0.vespa.default.svc.cluster.local 2866/16887 topleveldispatch fdispatch event stopping/1 name="fdispatch" why="error getting config"
1507131129.304762 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="topleveldispatch" pid=2866 exitcode=1
1507131129.305099 vespa-0.vespa.default.svc.cluster.local 3308/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131129.401507 vespa-0.vespa.default.svc.cluster.local 2904/51843 distributor vds.application error Caught exception when starting: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.content.core.stor-server', configid 'music/distributor/0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7fc8a8752a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7fc8a87454b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7fc8a875627b]\n /opt/vespa/sbin/vespa-distributord-bin(std::unique_ptr<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const>, std::default_delete<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const> > > config::ConfigSubscriber::subscribe<vespa::config::content::core::internal::InternalStorServerType const>(std::string const&, unsigned long)+0xca) [0x40cfea]\n /opt/vespa/sbin/vespa-distributord-bin(config::ConfigGetter<vespa::config::content::core::internal::InternalStorServerType const>::getConfig(long&, std::string const&, std::shared_ptr<config::IConfigContext> const&, unsigned long)+0x48) [0x40d128]\n /opt/vespa/sbin/vespa-distributord-bin(storage::StorageApp::Main()+0xee) [0x40c31e]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7fc8a7cea0f1]\n /opt/vespa/sbin/vespa-distributord-bin(main+0x3b) [0x40b64b]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc8a620ec05]\n /opt/vespa/sbin/vespa-distributord-bin() [0x40b7dd]
1507131129.423586 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="distributor" pid=2904 exitcode=1
1507131129.424050 vespa-0.vespa.default.svc.cluster.local 3339/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131129.455682 vespa-0.vespa.default.svc.cluster.local 3308/65156 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131132.507 vespa-0.vespa.default.svc.cluster.local 283/189 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131132.508 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,22094,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131138.429163 vespa-0.vespa.default.svc.cluster.local 3271/9491 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131138.451799 vespa-0.vespa.default.svc.cluster.local 3271/9491 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:32:28 GMT
1507131138.451863 vespa-0.vespa.default.svc.cluster.local 3271/9491 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=zookeepers,namespace=cloud.config,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131143.631298 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131143.631338 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=6
1507131143.631363 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=8
1507131147.020 vespa-0.vespa.default.svc.cluster.local 283/190 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131147.020 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,20281,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131149.467345 vespa-0.vespa.default.svc.cluster.local 3308/27513 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131149.467481 vespa-0.vespa.default.svc.cluster.local 3308/27513 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:32:39 GMT
1507131149.467522 vespa-0.vespa.default.svc.cluster.local 3308/27513 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131149.581895 vespa-0.vespa.default.svc.cluster.local 3339/5120 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131149.582043 vespa-0.vespa.default.svc.cluster.local 3339/5120 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:32:39 GMT
1507131149.582123 vespa-0.vespa.default.svc.cluster.local 3339/5120 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131150.206998 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131150.207072 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:33:10 GMT
1507131150.207089 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=documenttypes,namespace=document,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 9000 milliseconds
1507131151.402 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131151.542 vespa-0.vespa.default.svc.cluster.local 283/179 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131151.543 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131154.582 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0 within timeout, will retry
1507131155.587 vespa-0.vespa.default.svc.cluster.local 283/180 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131155.588 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,22289,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131159.262 vespa-0.vespa.default.svc.cluster.local 283/185 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131159.404 vespa-0.vespa.default.svc.cluster.local 2946/1 docprocservice Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131159.446 vespa-0.vespa.default.svc.cluster.local 2946/1 docprocservice Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131159.456 vespa-0.vespa.default.svc.cluster.local 2945/1 container-clustercontroller Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131159.473 vespa-0.vespa.default.svc.cluster.local 2945/1 container-clustercontroller Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131159.502262 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="docprocservice" pid=2946 exitcode=6
1507131159.502655 vespa-0.vespa.default.svc.cluster.local 3391/1079 config-sentinel config-sentinel.service event starting/1 name="docprocservice"
1507131159.488 vespa-0.vespa.default.svc.cluster.local 3083/1 container Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131159.504 vespa-0.vespa.default.svc.cluster.local 3083/1 container Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131159.551306 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container-clustercontroller" pid=2945 exitcode=6
1507131159.551707 vespa-0.vespa.default.svc.cluster.local 3451/1079 config-sentinel config-sentinel.service event starting/1 name="container-clustercontroller"
1507131159.595593 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container" pid=3083 exitcode=6
1507131159.596346 vespa-0.vespa.default.svc.cluster.local 3518/1079 config-sentinel config-sentinel.service event starting/1 name="container"
1507131160.395 vespa-0.vespa.default.svc.cluster.local 283/186 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131160.396 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,21772,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131168.069 vespa-0.vespa.default.svc.cluster.local 283/187 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131168.070 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,21438,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131169.606073 vespa-0.vespa.default.svc.cluster.local 3391 docprocservice stdout debug starting docprocservice for docproc/cluster.music.indexing/0
1507131169.606102 vespa-0.vespa.default.svc.cluster.local 3391 docprocservice stdout debug without numactl (no permission or not available)
1507131169.606107 vespa-0.vespa.default.svc.cluster.local 3391 docprocservice stdout debug numactlcmd:
1507131169.606110 vespa-0.vespa.default.svc.cluster.local 3391 docprocservice stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131169.606116 vespa-0.vespa.default.svc.cluster.local 3391 docprocservice stderr warning error 103: (RPC) Invocation timed out
1507131169.645926 vespa-0.vespa.default.svc.cluster.local 3451 container-clustercontroller stderr warning error 103: (RPC) Invocation timed out
1507131169.687956 vespa-0.vespa.default.svc.cluster.local 3451 container-clustercontroller stdout debug starting container-clustercontroller for admin/cluster-controllers/0
1507131169.687978 vespa-0.vespa.default.svc.cluster.local 3451 container-clustercontroller stdout debug without numactl (no permission or not available)
1507131169.687983 vespa-0.vespa.default.svc.cluster.local 3451 container-clustercontroller stdout debug numactlcmd:
1507131169.687987 vespa-0.vespa.default.svc.cluster.local 3451 container-clustercontroller stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131169.687994 vespa-0.vespa.default.svc.cluster.local 3518 container stderr warning error 103: (RPC) Invocation timed out
1507131169.722733 vespa-0.vespa.default.svc.cluster.local 3518 container stdout debug starting container for container/container.0
1507131169.722790 vespa-0.vespa.default.svc.cluster.local 3518 container stdout debug without numactl (no permission or not available)
1507131169.722796 vespa-0.vespa.default.svc.cluster.local 3518 container stdout debug numactlcmd:
1507131169.722801 vespa-0.vespa.default.svc.cluster.local 3518 container stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131170.740017 vespa-0.vespa.default.svc.cluster.local 3451 container-clustercontroller stdout debug Initializing application without privileges.
1507131170.788396 vespa-0.vespa.default.svc.cluster.local 3391 docprocservice stdout debug Initializing application without privileges.
1507131170.788428 vespa-0.vespa.default.svc.cluster.local 3518 container stdout debug Initializing application without privileges.
1507131173.831696 vespa-0.vespa.default.svc.cluster.local 3271/47134 filedistributorservice filedistributor event stopping/1 name="filedistributor" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.zookeepers', configid 'filedistribution/vespa-0.vespa.default.svc.cluster.local'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f119d8f0a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f119d8e34b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f119d8f427b]\n /opt/vespa/sbin/vespa-filedistributor-bin(std::unique_ptr<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const>, std::default_delete<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const> > > config::ConfigSubscriber::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, unsigned long)+0xca) [0x4341fa]\n /opt/vespa/sbin/vespa-filedistributor-bin(void config::ConfigPoller::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, config::IFetcherCallback<cloud::config::internal::InternalZookeepersType const>*, unsigned long)+0x27) [0x435187]\n /opt/vespa/sbin/vespa-filedistributor-bin(FileDistributorApplication::Main()+0x9a) [0x42f4ea]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f119ce880f1]\n /opt/vespa/sbin/vespa-filedistributor-bin(executeApplication(int, char**)+0xa0e) [0x42f12e]\n /opt/vespa/sbin/vespa-filedistributor-bin(main+0x50) [0x42cd20]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f119b804c05]\n /opt/vespa/sbin/vespa-filedistributor-bin() [0x42e021]"
1507131173.857282 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="filedistributorservice" pid=3271 exitcode=4
1507131173.862649 vespa-0.vespa.default.svc.cluster.local 3714/1079 config-sentinel config-sentinel.service event starting/1 name="filedistributorservice"
1507131174.139339 vespa-0.vespa.default.svc.cluster.local 3714/30763 filedistributorservice filedistributor event started/1 name="filedistributor"
1507131175.768 vespa-0.vespa.default.svc.cluster.local 283/188 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131175.772 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,23143,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131180.000 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.failedRequests value=26
1507131180.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=procTime value=347
1507131180.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.requests value=51
1507131182.491 vespa-0.vespa.default.svc.cluster.local 283/189 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131182.493 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,23393,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131184.474399 vespa-0.vespa.default.svc.cluster.local 3308/65156 topleveldispatch fdispatch error getting config: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.search.core.fdispatchrc', configid 'music/search/cluster.music/tlds/container.0.tld.0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7ff370878a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7ff37086b4b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7ff37087c27b]\n /opt/vespa/sbin/vespa-dispatch-bin(std::unique_ptr<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const>, std::default_delete<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const> > > config::ConfigSubscriber::subscribe<vespa::config::search::core::internal::InternalFdispatchrcType const>(std::string const&, unsigned long)+0xca) [0x41f22a]\n /opt/vespa/sbin/vespa-dispatch-bin(fdispatch::Fdispatch::Fdispatch(config::ConfigUri const&)+0x270) [0x41e580]\n /opt/vespa/sbin/vespa-dispatch-bin(FastS_FDispatchApp::Main()+0xa9) [0x41c6e9]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7ff36fa9b0f1]\n /opt/vespa/sbin/vespa-dispatch-bin(main+0x45) [0x41ad55]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff36b81dc05]\n /opt/vespa/sbin/vespa-dispatch-bin() [0x41c219]
1507131184.474442 vespa-0.vespa.default.svc.cluster.local 3308/65156 topleveldispatch fdispatch event stopping/1 name="fdispatch" why="error getting config"
1507131184.493648 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="topleveldispatch" pid=3308 exitcode=1
1507131184.494250 vespa-0.vespa.default.svc.cluster.local 3756/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131184.570138 vespa-0.vespa.default.svc.cluster.local 3339/42763 distributor vds.application error Caught exception when starting: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.content.core.stor-server', configid 'music/distributor/0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7fb8b50cfa65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7fb8b50c24b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7fb8b50d327b]\n /opt/vespa/sbin/vespa-distributord-bin(std::unique_ptr<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const>, std::default_delete<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const> > > config::ConfigSubscriber::subscribe<vespa::config::content::core::internal::InternalStorServerType const>(std::string const&, unsigned long)+0xca) [0x40cfea]\n /opt/vespa/sbin/vespa-distributord-bin(config::ConfigGetter<vespa::config::content::core::internal::InternalStorServerType const>::getConfig(long&, std::string const&, std::shared_ptr<config::IConfigContext> const&, unsigned long)+0x48) [0x40d128]\n /opt/vespa/sbin/vespa-distributord-bin(storage::StorageApp::Main()+0xee) [0x40c31e]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7fb8b46670f1]\n /opt/vespa/sbin/vespa-distributord-bin(main+0x3b) [0x40b64b]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fb8b2b8bc05]\n /opt/vespa/sbin/vespa-distributord-bin() [0x40b7dd]
1507131184.584252 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="distributor" pid=3339 exitcode=1
1507131184.584571 vespa-0.vespa.default.svc.cluster.local 3787/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131184.601366 vespa-0.vespa.default.svc.cluster.local 3756/61507 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131189.227317 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131189.227410 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:33:59 GMT
1507131189.227428 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=documenttypes,namespace=document,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 10000 milliseconds
1507131192.435 vespa-0.vespa.default.svc.cluster.local 283/190 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131192.436 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,17635,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131194.155623 vespa-0.vespa.default.svc.cluster.local 3714/58655 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131194.156259 vespa-0.vespa.default.svc.cluster.local 3714/58655 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:33:24 GMT
1507131194.156302 vespa-0.vespa.default.svc.cluster.local 3714/58655 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=zookeepers,namespace=cloud.config,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131195.548 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131195.550 vespa-0.vespa.default.svc.cluster.local 283/175 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131195.551 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131198.582 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0 within timeout, will retry
1507131203.749 vespa-0.vespa.default.svc.cluster.local 283/176 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131203.871619 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131203.871657 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=6
1507131203.871663 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=14
1507131204.616847 vespa-0.vespa.default.svc.cluster.local 3756/23864 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131204.616962 vespa-0.vespa.default.svc.cluster.local 3756/23864 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:33:34 GMT
1507131204.616997 vespa-0.vespa.default.svc.cluster.local 3756/23864 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131204.709682 vespa-0.vespa.default.svc.cluster.local 3787/8147 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131204.709822 vespa-0.vespa.default.svc.cluster.local 3787/8147 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:33:34 GMT
1507131204.709865 vespa-0.vespa.default.svc.cluster.local 3787/8147 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131205.346 vespa-0.vespa.default.svc.cluster.local 283/177 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131205.347 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,21677,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131215.777 vespa-0.vespa.default.svc.cluster.local 283/178 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131215.778 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,19439,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131217.491 vespa-0.vespa.default.svc.cluster.local 543/11 configproxy configproxy.com.yahoo.vespa.config.proxy.ConfigProxyStatistics event count/1 name=rpc_requests value=1204
1507131217.491 vespa-0.vespa.default.svc.cluster.local 543/11 configproxy configproxy.com.yahoo.vespa.config.proxy.ConfigProxyStatistics event count/1 name=processed_messages value=1204
1507131217.491 vespa-0.vespa.default.svc.cluster.local 543/11 configproxy configproxy.com.yahoo.vespa.config.proxy.ConfigProxyStatistics event count/1 name=errors value=0
1507131217.491 vespa-0.vespa.default.svc.cluster.local 543/11 configproxy configproxy.com.yahoo.vespa.config.proxy.ConfigProxyStatistics event value/1 name=delayed_responses value=4.0
1507131227.331 vespa-0.vespa.default.svc.cluster.local 283/179 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131227.337 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,23082,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131228.666 vespa-0.vespa.default.svc.cluster.local 3518/1 container Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131228.736 vespa-0.vespa.default.svc.cluster.local 3518/1 container Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131228.821 vespa-0.vespa.default.svc.cluster.local 3391/1 docprocservice Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131228.837 vespa-0.vespa.default.svc.cluster.local 3391/1 docprocservice Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131228.850356 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container" pid=3518 exitcode=6
1507131228.850757 vespa-0.vespa.default.svc.cluster.local 3834/1079 config-sentinel config-sentinel.service event starting/1 name="container"
1507131228.928624 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="docprocservice" pid=3391 exitcode=6
1507131228.929009 vespa-0.vespa.default.svc.cluster.local 3905/1079 config-sentinel config-sentinel.service event starting/1 name="docprocservice"
1507131229.023 vespa-0.vespa.default.svc.cluster.local 3451/1 container-clustercontroller Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131229.043 vespa-0.vespa.default.svc.cluster.local 3451/1 container-clustercontroller Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131229.143753 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container-clustercontroller" pid=3451 exitcode=6
1507131229.144133 vespa-0.vespa.default.svc.cluster.local 3978/1079 config-sentinel config-sentinel.service event starting/1 name="container-clustercontroller"
1507131229.148832 vespa-0.vespa.default.svc.cluster.local 3714/30763 filedistributorservice filedistributor event stopping/1 name="filedistributor" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.zookeepers', configid 'filedistribution/vespa-0.vespa.default.svc.cluster.local'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f474d661a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f474d6544b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f474d66527b]\n /opt/vespa/sbin/vespa-filedistributor-bin(std::unique_ptr<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const>, std::default_delete<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const> > > config::ConfigSubscriber::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, unsigned long)+0xca) [0x4341fa]\n /opt/vespa/sbin/vespa-filedistributor-bin(void config::ConfigPoller::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, config::IFetcherCallback<cloud::config::internal::InternalZookeepersType const>*, unsigned long)+0x27) [0x435187]\n /opt/vespa/sbin/vespa-filedistributor-bin(FileDistributorApplication::Main()+0x9a) [0x42f4ea]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f474cbf90f1]\n /opt/vespa/sbin/vespa-filedistributor-bin(executeApplication(int, char**)+0xa0e) [0x42f12e]\n /opt/vespa/sbin/vespa-filedistributor-bin(main+0x50) [0x42cd20]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f474b575c05]\n /opt/vespa/sbin/vespa-filedistributor-bin() [0x42e021]"
1507131229.164921 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="filedistributorservice" pid=3714 exitcode=4
1507131229.165275 vespa-0.vespa.default.svc.cluster.local 4010/1079 config-sentinel config-sentinel.service event starting/1 name="filedistributorservice"
1507131229.221684 vespa-0.vespa.default.svc.cluster.local 4010/6569 filedistributorservice filedistributor event started/1 name="filedistributor"
1507131229.250985 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131229.251166 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:34:49 GMT
1507131229.251209 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=documenttypes,namespace=document,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 11000 milliseconds
1507131238.070 vespa-0.vespa.default.svc.cluster.local 283/180 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131238.071 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,17471,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131238.944996 vespa-0.vespa.default.svc.cluster.local 3834 container stderr warning error 103: (RPC) Invocation timed out
1507131238.960158 vespa-0.vespa.default.svc.cluster.local 3834 container stdout debug starting container for container/container.0
1507131238.960193 vespa-0.vespa.default.svc.cluster.local 3834 container stdout debug without numactl (no permission or not available)
1507131238.960198 vespa-0.vespa.default.svc.cluster.local 3834 container stdout debug numactlcmd:
1507131238.960201 vespa-0.vespa.default.svc.cluster.local 3834 container stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131239.16148 vespa-0.vespa.default.svc.cluster.local 3905 docprocservice stdout debug starting docprocservice for docproc/cluster.music.indexing/0
1507131239.16175 vespa-0.vespa.default.svc.cluster.local 3905 docprocservice stderr warning error 103: (RPC) Invocation timed out
1507131239.30149 vespa-0.vespa.default.svc.cluster.local 3905 docprocservice stdout debug without numactl (no permission or not available)
1507131239.30169 vespa-0.vespa.default.svc.cluster.local 3905 docprocservice stdout debug numactlcmd:
1507131239.30173 vespa-0.vespa.default.svc.cluster.local 3905 docprocservice stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131239.250144 vespa-0.vespa.default.svc.cluster.local 3978 container-clustercontroller stderr warning error 103: (RPC) Invocation timed out
1507131239.265091 vespa-0.vespa.default.svc.cluster.local 3978 container-clustercontroller stdout debug starting container-clustercontroller for admin/cluster-controllers/0
1507131239.265125 vespa-0.vespa.default.svc.cluster.local 3978 container-clustercontroller stdout debug without numactl (no permission or not available)
1507131239.265130 vespa-0.vespa.default.svc.cluster.local 3978 container-clustercontroller stdout debug numactlcmd:
1507131239.265134 vespa-0.vespa.default.svc.cluster.local 3978 container-clustercontroller stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131239.556 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131239.558 vespa-0.vespa.default.svc.cluster.local 283/181 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131239.559 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131239.604437 vespa-0.vespa.default.svc.cluster.local 3756/61507 topleveldispatch fdispatch error getting config: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.search.core.fdispatchrc', configid 'music/search/cluster.music/tlds/container.0.tld.0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7fce726b7a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7fce726aa4b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7fce726bb27b]\n /opt/vespa/sbin/vespa-dispatch-bin(std::unique_ptr<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const>, std::default_delete<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const> > > config::ConfigSubscriber::subscribe<vespa::config::search::core::internal::InternalFdispatchrcType const>(std::string const&, unsigned long)+0xca) [0x41f22a]\n /opt/vespa/sbin/vespa-dispatch-bin(fdispatch::Fdispatch::Fdispatch(config::ConfigUri const&)+0x270) [0x41e580]\n /opt/vespa/sbin/vespa-dispatch-bin(FastS_FDispatchApp::Main()+0xa9) [0x41c6e9]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7fce718da0f1]\n /opt/vespa/sbin/vespa-dispatch-bin(main+0x45) [0x41ad55]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fce6d65cc05]\n /opt/vespa/sbin/vespa-dispatch-bin() [0x41c219]
1507131239.604490 vespa-0.vespa.default.svc.cluster.local 3756/61507 topleveldispatch fdispatch event stopping/1 name="fdispatch" why="error getting config"
1507131239.608832 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="topleveldispatch" pid=3756 exitcode=1
1507131239.617436 vespa-0.vespa.default.svc.cluster.local 4152/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131239.707179 vespa-0.vespa.default.svc.cluster.local 3787/45790 distributor vds.application error Caught exception when starting: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.content.core.stor-server', configid 'music/distributor/0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f4d314b8a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f4d314ab4b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f4d314bc27b]\n /opt/vespa/sbin/vespa-distributord-bin(std::unique_ptr<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const>, std::default_delete<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const> > > config::ConfigSubscriber::subscribe<vespa::config::content::core::internal::InternalStorServerType const>(std::string const&, unsigned long)+0xca) [0x40cfea]\n /opt/vespa/sbin/vespa-distributord-bin(config::ConfigGetter<vespa::config::content::core::internal::InternalStorServerType const>::getConfig(long&, std::string const&, std::shared_ptr<config::IConfigContext> const&, unsigned long)+0x48) [0x40d128]\n /opt/vespa/sbin/vespa-distributord-bin(storage::StorageApp::Main()+0xee) [0x40c31e]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f4d30a500f1]\n /opt/vespa/sbin/vespa-distributord-bin(main+0x3b) [0x40b64b]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4d2ef74c05]\n /opt/vespa/sbin/vespa-distributord-bin() [0x40b7dd]
1507131239.712487 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="distributor" pid=3787 exitcode=1
1507131239.713316 vespa-0.vespa.default.svc.cluster.local 4180/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131239.805091 vespa-0.vespa.default.svc.cluster.local 4152/15730 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131240.000 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.requests value=60
1507131240.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.failedRequests value=35
1507131240.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=procTime value=347
1507131240.46080 vespa-0.vespa.default.svc.cluster.local 3834 container stdout debug Initializing application without privileges.
1507131240.46115 vespa-0.vespa.default.svc.cluster.local 3905 docprocservice stdout debug Initializing application without privileges.
1507131240.189349 vespa-0.vespa.default.svc.cluster.local 3978 container-clustercontroller stdout debug Initializing application without privileges.
1507131242.582 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=container/container.0 within timeout, will retry
1507131249.242902 vespa-0.vespa.default.svc.cluster.local 4010/34461 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131249.243369 vespa-0.vespa.default.svc.cluster.local 4010/34461 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:34:19 GMT
1507131249.243416 vespa-0.vespa.default.svc.cluster.local 4010/34461 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=zookeepers,namespace=cloud.config,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131253.132 vespa-0.vespa.default.svc.cluster.local 283/182 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131253.133 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,18958,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131253.420 vespa-0.vespa.default.svc.cluster.local 283/183 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131259.818894 vespa-0.vespa.default.svc.cluster.local 4152/43622 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131259.819077 vespa-0.vespa.default.svc.cluster.local 4152/43622 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:34:29 GMT
1507131259.819162 vespa-0.vespa.default.svc.cluster.local 4152/43622 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131259.879835 vespa-0.vespa.default.svc.cluster.local 4180/61942 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131259.879959 vespa-0.vespa.default.svc.cluster.local 4180/61942 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:34:29 GMT
1507131259.880003 vespa-0.vespa.default.svc.cluster.local 4180/61942 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131263.534453 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131263.534499 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=6
1507131263.534506 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=20
1507131263.624 vespa-0.vespa.default.svc.cluster.local 283/184 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131263.624 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,19978,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131270.266051 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131270.266110 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:35:30 GMT
1507131270.266162 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=documenttypes,namespace=document,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 12000 milliseconds
1507131277.629 vespa-0.vespa.default.svc.cluster.local 283/185 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131277.631 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,22480,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131283.564 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=container/container.0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131283.568 vespa-0.vespa.default.svc.cluster.local 283/186 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131283.568 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131284.229169 vespa-0.vespa.default.svc.cluster.local 4010/6569 filedistributorservice filedistributor event stopping/1 name="filedistributor" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.zookeepers', configid 'filedistribution/vespa-0.vespa.default.svc.cluster.local'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7fa55e85aa65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7fa55e84d4b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7fa55e85e27b]\n /opt/vespa/sbin/vespa-filedistributor-bin(std::unique_ptr<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const>, std::default_delete<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const> > > config::ConfigSubscriber::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, unsigned long)+0xca) [0x4341fa]\n /opt/vespa/sbin/vespa-filedistributor-bin(void config::ConfigPoller::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, config::IFetcherCallback<cloud::config::internal::InternalZookeepersType const>*, unsigned long)+0x27) [0x435187]\n /opt/vespa/sbin/vespa-filedistributor-bin(FileDistributorApplication::Main()+0x9a) [0x42f4ea]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7fa55ddf20f1]\n /opt/vespa/sbin/vespa-filedistributor-bin(executeApplication(int, char**)+0xa0e) [0x42f12e]\n /opt/vespa/sbin/vespa-filedistributor-bin(main+0x50) [0x42cd20]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fa55c76ec05]\n /opt/vespa/sbin/vespa-filedistributor-bin() [0x42e021]"
1507131284.236670 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="filedistributorservice" pid=4010 exitcode=4
1507131284.237525 vespa-0.vespa.default.svc.cluster.local 4273/1079 config-sentinel config-sentinel.service event starting/1 name="filedistributorservice"
1507131284.291959 vespa-0.vespa.default.svc.cluster.local 4273/4114 filedistributorservice filedistributor event started/1 name="filedistributor"
1507131286.583 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0 within timeout, will retry
1507131291.537 vespa-0.vespa.default.svc.cluster.local 283/187 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131294.820284 vespa-0.vespa.default.svc.cluster.local 4152/15730 topleveldispatch fdispatch error getting config: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.search.core.fdispatchrc', configid 'music/search/cluster.music/tlds/container.0.tld.0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7fa04fc7ea65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7fa04fc714b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7fa04fc8227b]\n /opt/vespa/sbin/vespa-dispatch-bin(std::unique_ptr<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const>, std::default_delete<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const> > > config::ConfigSubscriber::subscribe<vespa::config::search::core::internal::InternalFdispatchrcType const>(std::string const&, unsigned long)+0xca) [0x41f22a]\n /opt/vespa/sbin/vespa-dispatch-bin(fdispatch::Fdispatch::Fdispatch(config::ConfigUri const&)+0x270) [0x41e580]\n /opt/vespa/sbin/vespa-dispatch-bin(FastS_FDispatchApp::Main()+0xa9) [0x41c6e9]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7fa04eea10f1]\n /opt/vespa/sbin/vespa-dispatch-bin(main+0x45) [0x41ad55]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fa04ac23c05]\n /opt/vespa/sbin/vespa-dispatch-bin() [0x41c219]
1507131294.820341 vespa-0.vespa.default.svc.cluster.local 4152/15730 topleveldispatch fdispatch event stopping/1 name="fdispatch" why="error getting config"
1507131294.833566 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="topleveldispatch" pid=4152 exitcode=1
1507131294.833947 vespa-0.vespa.default.svc.cluster.local 4310/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131294.876017 vespa-0.vespa.default.svc.cluster.local 4180/34050 distributor vds.application error Caught exception when starting: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.content.core.stor-server', configid 'music/distributor/0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f0503b78a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f0503b6b4b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f0503b7c27b]\n /opt/vespa/sbin/vespa-distributord-bin(std::unique_ptr<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const>, std::default_delete<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const> > > config::ConfigSubscriber::subscribe<vespa::config::content::core::internal::InternalStorServerType const>(std::string const&, unsigned long)+0xca) [0x40cfea]\n /opt/vespa/sbin/vespa-distributord-bin(config::ConfigGetter<vespa::config::content::core::internal::InternalStorServerType const>::getConfig(long&, std::string const&, std::shared_ptr<config::IConfigContext> const&, unsigned long)+0x48) [0x40d128]\n /opt/vespa/sbin/vespa-distributord-bin(storage::StorageApp::Main()+0xee) [0x40c31e]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f05031100f1]\n /opt/vespa/sbin/vespa-distributord-bin(main+0x3b) [0x40b64b]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f0501634c05]\n /opt/vespa/sbin/vespa-distributord-bin() [0x40b7dd]
1507131294.883666 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="distributor" pid=4180 exitcode=1
1507131294.884148 vespa-0.vespa.default.svc.cluster.local 4341/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131294.911565 vespa-0.vespa.default.svc.cluster.local 4310/10897 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131297.039 vespa-0.vespa.default.svc.cluster.local 283/188 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131297.040 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,16438,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131298.087 vespa-0.vespa.default.svc.cluster.local 3905/1 docprocservice Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131298.077 vespa-0.vespa.default.svc.cluster.local 3978/1 container-clustercontroller Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131298.159 vespa-0.vespa.default.svc.cluster.local 3905/1 docprocservice Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131298.159 vespa-0.vespa.default.svc.cluster.local 3978/1 container-clustercontroller Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131298.264976 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="docprocservice" pid=3905 exitcode=6
1507131298.265272 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container-clustercontroller" pid=3978 exitcode=6
1507131298.265327 vespa-0.vespa.default.svc.cluster.local 4388/1079 config-sentinel config-sentinel.service event starting/1 name="docprocservice"
1507131298.265538 vespa-0.vespa.default.svc.cluster.local 4389/1079 config-sentinel config-sentinel.service event starting/1 name="container-clustercontroller"
1507131298.393 vespa-0.vespa.default.svc.cluster.local 3834/1 container Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131298.412 vespa-0.vespa.default.svc.cluster.local 3834/1 container Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131298.459678 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container" pid=3834 exitcode=6
1507131298.460016 vespa-0.vespa.default.svc.cluster.local 4532/1079 config-sentinel config-sentinel.service event starting/1 name="container"
1507131300.000 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=procTime value=347
1507131300.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.requests value=67
1507131300.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.failedRequests value=42
1507131303.572791 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131303.572922 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:36:03 GMT
1507131303.572975 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=filedistributorrpc,namespace=cloud.config.filedistribution,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131304.307845 vespa-0.vespa.default.svc.cluster.local 4273/32006 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131304.308028 vespa-0.vespa.default.svc.cluster.local 4273/32006 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:35:14 GMT
1507131304.308100 vespa-0.vespa.default.svc.cluster.local 4273/32006 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=zookeepers,namespace=cloud.config,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131308.365895 vespa-0.vespa.default.svc.cluster.local 4388 docprocservice stderr warning error 103: (RPC) Invocation timed out
1507131308.365947 vespa-0.vespa.default.svc.cluster.local 4389 container-clustercontroller stderr warning error 103: (RPC) Invocation timed out
1507131308.383005 vespa-0.vespa.default.svc.cluster.local 4388 docprocservice stdout debug starting docprocservice for docproc/cluster.music.indexing/0
1507131308.383055 vespa-0.vespa.default.svc.cluster.local 4388 docprocservice stdout debug without numactl (no permission or not available)
1507131308.383060 vespa-0.vespa.default.svc.cluster.local 4388 docprocservice stdout debug numactlcmd:
1507131308.383064 vespa-0.vespa.default.svc.cluster.local 4388 docprocservice stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131308.383072 vespa-0.vespa.default.svc.cluster.local 4389 container-clustercontroller stdout debug starting container-clustercontroller for admin/cluster-controllers/0
1507131308.383076 vespa-0.vespa.default.svc.cluster.local 4389 container-clustercontroller stdout debug without numactl (no permission or not available)
1507131308.383080 vespa-0.vespa.default.svc.cluster.local 4389 container-clustercontroller stdout debug numactlcmd:
1507131308.383084 vespa-0.vespa.default.svc.cluster.local 4389 container-clustercontroller stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131308.538305 vespa-0.vespa.default.svc.cluster.local 4532 container stdout debug starting container for container/container.0
1507131308.538329 vespa-0.vespa.default.svc.cluster.local 4532 container stdout debug without numactl (no permission or not available)
1507131308.538334 vespa-0.vespa.default.svc.cluster.local 4532 container stdout debug numactlcmd:
1507131308.538339 vespa-0.vespa.default.svc.cluster.local 4532 container stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131308.538344 vespa-0.vespa.default.svc.cluster.local 4532 container stderr warning error 103: (RPC) Invocation timed out
1507131309.167 vespa-0.vespa.default.svc.cluster.local 283/189 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131309.168 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,21212,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131309.415230 vespa-0.vespa.default.svc.cluster.local 4388 docprocservice stdout debug Initializing application without privileges.
1507131309.415265 vespa-0.vespa.default.svc.cluster.local 4389 container-clustercontroller stdout debug Initializing application without privileges.
1507131309.449242 vespa-0.vespa.default.svc.cluster.local 4532 container stdout debug Initializing application without privileges.
1507131314.931497 vespa-0.vespa.default.svc.cluster.local 4310/38789 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131314.931730 vespa-0.vespa.default.svc.cluster.local 4310/38789 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:35:24 GMT
1507131314.931809 vespa-0.vespa.default.svc.cluster.local 4310/38789 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131314.972138 vespa-0.vespa.default.svc.cluster.local 4341/55481 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131314.972420 vespa-0.vespa.default.svc.cluster.local 4341/55481 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:35:24 GMT
1507131314.972476 vespa-0.vespa.default.svc.cluster.local 4341/55481 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131318.430323 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="heartbeats_failed" value=0
1507131318.430369 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="register_reqs" value=0
1507131318.430376 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="mirror_reqs" value=59
1507131318.430381 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="wantadd_reqs" value=0
1507131318.430386 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="doadd_reqs" value=0
1507131318.430392 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="doremove_reqs" value=0
1507131318.430397 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="admin_reqs" value=0
1507131318.430402 vespa-0.vespa.default.svc.cluster.local 2314/35998 slobrok vespa-slobrok.rpchooks event count/1 name="other_reqs" value=0
1507131319.564 vespa-0.vespa.default.svc.cluster.local 2308/16 logserver ADM.com.yahoo.logserver.handlers.logmetrics.LogMetricsHandler event count/1 name=log_message.error value=22
1507131319.564 vespa-0.vespa.default.svc.cluster.local 2308/16 logserver ADM.com.yahoo.logserver.handlers.logmetrics.LogMetricsHandler event count/1 name=log_message.fatal value=0
1507131319.565 vespa-0.vespa.default.svc.cluster.local 2308/16 logserver ADM.com.yahoo.logserver.handlers.logmetrics.LogMetricsHandler event count/1 name=log_message.info value=2658
1507131319.565 vespa-0.vespa.default.svc.cluster.local 2308/16 logserver ADM.com.yahoo.logserver.handlers.logmetrics.LogMetricsHandler event count/1 name=log_message.severe value=0
1507131319.565 vespa-0.vespa.default.svc.cluster.local 2308/16 logserver ADM.com.yahoo.logserver.handlers.logmetrics.LogMetricsHandler event count/1 name=log_message.warning value=4293
1507131322.488 vespa-0.vespa.default.svc.cluster.local 283/190 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131322.489 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,vespa-0.vespa.default.svc.cluster.local,,0,23512,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131323.654051 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131323.654084 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=6
1507131323.654090 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=26
1507131327.571 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131327.573 vespa-0.vespa.default.svc.cluster.local 283/175 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131327.574 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131330.583 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0 within timeout, will retry
1507131334.461 vespa-0.vespa.default.svc.cluster.local 283/176 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=docproc/cluster.music.indexing/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131339.303028 vespa-0.vespa.default.svc.cluster.local 4273/4114 filedistributorservice filedistributor event stopping/1 name="filedistributor" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.zookeepers', configid 'filedistribution/vespa-0.vespa.default.svc.cluster.local'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7fbe24d5da65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7fbe24d504b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7fbe24d6127b]\n /opt/vespa/sbin/vespa-filedistributor-bin(std::unique_ptr<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const>, std::default_delete<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const> > > config::ConfigSubscriber::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, unsigned long)+0xca) [0x4341fa]\n /opt/vespa/sbin/vespa-filedistributor-bin(void config::ConfigPoller::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, config::IFetcherCallback<cloud::config::internal::InternalZookeepersType const>*, unsigned long)+0x27) [0x435187]\n /opt/vespa/sbin/vespa-filedistributor-bin(FileDistributorApplication::Main()+0x9a) [0x42f4ea]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7fbe242f50f1]\n /opt/vespa/sbin/vespa-filedistributor-bin(executeApplication(int, char**)+0xa0e) [0x42f12e]\n /opt/vespa/sbin/vespa-filedistributor-bin(main+0x50) [0x42cd20]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fbe22c71c05]\n /opt/vespa/sbin/vespa-filedistributor-bin() [0x42e021]"
1507131339.309850 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="filedistributorservice" pid=4273 exitcode=4
1507131339.310302 vespa-0.vespa.default.svc.cluster.local 4715/1079 config-sentinel config-sentinel.service event starting/1 name="filedistributorservice"
1507131339.361390 vespa-0.vespa.default.svc.cluster.local 4715/38297 filedistributorservice filedistributor event started/1 name="filedistributor"
1507131339.507 vespa-0.vespa.default.svc.cluster.local 283/177 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131339.508 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,21670,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131339.600627 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131339.600708 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:36:39 GMT
1507131339.600739 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=filedistributorrpc,namespace=cloud.config.filedistribution,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 7000 milliseconds
1507131349.918288 vespa-0.vespa.default.svc.cluster.local 4310/10897 topleveldispatch fdispatch error getting config: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.search.core.fdispatchrc', configid 'music/search/cluster.music/tlds/container.0.tld.0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f08bd7e6a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f08bd7d94b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f08bd7ea27b]\n /opt/vespa/sbin/vespa-dispatch-bin(std::unique_ptr<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const>, std::default_delete<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const> > > config::ConfigSubscriber::subscribe<vespa::config::search::core::internal::InternalFdispatchrcType const>(std::string const&, unsigned long)+0xca) [0x41f22a]\n /opt/vespa/sbin/vespa-dispatch-bin(fdispatch::Fdispatch::Fdispatch(config::ConfigUri const&)+0x270) [0x41e580]\n /opt/vespa/sbin/vespa-dispatch-bin(FastS_FDispatchApp::Main()+0xa9) [0x41c6e9]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f08bca090f1]\n /opt/vespa/sbin/vespa-dispatch-bin(main+0x45) [0x41ad55]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f08b878bc05]\n /opt/vespa/sbin/vespa-dispatch-bin() [0x41c219]
1507131349.918442 vespa-0.vespa.default.svc.cluster.local 4310/10897 topleveldispatch fdispatch event stopping/1 name="fdispatch" why="error getting config"
1507131349.940399 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="topleveldispatch" pid=4310 exitcode=1
1507131349.940897 vespa-0.vespa.default.svc.cluster.local 4752/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131349.967771 vespa-0.vespa.default.svc.cluster.local 4341/27589 distributor vds.application error Caught exception when starting: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.content.core.stor-server', configid 'music/distributor/0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f288a56ea65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f288a5614b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f288a57227b]\n /opt/vespa/sbin/vespa-distributord-bin(std::unique_ptr<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const>, std::default_delete<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const> > > config::ConfigSubscriber::subscribe<vespa::config::content::core::internal::InternalStorServerType const>(std::string const&, unsigned long)+0xca) [0x40cfea]\n /opt/vespa/sbin/vespa-distributord-bin(config::ConfigGetter<vespa::config::content::core::internal::InternalStorServerType const>::getConfig(long&, std::string const&, std::shared_ptr<config::IConfigContext> const&, unsigned long)+0x48) [0x40d128]\n /opt/vespa/sbin/vespa-distributord-bin(storage::StorageApp::Main()+0xee) [0x40c31e]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f2889b060f1]\n /opt/vespa/sbin/vespa-distributord-bin(main+0x3b) [0x40b64b]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f288802ac05]\n /opt/vespa/sbin/vespa-distributord-bin() [0x40b7dd]
1507131350.006088 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="distributor" pid=4341 exitcode=1
1507131350.006471 vespa-0.vespa.default.svc.cluster.local 4783/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131350.007228 vespa-0.vespa.default.svc.cluster.local 4752/20116 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131350.480 vespa-0.vespa.default.svc.cluster.local 283/178 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131350.481 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,20197,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131359.383449 vespa-0.vespa.default.svc.cluster.local 4715/654 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131359.383575 vespa-0.vespa.default.svc.cluster.local 4715/654 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:36:09 GMT
1507131359.383614 vespa-0.vespa.default.svc.cluster.local 4715/654 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=zookeepers,namespace=cloud.config,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131360.000 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.failedRequests value=48
1507131360.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=procTime value=347
1507131360.001 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.requests value=73
1507131365.959 vespa-0.vespa.default.svc.cluster.local 283/179 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131365.960 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,vespa-0.vespa.default.svc.cluster.local,,0,16607,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131367.176 vespa-0.vespa.default.svc.cluster.local 4388/1 docprocservice Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131367.213 vespa-0.vespa.default.svc.cluster.local 4388/1 docprocservice Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131367.293674 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="docprocservice" pid=4388 exitcode=6
1507131367.294030 vespa-0.vespa.default.svc.cluster.local 4828/1079 config-sentinel config-sentinel.service event starting/1 name="docprocservice"
1507131367.336 vespa-0.vespa.default.svc.cluster.local 4532/1 container Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131367.353 vespa-0.vespa.default.svc.cluster.local 4532/1 container Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131367.347 vespa-0.vespa.default.svc.cluster.local 4389/1 container-clustercontroller Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131367.371 vespa-0.vespa.default.svc.cluster.local 4389/1 container-clustercontroller Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131367.440617 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container" pid=4532 exitcode=6
1507131367.440953 vespa-0.vespa.default.svc.cluster.local 4903/1079 config-sentinel config-sentinel.service event starting/1 name="container"
1507131367.443314 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container-clustercontroller" pid=4389 exitcode=6
1507131367.443664 vespa-0.vespa.default.svc.cluster.local 4905/1079 config-sentinel config-sentinel.service event starting/1 name="container-clustercontroller"
1507131370.022605 vespa-0.vespa.default.svc.cluster.local 4752/48008 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131370.022804 vespa-0.vespa.default.svc.cluster.local 4752/48008 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:36:20 GMT
1507131370.022864 vespa-0.vespa.default.svc.cluster.local 4752/48008 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131370.084105 vespa-0.vespa.default.svc.cluster.local 4783/11638 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131370.084214 vespa-0.vespa.default.svc.cluster.local 4783/11638 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:36:20 GMT
1507131370.084249 vespa-0.vespa.default.svc.cluster.local 4783/11638 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131371.579 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131371.592 vespa-0.vespa.default.svc.cluster.local 283/181 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131371.593 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131375.582 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=container/container.0 within timeout, will retry
1507131376.299 vespa-0.vespa.default.svc.cluster.local 283/183 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131376.300 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,21554,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131376.618984 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131376.619072 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:37:16 GMT
1507131376.619090 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=filedistributorrpc,namespace=cloud.config.filedistribution,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 8000 milliseconds
1507131377.403471 vespa-0.vespa.default.svc.cluster.local 4828 docprocservice stderr warning error 103: (RPC) Invocation timed out
1507131377.420590 vespa-0.vespa.default.svc.cluster.local 4828 docprocservice stdout debug starting docprocservice for docproc/cluster.music.indexing/0
1507131377.420615 vespa-0.vespa.default.svc.cluster.local 4828 docprocservice stdout debug without numactl (no permission or not available)
1507131377.420620 vespa-0.vespa.default.svc.cluster.local 4828 docprocservice stdout debug numactlcmd:
1507131377.420623 vespa-0.vespa.default.svc.cluster.local 4828 docprocservice stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131377.535478 vespa-0.vespa.default.svc.cluster.local 4903 container stderr warning error 103: (RPC) Invocation timed out
1507131377.535505 vespa-0.vespa.default.svc.cluster.local 4905 container-clustercontroller stderr warning error 103: (RPC) Invocation timed out
1507131377.550005 vespa-0.vespa.default.svc.cluster.local 4903 container stdout debug starting container for container/container.0
1507131377.550029 vespa-0.vespa.default.svc.cluster.local 4903 container stdout debug without numactl (no permission or not available)
1507131377.550034 vespa-0.vespa.default.svc.cluster.local 4903 container stdout debug numactlcmd:
1507131377.550038 vespa-0.vespa.default.svc.cluster.local 4903 container stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131377.550063 vespa-0.vespa.default.svc.cluster.local 4905 container-clustercontroller stdout debug starting container-clustercontroller for admin/cluster-controllers/0
1507131377.550069 vespa-0.vespa.default.svc.cluster.local 4905 container-clustercontroller stdout debug without numactl (no permission or not available)
1507131377.550073 vespa-0.vespa.default.svc.cluster.local 4905 container-clustercontroller stdout debug numactlcmd:
1507131377.550077 vespa-0.vespa.default.svc.cluster.local 4905 container-clustercontroller stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131378.400885 vespa-0.vespa.default.svc.cluster.local 4905 container-clustercontroller stdout debug Initializing application without privileges.
1507131378.419217 vespa-0.vespa.default.svc.cluster.local 4828 docprocservice stdout debug Initializing application without privileges.
1507131378.419248 vespa-0.vespa.default.svc.cluster.local 4903 container stdout debug Initializing application without privileges.
1507131379.150 vespa-0.vespa.default.svc.cluster.local 283/184 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=admin/cluster-controllers/0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131382.500 vespa-0.vespa.default.svc.cluster.local 283/185 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131382.501 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,19532,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131383.496906 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131383.496972 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=6
1507131383.496985 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=32
1507131388.590 vespa-0.vespa.default.svc.cluster.local 283/186 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131388.591 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,23440,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131394.371630 vespa-0.vespa.default.svc.cluster.local 4715/38297 filedistributorservice filedistributor event stopping/1 name="filedistributor" why="ConfigTimeoutException: Timed out while subscribing to 'cloud.config.zookeepers', configid 'filedistribution/vespa-0.vespa.default.svc.cluster.local'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f046b171a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f046b1644b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f046b17527b]\n /opt/vespa/sbin/vespa-filedistributor-bin(std::unique_ptr<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const>, std::default_delete<config::ConfigHandle<cloud::config::internal::InternalZookeepersType const> > > config::ConfigSubscriber::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, unsigned long)+0xca) [0x4341fa]\n /opt/vespa/sbin/vespa-filedistributor-bin(void config::ConfigPoller::subscribe<cloud::config::internal::InternalZookeepersType const>(std::string const&, config::IFetcherCallback<cloud::config::internal::InternalZookeepersType const>*, unsigned long)+0x27) [0x435187]\n /opt/vespa/sbin/vespa-filedistributor-bin(FileDistributorApplication::Main()+0x9a) [0x42f4ea]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f046a7090f1]\n /opt/vespa/sbin/vespa-filedistributor-bin(executeApplication(int, char**)+0xa0e) [0x42f12e]\n /opt/vespa/sbin/vespa-filedistributor-bin(main+0x50) [0x42cd20]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f0469085c05]\n /opt/vespa/sbin/vespa-filedistributor-bin() [0x42e021]"
1507131394.391313 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="filedistributorservice" pid=4715 exitcode=4
1507131394.392122 vespa-0.vespa.default.svc.cluster.local 5159/1079 config-sentinel config-sentinel.service event starting/1 name="filedistributorservice"
1507131394.471209 vespa-0.vespa.default.svc.cluster.local 5159/47840 filedistributorservice filedistributor event started/1 name="filedistributor"
1507131394.869 vespa-0.vespa.default.svc.cluster.local 283/187 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131394.870 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,19178,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131402.857 vespa-0.vespa.default.svc.cluster.local 283/188 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131402.858 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,16063,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131405.014423 vespa-0.vespa.default.svc.cluster.local 4752/20116 topleveldispatch fdispatch error getting config: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.search.core.fdispatchrc', configid 'music/search/cluster.music/tlds/container.0.tld.0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7fcd30bd1a65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7fcd30bc44b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7fcd30bd527b]\n /opt/vespa/sbin/vespa-dispatch-bin(std::unique_ptr<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const>, std::default_delete<config::ConfigHandle<vespa::config::search::core::internal::InternalFdispatchrcType const> > > config::ConfigSubscriber::subscribe<vespa::config::search::core::internal::InternalFdispatchrcType const>(std::string const&, unsigned long)+0xca) [0x41f22a]\n /opt/vespa/sbin/vespa-dispatch-bin(fdispatch::Fdispatch::Fdispatch(config::ConfigUri const&)+0x270) [0x41e580]\n /opt/vespa/sbin/vespa-dispatch-bin(FastS_FDispatchApp::Main()+0xa9) [0x41c6e9]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7fcd2fdf40f1]\n /opt/vespa/sbin/vespa-dispatch-bin(main+0x45) [0x41ad55]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fcd2bb76c05]\n /opt/vespa/sbin/vespa-dispatch-bin() [0x41c219]
1507131405.014471 vespa-0.vespa.default.svc.cluster.local 4752/20116 topleveldispatch fdispatch event stopping/1 name="fdispatch" why="error getting config"
1507131405.038453 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="topleveldispatch" pid=4752 exitcode=1
1507131405.039166 vespa-0.vespa.default.svc.cluster.local 5196/1079 config-sentinel config-sentinel.service event starting/1 name="topleveldispatch"
1507131405.071639 vespa-0.vespa.default.svc.cluster.local 4783/49281 distributor vds.application error Caught exception when starting: ConfigTimeoutException: Timed out while subscribing to 'vespa.config.content.core.stor-server', configid 'music/distributor/0'\nBacktrace:\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigTimeoutException::ConfigTimeoutException(vespalib::stringref const&, vespalib::stringref const&, int)+0x15) [0x7f2fcf0bfa65]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigManager::subscribe(config::ConfigKey const&, unsigned long)+0x706) [0x7f2fcf0b24b6]\n /opt/vespa/lib64/libconfig_cloudconfig.so(config::ConfigSubscriptionSet::subscribe(config::ConfigKey const&, unsigned long)+0x4b) [0x7f2fcf0c327b]\n /opt/vespa/sbin/vespa-distributord-bin(std::unique_ptr<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const>, std::default_delete<config::ConfigHandle<vespa::config::content::core::internal::InternalStorServerType const> > > config::ConfigSubscriber::subscribe<vespa::config::content::core::internal::InternalStorServerType const>(std::string const&, unsigned long)+0xca) [0x40cfea]\n /opt/vespa/sbin/vespa-distributord-bin(config::ConfigGetter<vespa::config::content::core::internal::InternalStorServerType const>::getConfig(long&, std::string const&, std::shared_ptr<config::IConfigContext> const&, unsigned long)+0x48) [0x40d128]\n /opt/vespa/sbin/vespa-distributord-bin(storage::StorageApp::Main()+0xee) [0x40c31e]\n /opt/vespa/lib64/libfastos.so(FastOS_ApplicationInterface::Entry(int, char**)+0xa1) [0x7f2fce6570f1]\n /opt/vespa/sbin/vespa-distributord-bin(main+0x3b) [0x40b64b]\n /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2fccb7bc05]\n /opt/vespa/sbin/vespa-distributord-bin() [0x40b7dd]
1507131405.079324 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="distributor" pid=4783 exitcode=1
1507131405.079671 vespa-0.vespa.default.svc.cluster.local 5227/1079 config-sentinel config-sentinel.service event starting/1 name="distributor"
1507131405.156640 vespa-0.vespa.default.svc.cluster.local 5196/31752 topleveldispatch fdispatch event started/1 name="fdispatch"
1507131412.968 vespa-0.vespa.default.svc.cluster.local 283/189 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131412.969 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,17812,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131414.488205 vespa-0.vespa.default.svc.cluster.local 5159/10197 filedistributorservice filedistributor.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131414.490781 vespa-0.vespa.default.svc.cluster.local 5159/10197 filedistributorservice filedistributor.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:37:04 GMT
1507131414.490853 vespa-0.vespa.default.svc.cluster.local 5159/10197 filedistributorservice filedistributor.config.frt.frtconfigagent info Error response or no response from config server (key: name=zookeepers,namespace=cloud.config,configId=filedistribution/vespa-0.vespa.default.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131414.635498 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131414.635544 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:37:54 GMT
1507131414.635558 vespa-0.vespa.default.svc.cluster.local 2309/1189 searchnode proton.config.frt.frtconfigagent info Error response or no response from config server (key: name=filedistributorrpc,namespace=cloud.config.filedistribution,configId=music/search/cluster.music/0) (errcode=103, validresponse:0), trying again in 9000 milliseconds
1507131415.593 vespa-0.vespa.default.svc.cluster.local 543/12 configproxy configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient info Subscribe for 'name=qr-start,namespace=search.config,configId=container/container.0,d41d8cd98f00b204e9800998ecf8427e' failed, closing subscriber
1507131415.625 vespa-0.vespa.default.svc.cluster.local 283/190 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131415.652 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,44000,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131419.583 vespa-0.vespa.default.svc.cluster.local 543/24 configproxy configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler warning No config found for name=qr-start,namespace=search.config,configId=container/container.0 within timeout, will retry
1507131420.000 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.requests value=85
1507131420.002 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=configserver.failedRequests value=58
1507131420.002 vespa-0.vespa.default.svc.cluster.local 283/32 configserver Container.com.yahoo.statistics.Counter event count/1 name=procTime value=350
1507131422.887 vespa-0.vespa.default.svc.cluster.local 283/175 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131424.662 vespa-0.vespa.default.svc.cluster.local 283/176 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131424.663 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,16237,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131425.169601 vespa-0.vespa.default.svc.cluster.local 5196/59644 topleveldispatch fdispatch.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131425.169764 vespa-0.vespa.default.svc.cluster.local 5196/59644 topleveldispatch fdispatch.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:37:15 GMT
1507131425.169823 vespa-0.vespa.default.svc.cluster.local 5196/59644 topleveldispatch fdispatch.config.frt.frtconfigagent info Error response or no response from config server (key: name=fdispatchrc,namespace=vespa.config.search.core,configId=music/search/cluster.music/tlds/container.0.tld.0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131425.200125 vespa-0.vespa.default.svc.cluster.local 5227/12976 distributor vds.config.frt.frtconnection warning Connection to tcp/localhost:19090 failed or timed out
1507131425.200247 vespa-0.vespa.default.svc.cluster.local 5227/12976 distributor vds.config.frt.frtconnection warning FRT Connection tcp/localhost:19090 suspended until 2017-10-04 15:37:15 GMT
1507131425.200283 vespa-0.vespa.default.svc.cluster.local 5227/12976 distributor vds.config.frt.frtconfigagent info Error response or no response from config server (key: name=stor-server,namespace=vespa.config.content.core,configId=music/distributor/0) (errcode=103, validresponse:0), trying again in 6000 milliseconds
1507131435.808 vespa-0.vespa.default.svc.cluster.local 4828/1 docprocservice Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131435.834 vespa-0.vespa.default.svc.cluster.local 4828/1 docprocservice Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=docproc/cluster.music.indexing/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=docproc/cluster.music.indexing/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131435.944927 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="docprocservice" pid=4828 exitcode=6
1507131435.945306 vespa-0.vespa.default.svc.cluster.local 5272/1079 config-sentinel config-sentinel.service event starting/1 name="docprocservice"
1507131436.319 vespa-0.vespa.default.svc.cluster.local 283/177 configserver Container.com.yahoo.vespa.config.server.rpc.GetConfigProcessor info Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131436.321 vespa-0.vespa.default.svc.cluster.local 543/18 configproxy configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester warning Request callback failed. Req: request='name=qr-start,namespace=search.config,configId=container/container.0,vespa-0.vespa.default.svc.cluster.local,,0,17905,Optional[6.10.0]'\nresponse=',-1'\n\nSpec: tcp/localhost:19070 . Req error message: Failed request (Unknown config definition name=qr-start,namespace=search.config,configId=container/container.0) from Connection { Socket[addr=/127.0.0.1,port=49036,localport=19070] }
1507131436.449 vespa-0.vespa.default.svc.cluster.local 4903/1 container Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131436.468 vespa-0.vespa.default.svc.cluster.local 4903/1 container Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=container/container.0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=container/container.0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131436.551708 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container" pid=4903 exitcode=6
1507131436.552111 vespa-0.vespa.default.svc.cluster.local 5345/1079 config-sentinel config-sentinel.service event starting/1 name="container"
1507131436.583 vespa-0.vespa.default.svc.cluster.local 4905/1 container-clustercontroller Container.com.yahoo.jdisc.core.ApplicationLoader warning Exception thrown while activating application.\nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131436.598 vespa-0.vespa.default.svc.cluster.local 4905/1 container-clustercontroller Container.com.yahoo.jdisc.core.StandaloneMain error Unexpected: \nexception=\ncom.yahoo.config.ConfigurationRuntimeException: Subscribe for 'name=qr,namespace=container,configId=admin/cluster-controllers/0' timed out (timeout was 55000 ms): name=qr,namespace=container,configId=admin/cluster-controllers/0, Current generation: null, Generation changed: false, Config changed: false\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribeAndHandleErrors(ConfigSubscriber.java:119)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:102)\n\tat com.yahoo.config.subscription.ConfigSubscriber.subscribe(ConfigSubscriber.java:79)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.com$yahoo$container$di$CloudSubscriberFactory$CloudSubscriber$$subscribe(CloudSubscriberFactory.scala:57)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber$$anonfun$1.apply(CloudSubscriberFactory.scala:51)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)\n\tat scala.collection.immutable.Set$Set1.foreach(Set.scala:79)\n\tat scala.collection.TraversableLike$class.map(TraversableLike.scala:245)\n\tat scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)\n\tat scala.collection.SetLike$class.map(SetLike.scala:92)\n\tat scala.collection.AbstractSet.map(Set.scala:47)\n\tat com.yahoo.container.di.CloudSubscriberFactory$CloudSubscriber.<init>(CloudSubscriberFactory.scala:51)\n\tat com.yahoo.container.di.CloudSubscriberFactory.getSubscriber(CloudSubscriberFactory.scala:28)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.getConfig(ConfiguredApplication.java:135)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:121)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:154)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:34)\n
1507131436.645182 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.service event stopped/1 name="container-clustercontroller" pid=4905 exitcode=6
1507131436.645538 vespa-0.vespa.default.svc.cluster.local 5418/1079 config-sentinel config-sentinel.service event starting/1 name="container-clustercontroller"
1507131443.748354 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="currently_running_services" value=10
1507131443.748389 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event value/1 name="total_restarts_last_period" value=6
1507131443.748394 vespa-0.vespa.default.svc.cluster.local 2301/1079 config-sentinel config-sentinel.metrics event count/1 name="total_restarts_counter" value=38
1507131446.44933 vespa-0.vespa.default.svc.cluster.local 5272 docprocservice stderr warning error 103: (RPC) Invocation timed out
1507131446.60854 vespa-0.vespa.default.svc.cluster.local 5272 docprocservice stdout debug starting docprocservice for docproc/cluster.music.indexing/0
1507131446.60880 vespa-0.vespa.default.svc.cluster.local 5272 docprocservice stdout debug without numactl (no permission or not available)
1507131446.60884 vespa-0.vespa.default.svc.cluster.local 5272 docprocservice stdout debug numactlcmd:
1507131446.60888 vespa-0.vespa.default.svc.cluster.local 5272 docprocservice stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131446.653235 vespa-0.vespa.default.svc.cluster.local 5345 container stdout debug starting container for container/container.0
1507131446.653266 vespa-0.vespa.default.svc.cluster.local 5345 container stdout debug without numactl (no permission or not available)
1507131446.653272 vespa-0.vespa.default.svc.cluster.local 5345 container stdout debug numactlcmd:
1507131446.653276 vespa-0.vespa.default.svc.cluster.local 5345 container stdout config setting up extra preload: /usr/bin/env JAVAVM_LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so LD_PRELOAD=/opt/vespa/lib64/vespa/malloc/libvespamalloc.so
1507131446.653282 vespa-0.vespa.default.svc.cluster.local 5345 container stderr warning error 103: (RPC) Invocation timed out
1507131446.729650 vespa-0.vespa.default.svc.cluster.local 5418 container-clustercontroller stderr warning error 103: (RPC) Invocation timed out
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment