Skip to content

Instantly share code, notes, and snippets.

@y-kanno-im
Last active August 27, 2019 03:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save y-kanno-im/ec29ba23e142085254bfb49bddb1bb71 to your computer and use it in GitHub Desktop.
Save y-kanno-im/ec29ba23e142085254bfb49bddb1bb71 to your computer and use it in GitHub Desktop.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[o.e.c.l.LogConfigurator] [DATANODE:9200] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
/usr/share/elasticsearch/config/log4j2.properties
-
lo
inet 127.0.0.1 netmask:255.0.0.0 scope:host
UP LOOPBACK mtu:65536 index:1
-
eth0
inet 172.17.0.2 netmask:255.255.0.0 broadcast:172.17.255.255 scope:site
hardware 02:42:AC:11:00:02
UP MULTICAST mtu:1500 index:116
-
-> /mnt/nvme0/nodes/0, free_space [1.6tb], usable_space [1.6tb], total_space [1.6tb], mount [/mnt/nvme0 (/dev/nvme0n1)], type [ext4]
[o.e.e.NodeEnvironment] [DATANODE:9200] heap size [31.9gb], compressed ordinary object pointers [false]
[o.e.n.Node] [DATANODE:9200] node name [DATANODE:9200], node ID [lW1oIt9pTZGbxH4v466RJw], cluster name [201905]
[o.e.n.Node] [DATANODE:9200] version[7.3.0], pid[1], build[default/docker/de777fa/2019-07-24T18:30:11.767338Z], OS[Linux/4.14.106-79.86.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[o.e.n.Node] [DATANODE:9200] JVM home [/usr/share/elasticsearch/jdk]
[o.e.n.Node] [DATANODE:9200] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-6364837646770762595, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Xms32g, -Xmx32g, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=17179869184, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [aggs-matrix-stats]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [analysis-common]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [data-frame]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [flattened]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [ingest-common]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [ingest-geoip]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [ingest-user-agent]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [lang-expression]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [lang-mustache]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [lang-painless]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [mapper-extras]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [parent-join]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [percolator]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [rank-eval]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [reindex]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [repository-url]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [transport-netty4]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [vectors]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-ccr]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-core]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-deprecation]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-graph]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-ilm]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-logstash]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-ml]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-monitoring]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-rollup]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-security]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-sql]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-voting-only-node]
[o.e.p.PluginsService] [DATANODE:9200] loaded module [x-pack-watcher]
[o.e.p.PluginsService] [DATANODE:9200] loaded plugin [analysis-icu]
[o.e.p.PluginsService] [DATANODE:9200] loaded plugin [discovery-ec2]
[o.e.p.PluginsService] [DATANODE:9200] loaded plugin [repository-s3]
java.lang.UnsupportedOperationException: sun.misc.Unsafe unavailable
at io.netty.util.internal.CleanerJava9.<clinit>(CleanerJava9.java:68) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:174) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.ConstantPool.<init>(ConstantPool.java:32) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.AttributeKey$1.<init>(AttributeKey.java:27) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.AttributeKey.<clinit>(AttributeKey.java:27) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.<clinit>(Netty4HttpServerTransport.java:300) [transport-netty4-client-7.3.0.jar:7.3.0]
at org.elasticsearch.transport.Netty4Plugin.getSettings(Netty4Plugin.java:52) [transport-netty4-client-7.3.0.jar:7.3.0]
at org.elasticsearch.plugins.PluginsService.lambda$getPluginSettings$0(PluginsService.java:89) [elasticsearch-7.3.0.jar:7.3.0]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) [?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) [?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) [?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) [?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) [?:?]
at org.elasticsearch.plugins.PluginsService.getPluginSettings(PluginsService.java:89) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.node.Node.<init>(Node.java:339) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.node.Node.<init>(Node.java:258) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.3.0.jar:7.3.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.3.0.jar:7.3.0]
java.nio.file.NoSuchFileException: /sys/fs/cgroup/cpuacct/ecs/f9f16c79-8dda-4e9d-8342-fff5bc366780/68a2a68b3a050edc1e3a42fd6b7651321fea5b0a6b24090bd10fa657c259d3eb/cpuacct.usage
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:373) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:424) ~[?:?]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
at java.nio.file.Files.newInputStream(Files.java:158) ~[?:?]
at java.nio.file.Files.newBufferedReader(Files.java:2913) ~[?:?]
at java.nio.file.Files.readAllLines(Files.java:3402) ~[?:?]
at java.nio.file.Files.readAllLines(Files.java:3442) ~[?:?]
at org.elasticsearch.monitor.os.OsProbe.readSingleLine(OsProbe.java:220) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:309) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:296) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:515) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:635) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:49) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:46) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.node.Node.<init>(Node.java:385) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.node.Node.<init>(Node.java:258) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.3.0.jar:7.3.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.3.0.jar:7.3.0]
[o.e.n.Node] [DATANODE:9200] initialized
[o.e.n.Node] [DATANODE:9200] starting ...
[o.e.t.TransportService] [DATANODE:9200] publish_address {DATANODE:9300}, bound_addresses {0.0.0.0:9300}
[o.e.b.BootstrapChecks] [DATANODE:9200] bound or publishing to a non-loopback address, enforcing bootstrap checks
[o.e.c.c.Coordinator] [DATANODE:9200] cluster UUID [BSRgW1mNS4ye1GfFHqVsjQ]
[o.e.t.TcpTransport] [DATANODE:9200] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.17.0.2:46190, remoteAddress=master-a/MASTER-A-IP:9300}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)
at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:745) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:731) ~[elasticsearch-7.3.0.jar:7.3.0]
at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-7.3.0.jar:7.3.0]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]
... 20 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment