Skip to content

Instantly share code, notes, and snippets.

@ashishb888

ashishb888/ignite-persistence-data-node1.log

Last active December 31, 2019 08:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ashishb888/4afe60c681befd0e167ee280d3625521 to your computer and use it in GitHub Desktop.
Save ashishb888/4afe60c681befd0e167ee280d3625521 to your computer and use it in GitHub Desktop.
Download ZIP
Ignite persistence logs
Raw
ignite-persistence-data-node1.log
2019-12-31 13:23:28.035 INFO 50839 --- [main] poc.ignite.IgniteApp : Starting IgniteApp v0.0.1-SNAPSHOT on hdpdev6 with PID 50839 (/opt/ngs/ashishb/apps/ignite/ignite-persistence-data-node/ignite-persistence-data-node-0.0.1-SNAPSHOT.jar started by hdpusr in /opt/ngs/ashishb/apps/ignite/ignite-persistence-data-node)
2019-12-31 13:23:28.038 DEBUG 50839 --- [main] poc.ignite.IgniteApp : Running with Spring Boot v2.1.6.RELEASE, Spring v5.1.8.RELEASE
2019-12-31 13:23:28.038 INFO 50839 --- [main] poc.ignite.IgniteApp : No active profile set, falling back to default profiles: default
2019-12-31 13:23:28.740 INFO 50839 --- [main] poc.ignite.config.BeansConfig : ignite bean service
2019-12-31 13:23:28.956 WARN 50839 --- [main] o.apache.ignite.internal.util.typedef.G : Ignite work directory is not provided, automatically resolved to: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work
2019-12-31 13:23:29.108 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 2.7.6#20190911-sha1:21f7ca41
>>> 2019 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
2019-12-31 13:23:29.116 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Config URL: n/a
2019-12-31 13:23:29.136 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : IgniteConfiguration [igniteInstanceName=null, pubPoolSize=32, svcPoolSize=32, callbackPoolSize=32, stripedPoolSize=32, sysPoolSize=32, mgmtPoolSize=4, igfsPoolSize=32, dataStreamerPoolSize=32, utilityCachePoolSize=32, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=32, igniteHome=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/, igniteWorkDir=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@77846d2c, nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@1dd02175, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=42100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=10000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=16, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@31206beb[Count = 1], stopping=false], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@3e77a1ed, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@1176dcec, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@120d6fe6, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=90000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=32, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=Default_Region, maxSize=268435456, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=[DataRegionConfiguration [name=Data_Region, maxSize=268435456, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0]], storagePath=/var/tmp/db, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=/var/tmp/db/wal, walArchivePath=/var/tmp/db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@21a947fe, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=32, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null]
2019-12-31 13:23:29.136 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Daemon mode: off
2019-12-31 13:23:29.140 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : OS: Linux 3.10.0-957.10.1.el7.x86_64 amd64
2019-12-31 13:23:29.140 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : OS user: hdpusr
2019-12-31 13:23:29.140 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : PID: 50839
2019-12-31 13:23:29.140 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Language runtime: Java Platform API Specification ver. 1.8
2019-12-31 13:23:29.141 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : VM information: Java(TM) SE Runtime Environment 1.8.0_191-b12 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.191-b12
2019-12-31 13:23:29.142 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : VM total memory: 0.96GB
2019-12-31 13:23:29.142 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Remote Management [restart: off, REST: on, JMX (remote: off)]
2019-12-31 13:23:29.142 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Logger: Slf4jLogger [impl=Logger[org.apache.ignite.internal.IgniteKernal], quiet=true]
2019-12-31 13:23:29.142 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : IGNITE_HOME=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/
2019-12-31 13:23:29.143 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : VM arguments: [-Xms1G, -Xmx1G, -Dlogging.file=/var/tmp/logs/ignite-persistence-data-node/ignite-persistence-data-node1]
2019-12-31 13:23:29.143 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting.
2019-12-31 13:23:29.153 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
2019-12-31 13:23:29.153 WARN 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Peer class loading is enabled (disable it in production for performance and deployment consistency reasons)
2019-12-31 13:23:29.153 WARN 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
2019-12-31 13:23:29.156 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : 3-rd party licenses can be found at: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin//libs/licenses
2019-12-31 13:23:29.157 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Local node user attribute [nodeName=data-node]
2019-12-31 13:23:29.225 INFO 50839 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : Configured plugins:
2019-12-31 13:23:29.226 INFO 50839 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : ^-- None
2019-12-31 13:23:29.226 INFO 50839 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor :
2019-12-31 13:23:29.228 INFO 50839 --- [main] o.a.i.i.p.failure.FailureProcessor : Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
2019-12-31 13:23:29.275 INFO 50839 --- [main] o.a.i.s.c.tcp.TcpCommunicationSpi : Successfully bound communication NIO server to TCP port [port=42100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=16, selectorSpins=0, pairedConn=false]
2019-12-31 13:23:29.335 WARN 50839 --- [main] o.a.i.s.c.noop.NoopCheckpointSpi : Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
2019-12-31 13:23:29.373 WARN 50839 --- [main] o.a.i.i.m.c.GridCollisionManager : Collision resolution is disabled (all jobs will be activated upon arrival).
2019-12-31 13:23:29.381 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Security status [authentication=off, tls/ssl=off]
2019-12-31 13:23:29.471 INFO 50839 --- [main] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Successfully bound to TCP port [port=42500, localHost=0.0.0.0/0.0.0.0, locNodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20]
2019-12-31 13:23:29.481 INFO 50839 --- [main] o.a.i.i.p.c.p.f.PdsFoldersResolver : Successfully created new persistent storage folder [/var/tmp/db/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4]
2019-12-31 13:23:29.483 INFO 50839 --- [main] o.a.i.i.p.c.p.f.PdsFoldersResolver : Consistent ID used for local node is [65e3ea66-d894-4978-a20c-30d1caffe6e4] according to persistence data storage folders
2019-12-31 13:23:29.483 INFO 50839 --- [main] i.i.p.c.b.CacheObjectBinaryProcessorImpl : Resolved directory for serialized binary metadata: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work/binary_meta/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4
2019-12-31 13:23:29.707 INFO 50839 --- [main] o.a.i.i.p.c.p.file.FilePageStoreManager : Resolved page store work directory: /var/tmp/db/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4
2019-12-31 13:23:29.708 WARN 50839 --- [main] o.a.i.i.p.c.p.file.FilePageStoreManager : Persistence store directory is in the temp directory and may be cleaned.To avoid this set "IGNITE_HOME" environment variable properly or change location of persistence directories in data storage configuration (see DataStorageConfiguration#walPath, DataStorageConfiguration#walArchivePath, DataStorageConfiguration#storagePath properties). Current persistence store directory is: [/tmp]
2019-12-31 13:23:29.708 INFO 50839 --- [main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log work directory: /var/tmp/db/wal/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4
2019-12-31 13:23:29.709 INFO 50839 --- [main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log archive directory: /var/tmp/db/wal/archive/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4
2019-12-31 13:23:29.809 INFO 50839 --- [main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Started write-ahead log manager [mode=LOG_ONLY]
2019-12-31 13:23:29.816 WARN 50839 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files
2019-12-31 13:23:29.858 INFO 50839 --- [main] o.a.i.i.p.c.d.d.t.PartitionsEvictManager : Evict partition permits=8
2019-12-31 13:23:30.210 INFO 50839 --- [main] o.a.i.i.p.odbc.ClientListenerProcessor : Client connector processor has started on TCP port 10800
2019-12-31 13:23:30.267 INFO 50839 --- [main] o.a.i.i.p.r.p.tcp.GridTcpRestProtocol : Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
2019-12-31 13:23:30.318 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Non-loopback local IPs: 172.17.241.80, 172.17.5.36
2019-12-31 13:23:30.319 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Enabled local MACs: 1866DAE84428, F4E9D4A04FD2
2019-12-31 13:23:30.325 INFO 50839 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=null, endMarker=null]
2019-12-31 13:23:30.338 INFO 50839 --- [main] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:23:30.339 INFO 50839 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:23:30.357 INFO 50839 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:23:30.357 INFO 50839 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished applying WAL changes [updatesApplied=0, time=0ms]
2019-12-31 13:23:30.408 INFO 50839 --- [main] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Connection check threshold is calculated: 90000
2019-12-31 13:23:30.545 INFO 50839 --- [disco-notifier-worker-#101] o.a.i.i.p.cache.mvcc.MvccProcessorImpl : Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778810406, loc=true, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:23:30.601 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Performance suggestions for grid (fix if possible)
2019-12-31 13:23:30.601 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
2019-12-31 13:23:30.602 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
2019-12-31 13:23:30.602 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
2019-12-31 13:23:30.602 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
2019-12-31 13:23:30.602 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Speed up flushing of dirty pages by OS (alter vm.dirty_expire_centisecs parameter by setting to 500)
2019-12-31 13:23:30.602 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Reduce pages swapping ratio (set vm.swappiness=10.000000 or less)
2019-12-31 13:23:30.602 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning
2019-12-31 13:23:30.603 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal :
2019-12-31 13:23:30.603 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
2019-12-31 13:23:30.605 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : Data Regions Configured:
2019-12-31 13:23:30.606 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Default_Region [initSize=256.0 MiB, maxSize=256.0 MiB, persistence=false]
2019-12-31 13:23:30.606 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Data_Region [initSize=256.0 MiB, maxSize=256.0 MiB, persistence=true]
2019-12-31 13:23:30.607 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.7.6#20190911-sha1:21f7ca41c4348909e2fd26ccf59b5b2ce1f4474e
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 3.10.0-957.10.1.el7.x86_64 amd64
>>> CPU(s): 32
>>> Heap: 1.0GB
>>> VM name: 50839@hdpdev6
>>> Local node [ID=0F6ADFB0-8641-487A-B8E9-EF0ACE842B20, order=1, clientMode=false]
>>> Local node addresses: [hdpdev6/127.0.0.1, hdpdev6_oob.nseroot.com/172.17.241.80, /172.17.5.36]
>>> Local ports: TCP:10800 TCP:11211 TCP:42100 TCP:42500
2019-12-31 13:23:30.607 INFO 50839 --- [main] org.apache.ignite.internal.IgniteKernal : >>> Ignite cluster is not active (limited functionality available). Use control.(sh|bat) script or IgniteCluster interface to activate.
2019-12-31 13:23:30.608 INFO 50839 --- [main] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=1, locNode=0f6adfb0, servers=1, clients=0, state=INACTIVE, CPUs=32, offheap=0.5GB, heap=1.0GB]
2019-12-31 13:23:30.681 INFO 50839 --- [main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'
2019-12-31 13:23:30.810 INFO 50839 --- [main] poc.ignite.IgniteApp : Started IgniteApp in 3.303 seconds (JVM running for 3.738)
2019-12-31 13:23:30.811 INFO 50839 --- [main] poc.ignite.IgniteApp : run service
2019-12-31 13:23:45.772 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/172.17.5.36, rmtPort=36587]
2019-12-31 13:23:45.783 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/172.17.5.36, rmtPort=36587]
2019-12-31 13:23:45.784 INFO 50839 --- [tcp-disco-sock-reader-#4] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/172.17.5.36:36587, rmtPort=36587]
2019-12-31 13:23:45.817 INFO 50839 --- [tcp-disco-sock-reader-#4] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Finished serving remote node connection [rmtAddr=/172.17.5.36:36587, rmtPort=36587
2019-12-31 13:23:45.824 INFO 50839 --- [tcp-disco-msg-worker-#2] o.a.i.i.m.e.GridEncryptionManager : Joining node doesn't have encryption data [node=55e3c6ea-0b92-4570-8712-ca78168a6312]
2019-12-31 13:23:45.842 INFO 50839 --- [tcp-disco-msg-worker-#2] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : New next node [newNext=TcpDiscoveryNode [id=55e3c6ea-0b92-4570-8712-ca78168a6312, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42501, /127.0.0.1:42501, hdpdev6_oob.nseroot.com/172.17.241.80:42501], discPort=42501, order=0, intOrder=2, lastExchangeTime=1577778825805, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:23:45.905 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=53891]
2019-12-31 13:23:45.906 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=53891]
2019-12-31 13:23:45.906 INFO 50839 --- [tcp-disco-sock-reader-#5] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/127.0.0.1:53891, rmtPort=53891]
2019-12-31 13:23:45.913 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Added new node to topology: TcpDiscoveryNode [id=55e3c6ea-0b92-4570-8712-ca78168a6312, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42501, /127.0.0.1:42501, hdpdev6_oob.nseroot.com/172.17.241.80:42501], discPort=42501, order=2, intOrder=2, lastExchangeTime=1577778825805, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]
2019-12-31 13:23:45.914 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=2, locNode=0f6adfb0, servers=2, clients=0, state=INACTIVE, CPUs=32, offheap=1.0GB, heap=2.0GB]
2019-12-31 13:23:54.179 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/172.17.5.36, rmtPort=58828]
2019-12-31 13:23:54.179 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/172.17.5.36, rmtPort=58828]
2019-12-31 13:23:54.180 INFO 50839 --- [tcp-disco-sock-reader-#6] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/172.17.5.36:58828, rmtPort=58828]
2019-12-31 13:23:54.197 INFO 50839 --- [tcp-disco-sock-reader-#6] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Finished serving remote node connection [rmtAddr=/172.17.5.36:58828, rmtPort=58828
2019-12-31 13:23:54.200 INFO 50839 --- [tcp-disco-msg-worker-#2] o.a.i.i.m.e.GridEncryptionManager : Joining node doesn't have encryption data [node=51f0bedf-ffe8-4963-a2ae-6bed6955eb23]
2019-12-31 13:23:54.213 INFO 50839 --- [tcp-disco-sock-reader-#5] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Finished serving remote node connection [rmtAddr=/127.0.0.1:53891, rmtPort=53891
2019-12-31 13:23:54.271 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=36525]
2019-12-31 13:23:54.271 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=36525]
2019-12-31 13:23:54.271 INFO 50839 --- [tcp-disco-sock-reader-#7] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/127.0.0.1:36525, rmtPort=36525]
2019-12-31 13:23:54.276 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Added new node to topology: TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834192, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]
2019-12-31 13:23:54.277 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=3, locNode=0f6adfb0, servers=3, clients=0, state=INACTIVE, CPUs=32, offheap=1.5GB, heap=3.0GB]
2019-12-31 13:24:30.608 INFO 50839 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=0f6adfb0, uptime=00:01:00.003]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.07%, avg=0.14%, GC=0%]
^-- PageMemory [pages=0]
^-- Heap [used=140MB, free=85.72%, comm=981MB]
^-- Off-heap [used=0MB, free=100%, comm=0MB]
^-- sysMemPlc region [used=0MB, free=100%, comm=0MB]
^-- metastoreMemPlc region [used=0MB, free=100%, comm=0MB]
^-- Data_Region region [used=0MB, free=100%, comm=0MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=0MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=5, qSize=0]
2019-12-31 13:24:40.258 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Added new node to topology: TcpDiscoveryNode [id=6ef2878f-6b05-4756-9417-8bcfcb9a1701, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:0, /127.0.0.1:0, hdpdev6_oob.nseroot.com/172.17.241.80:0], discPort=0, order=4, intOrder=4, lastExchangeTime=1577778880186, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=true]
2019-12-31 13:24:40.259 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=4, locNode=0f6adfb0, servers=3, clients=1, state=INACTIVE, CPUs=32, offheap=1.5GB, heap=4.0GB]
2019-12-31 13:24:40.568 INFO 50839 --- [ignite-update-notifier-timer] o.a.i.i.p.cluster.GridUpdateNotifier : Update status is not available.
2019-12-31 13:24:40.732 INFO 50839 --- [grid-nio-worker-tcp-comm-0-#72] o.a.i.s.c.tcp.TcpCommunicationSpi : Accepted incoming communication connection [locAddr=/127.0.0.1:42100, rmtAddr=/127.0.0.1:49480]
2019-12-31 13:24:40.793 INFO 50839 --- [mgmt-#111] o.a.i.i.p.c.GridClusterStateProcessor : Sending activate request with BaselineTopology BaselineTopology [id=0, branchingHash=1818609622, branchingType='New BaselineTopology', baselineNodes=[d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b, 127.0.0.1,172.17.241.80,172.17.5.36:42502, 65e3ea66-d894-4978-a20c-30d1caffe6e4]]
2019-12-31 13:24:40.798 INFO 50839 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Received activate request with BaselineTopology[id=0]
2019-12-31 13:24:40.800 INFO 50839 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Started state transition: true
2019-12-31 13:24:40.838 INFO 50839 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], mvccCrd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=true, crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=0f6adfb0-8641-487a-b8e9-ef0ace842b20, customEvt=ChangeGlobalStateMessage [id=15a02fa5f61-fb354dfe-aed5-4d86-8fc8-cc9dce3672e6, reqId=3b5c20df-697e-423d-9208-6a02d536305b, initiatingNodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=1818609622, branchingType='New BaselineTopology', baselineNodes=[d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b, 127.0.0.1,172.17.241.80,172.17.5.36:42502, 65e3ea66-d894-4978-a20c-30d1caffe6e4]], forceChangeBaselineTopology=false, timestamp=1577778880793], allowMerge=false]
2019-12-31 13:24:40.838 INFO 50839 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Start activation process [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:40.839 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.file.FilePageStoreManager : Resolved page store work directory: /var/tmp/db/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4
2019-12-31 13:24:40.839 WARN 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.file.FilePageStoreManager : Persistence store directory is in the temp directory and may be cleaned.To avoid this set "IGNITE_HOME" environment variable properly or change location of persistence directories in data storage configuration (see DataStorageConfiguration#walPath, DataStorageConfiguration#walArchivePath, DataStorageConfiguration#storagePath properties). Current persistence store directory is: [/tmp]
2019-12-31 13:24:40.839 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log work directory: /var/tmp/db/wal/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4
2019-12-31 13:24:40.840 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log archive directory: /var/tmp/db/wal/archive/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4
2019-12-31 13:24:40.841 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Started write-ahead log manager [mode=LOG_ONLY]
2019-12-31 13:24:40.851 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:24:40.855 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:24:40.871 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=256.0 MiB, pages=63520, tableSize=4.9 MiB, checkpointBuffer=256.0 MiB]
2019-12-31 13:24:40.875 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:24:40.889 INFO 50839 --- [exchange-worker-#103] org.apache.ignite.cache.msg : Components activation performed in 50 ms.
2019-12-31 13:24:40.902 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=null, endMarker=null]
2019-12-31 13:24:40.902 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:24:40.905 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resuming logging to WAL segment [file=/var/tmp/db/wal/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4/0000000000000000.wal, offset=0, ver=2]
2019-12-31 13:24:40.943 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.GridClusterStateProcessor : Writing BaselineTopology[id=0]
2019-12-31 13:24:40.963 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Binary recovery performed in 73 ms.
2019-12-31 13:24:40.967 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Updating caches registry performed in 3 ms.
2019-12-31 13:24:41.011 INFO 50839 --- [sys-#115] .i.p.c.GridCachePartitionExchangeManager : Waiting for coordinator initialization [node=6ef2878f-6b05-4756-9417-8bcfcb9a1701, nodeOrder=4, ver=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.106 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false], encryptionEnabled=false]
2019-12-31 13:24:41.107 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Caches starting performed in 139 ms.
2019-12-31 13:24:41.111 INFO 50839 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity initialization for started caches performed in 4 ms.
2019-12-31 13:24:41.112 INFO 50839 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Successfully activated caches [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.121 INFO 50839 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL]
2019-12-31 13:24:41.123 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=null, endMarker=null]
2019-12-31 13:24:41.123 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:24:41.184 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Restoring partition state for local groups [cntPartStateWal=0, lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:24:41.188 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished restoring partition state for local groups [cntProcessed=100, cntPartStateWal=0, time=10ms]
2019-12-31 13:24:41.189 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished applying WAL changes [updatesApplied=0, time=70ms]
2019-12-31 13:24:41.189 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Logical recovery performed in 66 ms.
2019-12-31 13:24:41.233 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpointer initilialzation performed in 15 ms.
2019-12-31 13:24:41.234 INFO 50839 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], crd=true]
2019-12-31 13:24:41.236 INFO 50839 --- [db-checkpoint-thread-#116] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint started [checkpointId=d2044ba5-5400-4ff1-bf54-94f7fcc1f8fe, startPtr=FileWALPointer [idx=0, fileOff=240085, len=1947], checkpointLockWait=0ms, checkpointLockHoldTime=13ms, walCpRecordFsyncDuration=0ms, pages=25, reason='node started']
2019-12-31 13:24:41.254 INFO 50839 --- [grid-nio-worker-tcp-comm-1-#73] o.a.i.s.c.tcp.TcpCommunicationSpi : Accepted incoming communication connection [locAddr=/127.0.0.1:42100, rmtAddr=/127.0.0.1:49484]
2019-12-31 13:24:41.255 INFO 50839 --- [db-checkpoint-thread-#116] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=d2044ba5-5400-4ff1-bf54-94f7fcc1f8fe, pages=25, markPos=FileWALPointer [idx=0, fileOff=240085, len=1947], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=17ms, pagesWrite=8ms, fsync=10ms, total=35ms]
2019-12-31 13:24:41.279 INFO 50839 --- [sys-#121] .c.d.d.p.GridDhtPartitionsExchangeFuture : Coordinator received single message [ver=AffinityTopologyVersion [topVer=4, minorTopVer=1], node=55e3c6ea-0b92-4570-8712-ca78168a6312, allReceived=false]
2019-12-31 13:24:41.499 INFO 50839 --- [tcp-disco-sock-reader-#7] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Finished serving remote node connection [rmtAddr=/127.0.0.1:36525, rmtPort=36525
2019-12-31 13:24:41.503 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=39657]
2019-12-31 13:24:41.503 INFO 50839 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=39657]
2019-12-31 13:24:41.504 INFO 50839 --- [tcp-disco-sock-reader-#8] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/127.0.0.1:39657, rmtPort=39657]
2019-12-31 13:24:41.509 INFO 50839 --- [tcp-disco-sock-reader-#8] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Connection check done: [liveAddr=null, previousNode=TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834192, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], addressesToCheck=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], connectingNodeId=55e3c6ea-0b92-4570-8712-ca78168a6312]
2019-12-31 13:24:41.509 INFO 50839 --- [tcp-disco-sock-reader-#8] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Previous node alive: [alive=false, lastMessageReceivedTime=1577778881138, now=1577778881502, connCheckInterval=500]
2019-12-31 13:24:41.527 WARN 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Node FAILED: TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834192, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]
2019-12-31 13:24:41.528 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=5, locNode=0f6adfb0, servers=2, clients=1, state=ACTIVE, CPUs=32, offheap=1.0GB, heap=3.0GB]
2019-12-31 13:24:41.528 INFO 50839 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : ^-- Baseline [id=0, size=3, online=2, offline=1]
2019-12-31 13:24:41.540 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.540 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Affinity changes (coordinator) applied in 0 ms.
2019-12-31 13:24:41.549 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Partitions validation performed in 8 ms.
2019-12-31 13:24:41.553 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Partitions assignment performed in 3 ms.
2019-12-31 13:24:41.558 INFO 50839 --- [sys-#123] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity recalculation (custom message) performed in 5 ms.
2019-12-31 13:24:41.568 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Preparing Full Message performed in 9 ms.
2019-12-31 13:24:41.571 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Full Message performed in 1 ms.
2019-12-31 13:24:41.571 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Full Message to all nodes performed in 3 ms.
2019-12-31 13:24:41.572 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], err=null]
2019-12-31 13:24:41.573 INFO 50839 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Received state change finish message: true
2019-12-31 13:24:41.576 INFO 50839 --- [sys-#123] .i.p.c.GridCachePartitionExchangeManager : Sending Full Message for AffinityTopologyVersion [topVer=4, minorTopVer=1] performed in 1 ms.
2019-12-31 13:24:41.577 INFO 50839 --- [sys-#123] o.a.i.i.p.cache.mvcc.MvccProcessorImpl : Initialize local node as mvcc coordinator [node=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538]
2019-12-31 13:24:41.584 INFO 50839 --- [sys-#123] o.a.i.i.p.cache.GridCacheProcessor : Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20
2019-12-31 13:24:41.586 INFO 50839 --- [sys-#123] .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange [localNode=0f6adfb0-8641-487a-b8e9-ef0ace842b20, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778881567, loc=true, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], durationFromInit=751]
2019-12-31 13:24:41.591 INFO 50839 --- [exchange-worker-#103] .i.p.c.GridCachePartitionExchangeManager : Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=4, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, node=0f6adfb0-8641-487a-b8e9-ef0ace842b20]
2019-12-31 13:24:41.591 INFO 50839 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_FAILED, evtNode=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, customEvt=null, allowMerge=true]
2019-12-31 13:24:41.594 INFO 50839 --- [sys-#126] o.a.i.i.p.c.GridClusterStateProcessor : Successfully performed final activation steps [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.595 INFO 50839 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
2019-12-31 13:24:41.598 INFO 50839 --- [sys-#128] .i.p.c.GridCachePartitionExchangeManager : Ignore single message without exchange id (there is exchange in progress) [nodeId=55e3c6ea-0b92-4570-8712-ca78168a6312]
2019-12-31 13:24:41.617 INFO 50839 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=[], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]]
2019-12-31 13:24:41.617 INFO 50839 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL]
2019-12-31 13:24:41.617 INFO 50839 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Logical recovery performed in 0 ms.
2019-12-31 13:24:41.618 INFO 50839 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=true]
2019-12-31 13:24:41.621 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=0], node=55e3c6ea-0b92-4570-8712-ca78168a6312, allReceived=true]
2019-12-31 13:24:41.621 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=5, minorTopVer=0]]
2019-12-31 13:24:41.621 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Exchanges merging performed in 0 ms.
2019-12-31 13:24:41.621 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]
2019-12-31 13:24:41.622 INFO 50839 --- [sys-#131] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity recalculation (on server left) performed in 1 ms.
2019-12-31 13:24:41.623 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Affinity changes (coordinator) applied in 2 ms.
2019-12-31 13:24:41.624 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Partitions validation performed in 1 ms.
2019-12-31 13:24:41.624 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Detecting lost partitions performed in 0 ms.
2019-12-31 13:24:41.626 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Preparing Full Message performed in 2 ms.
2019-12-31 13:24:41.626 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Full Message performed in 0 ms.
2019-12-31 13:24:41.626 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Full Message to all nodes performed in 0 ms.
2019-12-31 13:24:41.626 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null]
2019-12-31 13:24:41.626 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Detecting lost partitions performed in 0 ms.
2019-12-31 13:24:41.627 INFO 50839 --- [sys-#131] .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange [localNode=0f6adfb0-8641-487a-b8e9-ef0ace842b20, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834192, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], durationFromInit=30]
2019-12-31 13:24:41.629 INFO 50839 --- [exchange-worker-#103] .i.p.c.GridCachePartitionExchangeManager : Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=5, minorTopVer=0], force=false, evt=NODE_FAILED, node=51f0bedf-ffe8-4963-a2ae-6bed6955eb23]
2019-12-31 13:25:30.610 INFO 50839 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=0f6adfb0, uptime=00:02:00.006]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.2%, avg=0.17%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=205MB, free=79.08%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=2, qSize=0]
^-- System thread pool [active=0, idle=17, qSize=0]
2019-12-31 13:26:30.614 INFO 50839 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=0f6adfb0, uptime=00:03:00.014]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.07%, avg=0.14%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=215MB, free=78.05%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
2019-12-31 13:27:30.621 INFO 50839 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=0f6adfb0, uptime=00:04:00.021]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.03%, avg=0.12%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=221MB, free=77.43%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
2019-12-31 13:27:41.227 INFO 50839 --- [db-checkpoint-thread-#116] i.i.p.c.p.GridCacheDatabaseSharedManager : Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=3ms, reason='timeout']
2019-12-31 13:28:30.629 INFO 50839 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=0f6adfb0, uptime=00:05:00.023]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=228MB, free=76.74%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
Raw
ignite-persistence-data-node2.log
2019-12-31 13:23:42.230 INFO 51071 --- [main] poc.ignite.IgniteApp : Starting IgniteApp v0.0.1-SNAPSHOT on hdpdev6 with PID 51071 (/opt/ngs/ashishb/apps/ignite/ignite-persistence-data-node/ignite-persistence-data-node-0.0.1-SNAPSHOT.jar started by hdpusr in /opt/ngs/ashishb/apps/ignite/ignite-persistence-data-node)
2019-12-31 13:23:42.233 DEBUG 51071 --- [main] poc.ignite.IgniteApp : Running with Spring Boot v2.1.6.RELEASE, Spring v5.1.8.RELEASE
2019-12-31 13:23:42.233 INFO 51071 --- [main] poc.ignite.IgniteApp : No active profile set, falling back to default profiles: default
2019-12-31 13:23:42.964 INFO 51071 --- [main] poc.ignite.config.BeansConfig : ignite bean service
2019-12-31 13:23:43.179 WARN 51071 --- [main] o.apache.ignite.internal.util.typedef.G : Ignite work directory is not provided, automatically resolved to: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work
2019-12-31 13:23:43.326 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 2.7.6#20190911-sha1:21f7ca41
>>> 2019 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
2019-12-31 13:23:43.334 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Config URL: n/a
2019-12-31 13:23:43.355 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : IgniteConfiguration [igniteInstanceName=null, pubPoolSize=32, svcPoolSize=32, callbackPoolSize=32, stripedPoolSize=32, sysPoolSize=32, mgmtPoolSize=4, igfsPoolSize=32, dataStreamerPoolSize=32, utilityCachePoolSize=32, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=32, igniteHome=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/, igniteWorkDir=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@67b467e9, nodeId=55e3c6ea-0b92-4570-8712-ca78168a6312, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@33cb5951, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=42100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=10000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=16, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@365c30cc[Count = 1], stopping=false], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@701fc37a, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@7fad8c79, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@71a794e5, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=90000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=32, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=Default_Region, maxSize=268435456, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=[DataRegionConfiguration [name=Data_Region, maxSize=268435456, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0]], storagePath=/var/tmp/db, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=/var/tmp/db/wal, walArchivePath=/var/tmp/db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@1dd02175, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=32, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null]
2019-12-31 13:23:43.355 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Daemon mode: off
2019-12-31 13:23:43.359 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : OS: Linux 3.10.0-957.10.1.el7.x86_64 amd64
2019-12-31 13:23:43.359 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : OS user: hdpusr
2019-12-31 13:23:43.359 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : PID: 51071
2019-12-31 13:23:43.359 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Language runtime: Java Platform API Specification ver. 1.8
2019-12-31 13:23:43.360 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : VM information: Java(TM) SE Runtime Environment 1.8.0_191-b12 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.191-b12
2019-12-31 13:23:43.361 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : VM total memory: 0.96GB
2019-12-31 13:23:43.361 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Remote Management [restart: off, REST: on, JMX (remote: off)]
2019-12-31 13:23:43.361 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Logger: Slf4jLogger [impl=Logger[org.apache.ignite.internal.IgniteKernal], quiet=true]
2019-12-31 13:23:43.362 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : IGNITE_HOME=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/
2019-12-31 13:23:43.362 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : VM arguments: [-Xms1G, -Xmx1G, -Dlogging.file=/var/tmp/logs/ignite-persistence-data-node/ignite-persistence-data-node2]
2019-12-31 13:23:43.362 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting.
2019-12-31 13:23:43.372 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
2019-12-31 13:23:43.373 WARN 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Peer class loading is enabled (disable it in production for performance and deployment consistency reasons)
2019-12-31 13:23:43.373 WARN 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
2019-12-31 13:23:43.376 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : 3-rd party licenses can be found at: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin//libs/licenses
2019-12-31 13:23:43.376 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Local node user attribute [nodeName=data-node]
2019-12-31 13:23:43.450 INFO 51071 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : Configured plugins:
2019-12-31 13:23:43.450 INFO 51071 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : ^-- None
2019-12-31 13:23:43.450 INFO 51071 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor :
2019-12-31 13:23:43.452 INFO 51071 --- [main] o.a.i.i.p.failure.FailureProcessor : Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
2019-12-31 13:23:43.505 INFO 51071 --- [main] o.a.i.s.c.tcp.TcpCommunicationSpi : Successfully bound communication NIO server to TCP port [port=42101, locHost=0.0.0.0/0.0.0.0, selectorsCnt=16, selectorSpins=0, pairedConn=false]
2019-12-31 13:23:43.563 WARN 51071 --- [main] o.a.i.s.c.noop.NoopCheckpointSpi : Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
2019-12-31 13:23:43.594 WARN 51071 --- [main] o.a.i.i.m.c.GridCollisionManager : Collision resolution is disabled (all jobs will be activated upon arrival).
2019-12-31 13:23:43.599 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Security status [authentication=off, tls/ssl=off]
2019-12-31 13:23:43.684 INFO 51071 --- [main] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Successfully bound to TCP port [port=42501, localHost=0.0.0.0/0.0.0.0, locNodeId=55e3c6ea-0b92-4570-8712-ca78168a6312]
2019-12-31 13:23:44.694 INFO 51071 --- [main] o.a.i.i.p.c.p.f.PdsFoldersResolver : Unable to acquire lock to file [/var/tmp/db/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4], reason: Failed to acquire file lock during 1 sec, (locked by [0f6adfb0-8641-487a-b8e9-ef0ace842b20][]): /var/tmp/db/node00-65e3ea66-d894-4978-a20c-30d1caffe6e4/lock
2019-12-31 13:23:44.697 INFO 51071 --- [main] o.a.i.i.p.c.p.f.PdsFoldersResolver : Successfully created new persistent storage folder [/var/tmp/db/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b]
2019-12-31 13:23:44.698 INFO 51071 --- [main] o.a.i.i.p.c.p.f.PdsFoldersResolver : Consistent ID used for local node is [d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b] according to persistence data storage folders
2019-12-31 13:23:44.699 INFO 51071 --- [main] i.i.p.c.b.CacheObjectBinaryProcessorImpl : Resolved directory for serialized binary metadata: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work/binary_meta/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b
2019-12-31 13:23:44.926 INFO 51071 --- [main] o.a.i.i.p.c.p.file.FilePageStoreManager : Resolved page store work directory: /var/tmp/db/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b
2019-12-31 13:23:44.926 WARN 51071 --- [main] o.a.i.i.p.c.p.file.FilePageStoreManager : Persistence store directory is in the temp directory and may be cleaned.To avoid this set "IGNITE_HOME" environment variable properly or change location of persistence directories in data storage configuration (see DataStorageConfiguration#walPath, DataStorageConfiguration#walArchivePath, DataStorageConfiguration#storagePath properties). Current persistence store directory is: [/tmp]
2019-12-31 13:23:44.926 INFO 51071 --- [main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log work directory: /var/tmp/db/wal/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b
2019-12-31 13:23:44.927 INFO 51071 --- [main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log archive directory: /var/tmp/db/wal/archive/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b
2019-12-31 13:23:45.027 INFO 51071 --- [main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Started write-ahead log manager [mode=LOG_ONLY]
2019-12-31 13:23:45.034 WARN 51071 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files
2019-12-31 13:23:45.077 INFO 51071 --- [main] o.a.i.i.p.c.d.d.t.PartitionsEvictManager : Evict partition permits=8
2019-12-31 13:23:45.439 INFO 51071 --- [main] o.a.i.i.p.odbc.ClientListenerProcessor : Client connector processor has started on TCP port 10801
2019-12-31 13:23:45.494 INFO 51071 --- [main] o.a.i.i.p.r.p.tcp.GridTcpRestProtocol : Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11212]
2019-12-31 13:23:45.552 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Non-loopback local IPs: 172.17.241.80, 172.17.5.36
2019-12-31 13:23:45.552 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Enabled local MACs: 1866DAE84428, F4E9D4A04FD2
2019-12-31 13:23:45.561 INFO 51071 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=null, endMarker=null]
2019-12-31 13:23:45.575 INFO 51071 --- [main] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:23:45.576 INFO 51071 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:23:45.594 INFO 51071 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:23:45.595 INFO 51071 --- [main] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished applying WAL changes [updatesApplied=0, time=0ms]
2019-12-31 13:23:45.642 INFO 51071 --- [main] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Connection check threshold is calculated: 90000
2019-12-31 13:23:45.843 INFO 51071 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=48969]
2019-12-31 13:23:45.852 INFO 51071 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=48969]
2019-12-31 13:23:45.853 INFO 51071 --- [tcp-disco-sock-reader-#4] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/127.0.0.1:48969, rmtPort=48969]
2019-12-31 13:23:45.904 INFO 51071 --- [tcp-disco-msg-worker-#2] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : New next node [newNext=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778825871, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:23:45.918 INFO 51071 --- [disco-notifier-worker-#101] o.a.i.i.p.cache.mvcc.MvccProcessorImpl : Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778825871, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:23:45.972 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Performance suggestions for grid (fix if possible)
2019-12-31 13:23:45.972 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
2019-12-31 13:23:45.972 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
2019-12-31 13:23:45.972 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
2019-12-31 13:23:45.973 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
2019-12-31 13:23:45.973 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Speed up flushing of dirty pages by OS (alter vm.dirty_expire_centisecs parameter by setting to 500)
2019-12-31 13:23:45.973 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Reduce pages swapping ratio (set vm.swappiness=10.000000 or less)
2019-12-31 13:23:45.973 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning
2019-12-31 13:23:45.973 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal :
2019-12-31 13:23:45.974 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
2019-12-31 13:23:45.976 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : Data Regions Configured:
2019-12-31 13:23:45.976 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Default_Region [initSize=256.0 MiB, maxSize=256.0 MiB, persistence=false]
2019-12-31 13:23:45.976 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Data_Region [initSize=256.0 MiB, maxSize=256.0 MiB, persistence=true]
2019-12-31 13:23:45.977 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.7.6#20190911-sha1:21f7ca41c4348909e2fd26ccf59b5b2ce1f4474e
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 3.10.0-957.10.1.el7.x86_64 amd64
>>> CPU(s): 32
>>> Heap: 1.0GB
>>> VM name: 51071@hdpdev6
>>> Local node [ID=55E3C6EA-0B92-4570-8712-CA78168A6312, order=2, clientMode=false]
>>> Local node addresses: [hdpdev6/127.0.0.1, hdpdev6_oob.nseroot.com/172.17.241.80, /172.17.5.36]
>>> Local ports: TCP:10801 TCP:11212 TCP:42101 TCP:42501
2019-12-31 13:23:45.977 INFO 51071 --- [main] org.apache.ignite.internal.IgniteKernal : >>> Ignite cluster is not active (limited functionality available). Use control.(sh|bat) script or IgniteCluster interface to activate.
2019-12-31 13:23:45.979 INFO 51071 --- [main] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=2, locNode=55e3c6ea, servers=2, clients=0, state=INACTIVE, CPUs=32, offheap=1.0GB, heap=2.0GB]
2019-12-31 13:23:46.048 INFO 51071 --- [main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'
2019-12-31 13:23:46.169 INFO 51071 --- [main] poc.ignite.IgniteApp : Started IgniteApp in 4.489 seconds (JVM running for 4.922)
2019-12-31 13:23:46.170 INFO 51071 --- [main] poc.ignite.IgniteApp : run service
2019-12-31 13:23:54.212 INFO 51071 --- [tcp-disco-msg-worker-#2] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : New next node [newNext=TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=0, intOrder=3, lastExchangeTime=1577778834199, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:23:54.277 INFO 51071 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Added new node to topology: TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834199, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]
2019-12-31 13:23:54.278 INFO 51071 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=3, locNode=55e3c6ea, servers=3, clients=0, state=INACTIVE, CPUs=32, offheap=1.5GB, heap=3.0GB]
2019-12-31 13:24:40.149 INFO 51071 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/172.17.5.36, rmtPort=37328]
2019-12-31 13:24:40.149 INFO 51071 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/172.17.5.36, rmtPort=37328]
2019-12-31 13:24:40.150 INFO 51071 --- [tcp-disco-sock-reader-#5] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/172.17.5.36:37328, rmtPort=37328]
2019-12-31 13:24:40.273 INFO 51071 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Added new node to topology: TcpDiscoveryNode [id=6ef2878f-6b05-4756-9417-8bcfcb9a1701, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:0, /127.0.0.1:0, hdpdev6_oob.nseroot.com/172.17.241.80:0], discPort=0, order=4, intOrder=4, lastExchangeTime=1577778880204, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=true]
2019-12-31 13:24:40.273 INFO 51071 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=4, locNode=55e3c6ea, servers=3, clients=1, state=INACTIVE, CPUs=32, offheap=1.5GB, heap=4.0GB]
2019-12-31 13:24:40.809 INFO 51071 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Received activate request with BaselineTopology[id=0]
2019-12-31 13:24:40.811 INFO 51071 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Started state transition: true
2019-12-31 13:24:40.837 INFO 51071 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], mvccCrd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=DISCOVERY_CUSTOM_EVT, evtNode=0f6adfb0-8641-487a-b8e9-ef0ace842b20, customEvt=ChangeGlobalStateMessage [id=15a02fa5f61-fb354dfe-aed5-4d86-8fc8-cc9dce3672e6, reqId=3b5c20df-697e-423d-9208-6a02d536305b, initiatingNodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=1818609622, branchingType='New BaselineTopology', baselineNodes=[d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b, 127.0.0.1,172.17.241.80,172.17.5.36:42502, 65e3ea66-d894-4978-a20c-30d1caffe6e4]], forceChangeBaselineTopology=false, timestamp=1577778880793], allowMerge=false]
2019-12-31 13:24:40.837 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Start activation process [nodeId=55e3c6ea-0b92-4570-8712-ca78168a6312, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:40.838 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.file.FilePageStoreManager : Resolved page store work directory: /var/tmp/db/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b
2019-12-31 13:24:40.838 WARN 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.file.FilePageStoreManager : Persistence store directory is in the temp directory and may be cleaned.To avoid this set "IGNITE_HOME" environment variable properly or change location of persistence directories in data storage configuration (see DataStorageConfiguration#walPath, DataStorageConfiguration#walArchivePath, DataStorageConfiguration#storagePath properties). Current persistence store directory is: [/tmp]
2019-12-31 13:24:40.838 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log work directory: /var/tmp/db/wal/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b
2019-12-31 13:24:40.838 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log archive directory: /var/tmp/db/wal/archive/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b
2019-12-31 13:24:40.840 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Started write-ahead log manager [mode=LOG_ONLY]
2019-12-31 13:24:40.853 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:24:40.857 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:24:40.871 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=256.0 MiB, pages=63520, tableSize=4.9 MiB, checkpointBuffer=256.0 MiB]
2019-12-31 13:24:40.876 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24800, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB]
2019-12-31 13:24:40.892 INFO 51071 --- [exchange-worker-#103] org.apache.ignite.cache.msg : Components activation performed in 55 ms.
2019-12-31 13:24:40.910 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=null, endMarker=null]
2019-12-31 13:24:40.911 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:24:40.914 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resuming logging to WAL segment [file=/var/tmp/db/wal/node01-d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b/0000000000000000.wal, offset=0, ver=2]
2019-12-31 13:24:40.949 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.GridClusterStateProcessor : Writing BaselineTopology[id=0]
2019-12-31 13:24:40.969 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Binary recovery performed in 77 ms.
2019-12-31 13:24:40.974 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Updating caches registry performed in 5 ms.
2019-12-31 13:24:41.115 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false], encryptionEnabled=false]
2019-12-31 13:24:41.116 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Caches starting performed in 142 ms.
2019-12-31 13:24:41.120 INFO 51071 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity initialization for started caches performed in 4 ms.
2019-12-31 13:24:41.120 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Successfully activated caches [nodeId=55e3c6ea-0b92-4570-8712-ca78168a6312, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.129 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL]
2019-12-31 13:24:41.130 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=null, endMarker=null]
2019-12-31 13:24:41.131 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:24:41.191 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Restoring partition state for local groups [cntPartStateWal=0, lastCheckpointId=00000000-0000-0000-0000-000000000000]
2019-12-31 13:24:41.195 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished restoring partition state for local groups [cntProcessed=100, cntPartStateWal=0, time=10ms]
2019-12-31 13:24:41.195 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished applying WAL changes [updatesApplied=0, time=70ms]
2019-12-31 13:24:41.195 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Logical recovery performed in 65 ms.
2019-12-31 13:24:41.238 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpointer initilialzation performed in 14 ms.
2019-12-31 13:24:41.244 INFO 51071 --- [db-checkpoint-thread-#112] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint started [checkpointId=d51d2058-2d2e-4f38-b5c5-35587f4c2408, startPtr=FileWALPointer [idx=0, fileOff=240085, len=1947], checkpointLockWait=0ms, checkpointLockHoldTime=12ms, walCpRecordFsyncDuration=1ms, pages=25, reason='node started']
2019-12-31 13:24:41.258 INFO 51071 --- [db-checkpoint-thread-#112] i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished [cpId=d51d2058-2d2e-4f38-b5c5-35587f4c2408, pages=25, markPos=FileWALPointer [idx=0, fileOff=240085, len=1947], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=19ms, pagesWrite=8ms, fsync=5ms, total=32ms]
2019-12-31 13:24:41.261 INFO 51071 --- [grid-nio-worker-tcp-comm-0-#72] o.a.i.s.c.tcp.TcpCommunicationSpi : Established outgoing communication connection [locAddr=/127.0.0.1:49484, rmtAddr=/127.0.0.1:42100]
2019-12-31 13:24:41.264 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Single Message performed in 26 ms.
2019-12-31 13:24:41.265 INFO 51071 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], crd=false]
2019-12-31 13:24:41.502 WARN 51071 --- [tcp-disco-msg-worker-#2] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=8b142fa5f61-55e3c6ea-0b92-4570-8712-ca78168a6312, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834199, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=8b142fa5f61-55e3c6ea-0b92-4570-8712-ca78168a6312, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, order=3, addr=[127.0.0.1, 172.17.241.80, 172.17.5.36], daemon=false]]]
2019-12-31 13:24:41.502 INFO 51071 --- [tcp-disco-msg-worker-#2] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : New next node [newNext=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778825871, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:24:41.511 WARN 51071 --- [tcp-disco-msg-worker-#2] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi'
2019-12-31 13:24:41.522 WARN 51071 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Node FAILED: TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834199, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]
2019-12-31 13:24:41.523 INFO 51071 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=5, locNode=55e3c6ea, servers=2, clients=1, state=ACTIVE, CPUs=32, offheap=1.0GB, heap=3.0GB]
2019-12-31 13:24:41.523 INFO 51071 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : ^-- Baseline [id=0, size=3, online=2, offline=1]
2019-12-31 13:24:41.574 INFO 51071 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Received state change finish message: true
2019-12-31 13:24:41.587 INFO 51071 --- [sys-#121] .i.p.c.GridCachePartitionExchangeManager : Delay process full message without exchange id (there is exchange in progress) [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20]
2019-12-31 13:24:41.590 INFO 51071 --- [sys-#120] .c.d.d.p.GridDhtPartitionsExchangeFuture : Received full message, will finish exchange [node=0f6adfb0-8641-487a-b8e9-ef0ace842b20, resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.592 INFO 51071 --- [sys-#120] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity applying from full message performed in 2 ms.
2019-12-31 13:24:41.592 INFO 51071 --- [sys-#120] .c.d.d.p.GridDhtPartitionsExchangeFuture : Affinity changes applied in 2 ms.
2019-12-31 13:24:41.594 INFO 51071 --- [sys-#120] .c.d.d.p.GridDhtPartitionsExchangeFuture : Full map updating for 1 groups performed in 2 ms.
2019-12-31 13:24:41.594 INFO 51071 --- [sys-#120] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], err=null]
2019-12-31 13:24:41.602 INFO 51071 --- [sys-#120] o.a.i.i.p.cache.GridCacheProcessor : Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=55e3c6ea-0b92-4570-8712-ca78168a6312
2019-12-31 13:24:41.605 INFO 51071 --- [sys-#120] .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange [localNode=55e3c6ea-0b92-4570-8712-ca78168a6312, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778825871, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], durationFromInit=776]
2019-12-31 13:24:41.608 INFO 51071 --- [exchange-worker-#103] .i.p.c.GridCachePartitionExchangeManager : Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=4, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, node=0f6adfb0-8641-487a-b8e9-ef0ace842b20]
2019-12-31 13:24:41.609 INFO 51071 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, customEvt=null, allowMerge=true]
2019-12-31 13:24:41.612 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
2019-12-31 13:24:41.613 INFO 51071 --- [sys-#122] o.a.i.i.p.c.GridClusterStateProcessor : Successfully performed final activation steps [nodeId=55e3c6ea-0b92-4570-8712-ca78168a6312, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.617 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778825871, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]]
2019-12-31 13:24:41.617 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL]
2019-12-31 13:24:41.618 INFO 51071 --- [exchange-worker-#103] i.i.p.c.p.GridCacheDatabaseSharedManager : Logical recovery performed in 0 ms.
2019-12-31 13:24:41.619 INFO 51071 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Single Message performed in 1 ms.
2019-12-31 13:24:41.619 INFO 51071 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false]
2019-12-31 13:24:41.628 INFO 51071 --- [sys-#125] .c.d.d.p.GridDhtPartitionsExchangeFuture : Received full message, will finish exchange [node=0f6adfb0-8641-487a-b8e9-ef0ace842b20, resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]
2019-12-31 13:24:41.629 INFO 51071 --- [sys-#125] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity applying from full message performed in 1 ms.
2019-12-31 13:24:41.630 INFO 51071 --- [sys-#125] .c.d.d.p.GridDhtPartitionsExchangeFuture : Affinity changes applied in 2 ms.
2019-12-31 13:24:41.631 INFO 51071 --- [sys-#125] .c.d.d.p.GridDhtPartitionsExchangeFuture : Full map updating for 1 groups performed in 1 ms.
2019-12-31 13:24:41.631 INFO 51071 --- [sys-#125] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null]
2019-12-31 13:24:41.631 INFO 51071 --- [sys-#125] .c.d.d.p.GridDhtPartitionsExchangeFuture : Detecting lost partitions performed in 0 ms.
2019-12-31 13:24:41.632 INFO 51071 --- [sys-#125] .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange [localNode=55e3c6ea-0b92-4570-8712-ca78168a6312, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778834199, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], durationFromInit=20]
2019-12-31 13:24:41.633 INFO 51071 --- [exchange-worker-#103] .i.p.c.GridCachePartitionExchangeManager : Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=5, minorTopVer=0], force=false, evt=NODE_FAILED, node=51f0bedf-ffe8-4963-a2ae-6bed6955eb23]
2019-12-31 13:24:45.974 INFO 51071 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=55e3c6ea, uptime=00:01:00.002]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.07%, avg=0.28%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=201MB, free=79.5%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=2, qSize=0]
^-- System thread pool [active=0, idle=13, qSize=0]
2019-12-31 13:24:55.936 INFO 51071 --- [ignite-update-notifier-timer] o.a.i.i.p.cluster.GridUpdateNotifier : Update status is not available.
2019-12-31 13:25:45.988 INFO 51071 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=55e3c6ea, uptime=00:02:00.019]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.03%, avg=0.17%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=210MB, free=78.53%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=5, qSize=0]
2019-12-31 13:26:45.992 INFO 51071 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=55e3c6ea, uptime=00:03:00.027]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0%, avg=0.13%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=219MB, free=77.67%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
2019-12-31 13:27:41.233 INFO 51071 --- [db-checkpoint-thread-#112] i.i.p.c.p.GridCacheDatabaseSharedManager : Skipping checkpoint (no pages were modified) [checkpointLockWait=0ms, checkpointLockHoldTime=3ms, reason='timeout']
2019-12-31 13:27:45.995 INFO 51071 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=55e3c6ea, uptime=00:04:00.027]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.03%, avg=0.11%, GC=0%]
^-- PageMemory [pages=25]
^-- Heap [used=227MB, free=76.77%, comm=981MB]
^-- Off-heap [used=0MB, free=99.99%, comm=556MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- metastoreMemPlc region [used=0MB, free=99.92%, comm=100MB]
^-- Data_Region region [used=0MB, free=100%, comm=256MB]
^-- Default_Region region [used=0MB, free=100%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=0MB]
^-- sysMemPlc region [used=0MB]
^-- metastoreMemPlc region [used=unknown]
^-- Data_Region region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
Raw
ignite-persistence-worker-node.log
2019-12-31 13:23:51.912 INFO 51266 --- [main] poc.ignite.IgniteApp : Starting IgniteApp v0.0.1-SNAPSHOT on hdpdev6 with PID 51266 (/opt/ngs/ashishb/apps/ignite/ignite-persistence-worker-node/ignite-persistence-worker-node-0.0.1-SNAPSHOT.jar started by hdpusr in /opt/ngs/ashishb/apps/ignite/ignite-persistence-worker-node)
2019-12-31 13:23:51.915 DEBUG 51266 --- [main] poc.ignite.IgniteApp : Running with Spring Boot v2.1.6.RELEASE, Spring v5.1.8.RELEASE
2019-12-31 13:23:51.915 INFO 51266 --- [main] poc.ignite.IgniteApp : No active profile set, falling back to default profiles: default
2019-12-31 13:23:52.646 INFO 51266 --- [main] poc.ignite.config.BeansConfig : ignite bean service
2019-12-31 13:23:52.863 WARN 51266 --- [main] o.apache.ignite.internal.util.typedef.G : Ignite work directory is not provided, automatically resolved to: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work
2019-12-31 13:23:53.018 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 2.7.6#20190911-sha1:21f7ca41
>>> 2019 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
2019-12-31 13:23:53.026 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Config URL: n/a
2019-12-31 13:23:53.047 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : IgniteConfiguration [igniteInstanceName=null, pubPoolSize=32, svcPoolSize=32, callbackPoolSize=32, stripedPoolSize=32, sysPoolSize=32, mgmtPoolSize=4, igfsPoolSize=32, dataStreamerPoolSize=32, utilityCachePoolSize=32, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=32, igniteHome=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/, igniteWorkDir=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@704d6e83, nodeId=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@17776a8, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=42100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=10000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=16, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@69a10787[Count = 1], stopping=false], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@2d127a61, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@70beb599, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@4e41089d, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=90000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=32, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=Default_Region, maxSize=268435456, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=[DataRegionConfiguration [name=Data_Region, maxSize=268435456, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0]], storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@7fad8c79, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=32, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null]
2019-12-31 13:23:53.047 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Daemon mode: off
2019-12-31 13:23:53.051 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : OS: Linux 3.10.0-957.10.1.el7.x86_64 amd64
2019-12-31 13:23:53.051 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : OS user: hdpusr
2019-12-31 13:23:53.051 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : PID: 51266
2019-12-31 13:23:53.052 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Language runtime: Java Platform API Specification ver. 1.8
2019-12-31 13:23:53.052 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : VM information: Java(TM) SE Runtime Environment 1.8.0_191-b12 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.191-b12
2019-12-31 13:23:53.053 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : VM total memory: 0.96GB
2019-12-31 13:23:53.053 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Remote Management [restart: off, REST: on, JMX (remote: off)]
2019-12-31 13:23:53.053 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Logger: Slf4jLogger [impl=Logger[org.apache.ignite.internal.IgniteKernal], quiet=true]
2019-12-31 13:23:53.054 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : IGNITE_HOME=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/
2019-12-31 13:23:53.054 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : VM arguments: [-Xms1G, -Xmx1G]
2019-12-31 13:23:53.054 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting.
2019-12-31 13:23:53.067 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
2019-12-31 13:23:53.068 WARN 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Peer class loading is enabled (disable it in production for performance and deployment consistency reasons)
2019-12-31 13:23:53.068 WARN 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
2019-12-31 13:23:53.073 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : 3-rd party licenses can be found at: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin//libs/licenses
2019-12-31 13:23:53.073 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Local node user attribute [nodeName=worker-node]
2019-12-31 13:23:53.150 INFO 51266 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : Configured plugins:
2019-12-31 13:23:53.150 INFO 51266 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : ^-- None
2019-12-31 13:23:53.151 INFO 51266 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor :
2019-12-31 13:23:53.153 INFO 51266 --- [main] o.a.i.i.p.failure.FailureProcessor : Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
2019-12-31 13:23:53.205 INFO 51266 --- [main] o.a.i.s.c.tcp.TcpCommunicationSpi : Successfully bound communication NIO server to TCP port [port=42102, locHost=0.0.0.0/0.0.0.0, selectorsCnt=16, selectorSpins=0, pairedConn=false]
2019-12-31 13:23:53.270 WARN 51266 --- [main] o.a.i.s.c.noop.NoopCheckpointSpi : Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
2019-12-31 13:23:53.303 WARN 51266 --- [main] o.a.i.i.m.c.GridCollisionManager : Collision resolution is disabled (all jobs will be activated upon arrival).
2019-12-31 13:23:53.309 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Security status [authentication=off, tls/ssl=off]
2019-12-31 13:23:53.559 WARN 51266 --- [main] i.p.c.p.IgniteCacheDatabaseSharedManager : DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files
2019-12-31 13:23:53.580 INFO 51266 --- [main] o.a.i.i.p.c.d.d.t.PartitionsEvictManager : Evict partition permits=8
2019-12-31 13:23:53.832 INFO 51266 --- [main] o.a.i.i.p.odbc.ClientListenerProcessor : Client connector processor has started on TCP port 10803
2019-12-31 13:23:53.897 INFO 51266 --- [main] o.a.i.i.p.r.p.tcp.GridTcpRestProtocol : Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11214]
2019-12-31 13:23:54.056 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Non-loopback local IPs: 172.17.241.80, 172.17.5.36
2019-12-31 13:23:54.057 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Enabled local MACs: 1866DAE84428, F4E9D4A04FD2
2019-12-31 13:23:54.095 INFO 51266 --- [main] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Connection check threshold is calculated: 90000
2019-12-31 13:23:54.099 INFO 51266 --- [main] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Successfully bound to TCP port [port=42502, localHost=0.0.0.0/0.0.0.0, locNodeId=51f0bedf-ffe8-4963-a2ae-6bed6955eb23]
2019-12-31 13:23:54.213 INFO 51266 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33924]
2019-12-31 13:23:54.227 INFO 51266 --- [tcp-disco-srvr-#3] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33924]
2019-12-31 13:23:54.228 INFO 51266 --- [tcp-disco-sock-reader-#4] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Started serving remote node connection [rmtAddr=/127.0.0.1:33924, rmtPort=33924]
2019-12-31 13:23:54.270 INFO 51266 --- [tcp-disco-msg-worker-#2] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : New next node [newNext=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778834236, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:23:54.283 INFO 51266 --- [disco-notifier-worker-#101] o.a.i.i.p.cache.mvcc.MvccProcessorImpl : Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778834236, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:23:54.342 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Performance suggestions for grid (fix if possible)
2019-12-31 13:23:54.343 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
2019-12-31 13:23:54.343 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
2019-12-31 13:23:54.344 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
2019-12-31 13:23:54.344 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
2019-12-31 13:23:54.344 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Speed up flushing of dirty pages by OS (alter vm.dirty_expire_centisecs parameter by setting to 500)
2019-12-31 13:23:54.344 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Reduce pages swapping ratio (set vm.swappiness=10.000000 or less)
2019-12-31 13:23:54.344 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning
2019-12-31 13:23:54.344 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal :
2019-12-31 13:23:54.345 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
2019-12-31 13:23:54.347 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : Data Regions Configured:
2019-12-31 13:23:54.348 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Default_Region [initSize=256.0 MiB, maxSize=256.0 MiB, persistence=false]
2019-12-31 13:23:54.348 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Data_Region [initSize=256.0 MiB, maxSize=256.0 MiB, persistence=false]
2019-12-31 13:23:54.349 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.7.6#20190911-sha1:21f7ca41c4348909e2fd26ccf59b5b2ce1f4474e
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 3.10.0-957.10.1.el7.x86_64 amd64
>>> CPU(s): 32
>>> Heap: 1.0GB
>>> VM name: 51266@hdpdev6
>>> Local node [ID=51F0BEDF-FFE8-4963-A2AE-6BED6955EB23, order=3, clientMode=false]
>>> Local node addresses: [hdpdev6/127.0.0.1, hdpdev6_oob.nseroot.com/172.17.241.80, /172.17.5.36]
>>> Local ports: TCP:10803 TCP:11214 TCP:42102 TCP:42502
2019-12-31 13:23:54.349 INFO 51266 --- [main] org.apache.ignite.internal.IgniteKernal : >>> Ignite cluster is not active (limited functionality available). Use control.(sh|bat) script or IgniteCluster interface to activate.
2019-12-31 13:23:54.351 INFO 51266 --- [main] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=3, locNode=51f0bedf, servers=3, clients=0, state=INACTIVE, CPUs=32, offheap=1.5GB, heap=3.0GB]
2019-12-31 13:23:54.431 INFO 51266 --- [main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'
2019-12-31 13:23:54.565 INFO 51266 --- [main] poc.ignite.IgniteApp : Started IgniteApp in 3.182 seconds (JVM running for 3.622)
2019-12-31 13:23:54.566 INFO 51266 --- [main] poc.ignite.IgniteApp : run service
2019-12-31 13:24:40.281 INFO 51266 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Added new node to topology: TcpDiscoveryNode [id=6ef2878f-6b05-4756-9417-8bcfcb9a1701, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:0, /127.0.0.1:0, hdpdev6_oob.nseroot.com/172.17.241.80:0], discPort=0, order=4, intOrder=4, lastExchangeTime=1577778880225, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=true]
2019-12-31 13:24:40.282 INFO 51266 --- [disco-event-worker-#102] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=4, locNode=51f0bedf, servers=3, clients=1, state=INACTIVE, CPUs=32, offheap=1.5GB, heap=4.0GB]
2019-12-31 13:24:40.815 INFO 51266 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Received activate request with BaselineTopology[id=0]
2019-12-31 13:24:40.817 INFO 51266 --- [disco-notifier-worker-#101] o.a.i.i.p.c.GridClusterStateProcessor : Started state transition: true
2019-12-31 13:24:40.850 INFO 51266 --- [exchange-worker-#103] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], mvccCrd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=DISCOVERY_CUSTOM_EVT, evtNode=0f6adfb0-8641-487a-b8e9-ef0ace842b20, customEvt=ChangeGlobalStateMessage [id=15a02fa5f61-fb354dfe-aed5-4d86-8fc8-cc9dce3672e6, reqId=3b5c20df-697e-423d-9208-6a02d536305b, initiatingNodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=1818609622, branchingType='New BaselineTopology', baselineNodes=[d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b, 127.0.0.1,172.17.241.80,172.17.5.36:42502, 65e3ea66-d894-4978-a20c-30d1caffe6e4]], forceChangeBaselineTopology=false, timestamp=1577778880793], allowMerge=false]
2019-12-31 13:24:40.851 INFO 51266 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Start activation process [nodeId=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:40.874 INFO 51266 --- [exchange-worker-#103] org.apache.ignite.cache.msg : Components activation performed in 23 ms.
2019-12-31 13:24:40.877 INFO 51266 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Updating caches registry performed in 2 ms.
2019-12-31 13:24:41.017 INFO 51266 --- [exchange-worker-#103] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false], encryptionEnabled=false]
2019-12-31 13:24:41.018 INFO 51266 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Caches starting performed in 140 ms.
2019-12-31 13:24:41.022 INFO 51266 --- [exchange-worker-#103] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity initialization for started caches performed in 4 ms.
2019-12-31 13:24:41.022 INFO 51266 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Successfully activated caches [nodeId=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, client=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.030 INFO 51266 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL]
2019-12-31 13:24:41.075 ERROR 51266 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Failed to reinitialize local partitions (rebalancing will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=ChangeGlobalStateMessage [id=15a02fa5f61-fb354dfe-aed5-4d86-8fc8-cc9dce3672e6, reqId=3b5c20df-697e-423d-9208-6a02d536305b, initiatingNodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=1818609622, branchingType='New BaselineTopology', baselineNodes=[d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b, 127.0.0.1,172.17.241.80,172.17.5.36:42502, 65e3ea66-d894-4978-a20c-30d1caffe6e4]], forceChangeBaselineTopology=false, timestamp=1577778880793], affTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778834236, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], topVer=4, nodeId8=51f0bedf, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1577778880819]], nodeId=0f6adfb0, evt=DISCOVERY_CUSTOM_EVT]
java.lang.NullPointerException: null
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:224)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.getOrCreatePartition(GridDhtPartitionTopologyImpl.java:853)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.initPartitions(GridDhtPartitionTopologyImpl.java:406)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.java:585)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1473)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:809)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
2019-12-31 13:24:41.076 INFO 51266 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], resVer=null, err=java.lang.NullPointerException]
2019-12-31 13:24:41.082 INFO 51266 --- [exchange-worker-#103] .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange [localNode=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778834236, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], done=true], topVer=null, durationFromInit=242]
2019-12-31 13:24:41.086 ERROR 51266 --- [exchange-worker-#103] .i.p.c.GridCachePartitionExchangeManager : Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778834236, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], topVer=4, nodeId8=51f0bedf, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1577778880819]], crd=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778834236, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778834236, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], topVer=4, nodeId8=51f0bedf, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1577778880819]], nodeId=0f6adfb0, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=654574526], init=false, lastVer=null, partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], futures=[ExplicitLockReleaseFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], futures=[]], DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], futures=[]], LocalTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], futures=[]], AllTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], futures=[RemoteTxReleaseFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], futures=[]]]]]], exchActions=null, affChangeMsg=null, initTs=1577778880839, centralizedAff=false, forceAffReassignment=true, exchangeLocE=null, cacheChangeFailureMsgSent=false, done=true, state=SRV, registerCachesFuture=GridFinishedFuture [resFlag=2], partitionsSent=false, partitionsReceived=false, delayedLatestMsg=null, afterLsnrCompleteFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1520043094], evtLatch=0, remaining=[0f6adfb0-8641-487a-b8e9-ef0ace842b20, 55e3c6ea-0b92-4570-8712-ca78168a6312], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=java.lang.NullPointerException, hash=922137950]]
org.apache.ignite.IgniteCheckedException: null
at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7432)
at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:261)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:209)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2709)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:224)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.getOrCreatePartition(GridDhtPartitionTopologyImpl.java:853)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.initPartitions(GridDhtPartitionTopologyImpl.java:406)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.java:585)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1473)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:809)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
... 3 common frames omitted
2019-12-31 13:24:41.087 ERROR 51266 --- [exchange-worker-#103] ROOT : Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: null]]
org.apache.ignite.IgniteCheckedException: null
at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7432)
at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:261)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:209)
at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2709)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2553)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.<init>(GridDhtLocalPartition.java:224)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.getOrCreatePartition(GridDhtPartitionTopologyImpl.java:853)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.initPartitions(GridDhtPartitionTopologyImpl.java:406)
at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.java:585)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1473)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:809)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2681)
... 3 common frames omitted
2019-12-31 13:24:41.088 WARN 51266 --- [exchange-worker-#103] o.a.i.i.p.failure.FailureProcessor : No deadlocked threads detected.
2019-12-31 13:24:41.138 WARN 51266 --- [exchange-worker-#103] o.a.i.i.p.failure.FailureProcessor : Thread dump at 2019/12/31 13:24:41 IST
Thread [name="sys-#107", id=144, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e5bf76b, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-#106", id=143, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e5bf76b, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-#105", id=142, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e5bf76b, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-#104", id=141, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e5bf76b, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="DestroyJavaVM", id=140, state=RUNNABLE, blockCnt=0, waitCnt=0]
Thread [name="session-timeout-worker-#94", id=123, state=TIMED_WAITING, blockCnt=0, waitCnt=47]
at java.lang.Thread.sleep(Native Method)
at o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="exchange-worker-#103", id=137, state=RUNNABLE, blockCnt=0, waitCnt=10]
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1368)
at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:128)
- locked o.a.i.i.processors.failure.FailureProcessor@63e9b1f0
at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:104)
at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2569)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="ignite-update-notifier-timer", id=136, state=TIMED_WAITING, blockCnt=1, waitCnt=39]
at java.lang.Thread.sleep(Native Method)
at o.a.i.i.processors.cluster.ClusterProcessor$UpdateNotifierTimerTask.safeRun(ClusterProcessor.java:686)
at o.a.i.i.util.GridTimerTask.run(GridTimerTask.java:34)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Thread [name="upd-ver-checker", id=135, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
at java.lang.Thread.sleep(Native Method)
at o.a.i.i.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115)
at java.lang.Thread.run(Thread.java:748)
Thread [name="disco-event-worker-#102", id=134, state=WAITING, blockCnt=0, waitCnt=52]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@695bb16f, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2856)
at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2825)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="tcp-disco-sock-reader-#4", id=133, state=RUNNABLE, blockCnt=0, waitCnt=0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked java.io.BufferedInputStream@185fbfac
at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:43)
at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:137)
at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10113)
at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6246)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Thread [name="tcp-disco-srvr-#3", id=132, state=RUNNABLE, blockCnt=0, waitCnt=0]
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Thread [name="tcp-disco-msg-worker-#2", id=131, state=TIMED_WAITING, blockCnt=2, waitCnt=4687]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@193ee363, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7183)
at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Thread [name="disco-notifier-worker-#101", id=130, state=WAITING, blockCnt=0, waitCnt=55]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3776b785, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2660)
at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2705)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-data-loader-flusher-#100", id=129, state=WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@312a8cd, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-rest-3-#98", id=127, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@66d5c5dc
- locked java.util.Collections$UnmodifiableSet@81e079e
- locked sun.nio.ch.EPollSelectorImpl@a565387
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-rest-2-#97", id=126, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@1c14d6c0
- locked java.util.Collections$UnmodifiableSet@54a71bbb
- locked sun.nio.ch.EPollSelectorImpl@2c7dd446
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-rest-1-#96", id=125, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@148f7025
- locked java.util.Collections$UnmodifiableSet@2e1706a2
- locked sun.nio.ch.EPollSelectorImpl@4281ba18
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-rest-0-#95", id=124, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@9c99ea
- locked java.util.Collections$UnmodifiableSet@6ffdcf84
- locked sun.nio.ch.EPollSelectorImpl@7949f6cf
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="nio-acceptor-tcp-rest-#99", id=128, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$3@36d7c284
- locked java.util.Collections$UnmodifiableSet@52605e7d
- locked sun.nio.ch.EPollSelectorImpl@53602a5a
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-client-listener-3-#92", id=121, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@65295177
- locked java.util.Collections$UnmodifiableSet@6d626cb4
- locked sun.nio.ch.EPollSelectorImpl@3d2144d3
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-client-listener-2-#91", id=120, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@6f419bd7
- locked java.util.Collections$UnmodifiableSet@404ccda1
- locked sun.nio.ch.EPollSelectorImpl@3e350ed0
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-client-listener-1-#90", id=119, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@985d219
- locked java.util.Collections$UnmodifiableSet@6af97fdd
- locked sun.nio.ch.EPollSelectorImpl@4cfe96e6
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-client-listener-0-#89", id=118, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@5a2b5227
- locked java.util.Collections$UnmodifiableSet@7d994661
- locked sun.nio.ch.EPollSelectorImpl@5c3fd998
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="nio-acceptor-client-listener-#93", id=122, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$3@64ce2d5f
- locked java.util.Collections$UnmodifiableSet@64f4f18b
- locked sun.nio.ch.EPollSelectorImpl@53e9093a
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="tcp-comm-worker-#1", id=117, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7138e9c6, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Thread [name="grid-nio-worker-tcp-comm-15-#87", id=112, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@6489966b
- locked java.util.Collections$UnmodifiableSet@27b24bd8
- locked sun.nio.ch.EPollSelectorImpl@3c0936cd
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-14-#86", id=111, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@ac99e53
- locked java.util.Collections$UnmodifiableSet@22db7baf
- locked sun.nio.ch.EPollSelectorImpl@178208a7
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-13-#85", id=110, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@6248ebc4
- locked java.util.Collections$UnmodifiableSet@24b5dad8
- locked sun.nio.ch.EPollSelectorImpl@5dcacbe2
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-12-#84", id=109, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@5a0a27cc
- locked java.util.Collections$UnmodifiableSet@7f39e182
- locked sun.nio.ch.EPollSelectorImpl@755087a7
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-11-#83", id=108, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@7e4ed734
- locked java.util.Collections$UnmodifiableSet@75f1ab76
- locked sun.nio.ch.EPollSelectorImpl@45f471bb
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-10-#82", id=107, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@34e8bb1d
- locked java.util.Collections$UnmodifiableSet@3c172dc3
- locked sun.nio.ch.EPollSelectorImpl@44c58b2c
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-9-#81", id=106, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@22da43a6
- locked java.util.Collections$UnmodifiableSet@531b34b3
- locked sun.nio.ch.EPollSelectorImpl@56677226
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-8-#80", id=105, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@3e130f2d
- locked java.util.Collections$UnmodifiableSet@4ea4ac3a
- locked sun.nio.ch.EPollSelectorImpl@4490a7f1
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-7-#79", id=104, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@290b1b07
- locked java.util.Collections$UnmodifiableSet@f47136c
- locked sun.nio.ch.EPollSelectorImpl@6469abc2
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-6-#78", id=103, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@25073791
- locked java.util.Collections$UnmodifiableSet@54a5c315
- locked sun.nio.ch.EPollSelectorImpl@634e669e
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-5-#77", id=102, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@4ad0e68e
- locked java.util.Collections$UnmodifiableSet@5677fbfa
- locked sun.nio.ch.EPollSelectorImpl@2cb0374a
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-4-#76", id=101, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@3cdaced4
- locked java.util.Collections$UnmodifiableSet@7173ab57
- locked sun.nio.ch.EPollSelectorImpl@18b236a8
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-3-#75", id=100, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@35954893
- locked java.util.Collections$UnmodifiableSet@5f539c9b
- locked sun.nio.ch.EPollSelectorImpl@3316bd35
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-2-#74", id=99, state=RUNNABLE, blockCnt=0, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@3a18da89
- locked java.util.Collections$UnmodifiableSet@25d6cc89
- locked sun.nio.ch.EPollSelectorImpl@66222bec
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-1-#73", id=98, state=RUNNABLE, blockCnt=1, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@60dbdd08
- locked java.util.Collections$UnmodifiableSet@1c6b9f08
- locked sun.nio.ch.EPollSelectorImpl@fcaba88
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-nio-worker-tcp-comm-0-#72", id=97, state=RUNNABLE, blockCnt=2, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked o.a.i.i.util.nio.SelectedSelectionKeySet@78409dd6
- locked java.util.Collections$UnmodifiableSet@464d3c4b
- locked sun.nio.ch.EPollSelectorImpl@61e7855
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2148)
at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1794)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="nio-acceptor-tcp-comm-#88", id=116, state=RUNNABLE, blockCnt=2, waitCnt=0]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked sun.nio.ch.Util$3@5cda8ce4
- locked java.util.Collections$UnmodifiableSet@20080484
- locked sun.nio.ch.EPollSelectorImpl@f88cbaa
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2926)
at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2874)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="grid-timeout-worker-#71", id=96, state=TIMED_WAITING, blockCnt=2, waitCnt=99]
Lock [object=java.lang.Object@7a21c6de, ownerName=null, ownerId=-1]
at java.lang.Object.wait(Native Method)
at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="ignite-clock", id=95, state=TIMED_WAITING, blockCnt=0, waitCnt=4751]
at java.lang.Thread.sleep(Native Method)
at o.a.i.i.util.IgniteUtils$22.run(IgniteUtils.java:3383)
at java.lang.Thread.run(Thread.java:748)
Thread [name="pub-#70", id=94, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7214d8d2, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="pub-#69", id=93, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7214d8d2, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="pub-#68", id=92, state=TIMED_WAITING, blockCnt=1, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7214d8d2, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="pub-#67", id=91, state=TIMED_WAITING, blockCnt=0, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7214d8d2, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="pub-#66", id=90, state=TIMED_WAITING, blockCnt=1, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7214d8d2, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="pub-#65", id=89, state=TIMED_WAITING, blockCnt=1, waitCnt=1]
Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7214d8d2, ownerName=null, ownerId=-1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thread [name="jvm-pause-detector-worker", id=88, state=TIMED_WAITING, blockCnt=0, waitCnt=959]
at java.lang.Thread.sleep(Native Method)
at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100)
Thread [name="data-streamer-stripe-31-#64", id=87, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-30-#63", id=86, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-29-#62", id=85, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-28-#61", id=84, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-27-#60", id=83, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-26-#59", id=82, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-25-#58", id=81, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-24-#57", id=80, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-23-#56", id=79, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-22-#55", id=78, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-21-#54", id=77, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-20-#53", id=76, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-19-#52", id=75, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-18-#51", id=74, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-17-#50", id=73, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-16-#49", id=72, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-15-#48", id=71, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-14-#47", id=70, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-13-#46", id=69, state=WAITING, blockCnt=2, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-12-#45", id=68, state=WAITING, blockCnt=3, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-11-#44", id=67, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-10-#43", id=66, state=WAITING, blockCnt=2, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-9-#42", id=65, state=WAITING, blockCnt=3, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-8-#41", id=64, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-7-#40", id=63, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-6-#39", id=62, state=WAITING, blockCnt=3, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-5-#38", id=61, state=WAITING, blockCnt=2, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-4-#37", id=60, state=WAITING, blockCnt=2, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-3-#36", id=59, state=WAITING, blockCnt=1, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-2-#35", id=58, state=WAITING, blockCnt=1, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-1-#34", id=57, state=WAITING, blockCnt=1, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="data-streamer-stripe-0-#33", id=56, state=WAITING, blockCnt=2, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-31-#32", id=55, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-30-#31", id=54, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-29-#30", id=53, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-28-#29", id=52, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-27-#28", id=51, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-26-#27", id=50, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-25-#26", id=49, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-24-#25", id=48, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-23-#24", id=47, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-22-#23", id=46, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-21-#22", id=45, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-20-#21", id=44, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-19-#20", id=43, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-18-#19", id=42, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-17-#18", id=41, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-16-#17", id=40, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-15-#16", id=39, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-14-#15", id=38, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-13-#14", id=37, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-12-#13", id=36, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-11-#12", id=35, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-10-#11", id=34, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-9-#10", id=33, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-8-#9", id=32, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-7-#8", id=31, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-6-#7", id=30, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-5-#6", id=29, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-4-#5", id=28, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-3-#4", id=27, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-2-#3", id=26, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-1-#2", id=25, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="sys-stripe-0-#1", id=24, state=WAITING, blockCnt=0, waitCnt=1]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:668)
at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]
Thread [name="Finalizer", id=3, state=WAITING, blockCnt=60, waitCnt=6]
Lock [object=java.lang.ref.ReferenceQueue$Lock@1258ce30, ownerName=null, ownerId=-1]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=6, waitCnt=5]
Lock [object=java.lang.ref.Reference$Lock@5bc4924a, ownerName=null, ownerId=-1]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
2019-12-31 13:24:41.153 ERROR 51266 --- [exchange-worker-#103] ROOT : JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: null]]
Raw
ignite-persistence.log
2019-12-31 13:24:38.006 INFO 51682 --- [main] poc.ignite.IgniteApp : Starting IgniteApp v0.0.1-SNAPSHOT on hdpdev6 with PID 51682 (/opt/ngs/ashishb/apps/ignite/ignite-persistence/ignite-persistence-0.0.1-SNAPSHOT.jar started by hdpusr in /opt/ngs/ashishb/apps/ignite/ignite-persistence)
2019-12-31 13:24:38.009 DEBUG 51682 --- [main] poc.ignite.IgniteApp : Running with Spring Boot v2.1.6.RELEASE, Spring v5.1.8.RELEASE
2019-12-31 13:24:38.010 INFO 51682 --- [main] poc.ignite.IgniteApp : No active profile set, falling back to default profiles: default
2019-12-31 13:24:38.728 INFO 51682 --- [main] poc.ignite.config.BeansConfig : ignite bean service
2019-12-31 13:24:38.945 WARN 51682 --- [main] o.apache.ignite.internal.util.typedef.G : Ignite work directory is not provided, automatically resolved to: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work
2019-12-31 13:24:39.098 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> __________ ________________
>>> / _/ ___/ |/ / _/_ __/ __/
>>> _/ // (7 7 // / / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 2.7.6#20190911-sha1:21f7ca41
>>> 2019 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org
2019-12-31 13:24:39.106 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Config URL: n/a
2019-12-31 13:24:39.125 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : IgniteConfiguration [igniteInstanceName=null, pubPoolSize=16, svcPoolSize=16, callbackPoolSize=32, stripedPoolSize=32, sysPoolSize=16, mgmtPoolSize=4, igfsPoolSize=32, dataStreamerPoolSize=32, utilityCachePoolSize=32, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=32, igniteHome=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/, igniteWorkDir=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@306279ee, nodeId=6ef2878f-6b05-4756-9417-8bcfcb9a1701, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@b59d31, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=42100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=10000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=16, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@62fdb4a6[Count = 1], stopping=false], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@11e21d0e, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@3e77a1ed, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@3ffcd140, clientMode=true, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=90000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=32, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=null, activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=32, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null]
2019-12-31 13:24:39.125 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Daemon mode: off
2019-12-31 13:24:39.129 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : OS: Linux 3.10.0-957.10.1.el7.x86_64 amd64
2019-12-31 13:24:39.129 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : OS user: hdpusr
2019-12-31 13:24:39.129 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : PID: 51682
2019-12-31 13:24:39.130 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Language runtime: Java Platform API Specification ver. 1.8
2019-12-31 13:24:39.130 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : VM information: Java(TM) SE Runtime Environment 1.8.0_191-b12 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.191-b12
2019-12-31 13:24:39.131 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : VM total memory: 0.96GB
2019-12-31 13:24:39.131 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Remote Management [restart: off, REST: off, JMX (remote: off)]
2019-12-31 13:24:39.132 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Logger: Slf4jLogger [impl=Logger[org.apache.ignite.internal.IgniteKernal], quiet=true]
2019-12-31 13:24:39.132 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : IGNITE_HOME=/opt/ngs/setups/apache-ignite-fabric-2.6.0-bin/
2019-12-31 13:24:39.132 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : VM arguments: [-Xms1G, -Xmx1G]
2019-12-31 13:24:39.142 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
2019-12-31 13:24:39.142 WARN 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Peer class loading is enabled (disable it in production for performance and deployment consistency reasons)
2019-12-31 13:24:39.142 WARN 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
2019-12-31 13:24:39.147 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : 3-rd party licenses can be found at: /opt/ngs/setups/apache-ignite-fabric-2.6.0-bin//libs/licenses
2019-12-31 13:24:39.217 INFO 51682 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : Configured plugins:
2019-12-31 13:24:39.218 INFO 51682 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor : ^-- None
2019-12-31 13:24:39.218 INFO 51682 --- [main] o.a.i.i.p.plugin.IgnitePluginProcessor :
2019-12-31 13:24:39.221 INFO 51682 --- [main] o.a.i.i.p.failure.FailureProcessor : Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]]
2019-12-31 13:24:39.273 INFO 51682 --- [main] o.a.i.s.c.tcp.TcpCommunicationSpi : Successfully bound communication NIO server to TCP port [port=42103, locHost=0.0.0.0/0.0.0.0, selectorsCnt=16, selectorSpins=0, pairedConn=false]
2019-12-31 13:24:39.341 WARN 51682 --- [main] o.a.i.s.c.noop.NoopCheckpointSpi : Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
2019-12-31 13:24:39.369 WARN 51682 --- [main] o.a.i.i.m.c.GridCollisionManager : Collision resolution is disabled (all jobs will be activated upon arrival).
2019-12-31 13:24:39.375 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Security status [authentication=off, tls/ssl=off]
2019-12-31 13:24:39.642 INFO 51682 --- [main] o.a.i.i.p.c.d.d.t.PartitionsEvictManager : Evict partition permits=4
2019-12-31 13:24:39.909 INFO 51682 --- [main] o.a.i.i.p.odbc.ClientListenerProcessor : Client connector processor has started on TCP port 10804
2019-12-31 13:24:39.931 INFO 51682 --- [main] o.a.i.i.p.rest.GridRestProcessor : REST protocols do not start on client node. To start the protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system property.
2019-12-31 13:24:40.114 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Non-loopback local IPs: 172.17.241.80, 172.17.5.36
2019-12-31 13:24:40.114 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Enabled local MACs: 1866DAE84428, F4E9D4A04FD2
2019-12-31 13:24:40.307 INFO 51682 --- [disco-notifier-worker-#96] o.a.i.i.p.cache.mvcc.MvccProcessorImpl : Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778880232, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]]
2019-12-31 13:24:40.346 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Performance suggestions for grid (fix if possible)
2019-12-31 13:24:40.346 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
2019-12-31 13:24:40.346 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options)
2019-12-31 13:24:40.347 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options)
2019-12-31 13:24:40.347 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options)
2019-12-31 13:24:40.347 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Speed up flushing of dirty pages by OS (alter vm.dirty_expire_centisecs parameter by setting to 500)
2019-12-31 13:24:40.347 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : ^-- Reduce pages swapping ratio (set vm.swappiness=10.000000 or less)
2019-12-31 13:24:40.347 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning
2019-12-31 13:24:40.348 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal :
2019-12-31 13:24:40.348 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
2019-12-31 13:24:40.350 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal :
>>> +----------------------------------------------------------------------+
>>> Ignite ver. 2.7.6#20190911-sha1:21f7ca41c4348909e2fd26ccf59b5b2ce1f4474e
>>> +----------------------------------------------------------------------+
>>> OS name: Linux 3.10.0-957.10.1.el7.x86_64 amd64
>>> CPU(s): 32
>>> Heap: 1.0GB
>>> VM name: 51682@hdpdev6
>>> Local node [ID=6EF2878F-6B05-4756-9417-8BCFCB9A1701, order=4, clientMode=true]
>>> Local node addresses: [hdpdev6/127.0.0.1, hdpdev6_oob.nseroot.com/172.17.241.80, /172.17.5.36]
>>> Local ports: TCP:10804 TCP:42103
2019-12-31 13:24:40.350 INFO 51682 --- [main] org.apache.ignite.internal.IgniteKernal : >>> Ignite cluster is not active (limited functionality available). Use control.(sh|bat) script or IgniteCluster interface to activate.
2019-12-31 13:24:40.352 INFO 51682 --- [main] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=4, locNode=6ef2878f, servers=3, clients=1, state=INACTIVE, CPUs=32, offheap=1.5GB, heap=4.0GB]
2019-12-31 13:24:40.446 INFO 51682 --- [main] o.s.j.d.e.EmbeddedDatabaseFactory : Starting embedded database: url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false', username='sa'
2019-12-31 13:24:40.607 INFO 51682 --- [main] poc.ignite.IgniteApp : Started IgniteApp in 3.128 seconds (JVM running for 3.585)
2019-12-31 13:24:40.609 INFO 51682 --- [main] poc.ignite.IgniteApp : run service
2019-12-31 13:24:40.609 DEBUG 51682 --- [main] poc.ignite.service.IgniteService : main service
2019-12-31 13:24:40.609 DEBUG 51682 --- [main] p.i.service.SimplePersistenceService : main service
2019-12-31 13:24:40.610 DEBUG 51682 --- [main] p.i.service.SimplePersistenceService : init service
2019-12-31 13:24:40.610 DEBUG 51682 --- [main] p.i.service.SimplePersistenceService : clusterActive: false
2019-12-31 13:24:40.613 INFO 51682 --- [main] o.a.i.i.p.c.GridClusterStateProcessor : Sending activate request from node [id=6ef2878f-6b05-4756-9417-8bcfcb9a1701, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], client=true, daemon=false]
2019-12-31 13:24:40.752 INFO 51682 --- [grid-nio-worker-tcp-comm-0-#72] o.a.i.s.c.tcp.TcpCommunicationSpi : Established outgoing communication connection [locAddr=/127.0.0.1:49480, rmtAddr=/127.0.0.1:42100]
2019-12-31 13:24:40.821 INFO 51682 --- [disco-notifier-worker-#96] o.a.i.i.p.c.GridClusterStateProcessor : Received activate request with BaselineTopology[id=0]
2019-12-31 13:24:40.826 INFO 51682 --- [disco-notifier-worker-#96] o.a.i.i.p.c.GridClusterStateProcessor : Started state transition: true
2019-12-31 13:24:40.848 INFO 51682 --- [exchange-worker-#98] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], mvccCrd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=DISCOVERY_CUSTOM_EVT, evtNode=0f6adfb0-8641-487a-b8e9-ef0ace842b20, customEvt=ChangeGlobalStateMessage [id=15a02fa5f61-fb354dfe-aed5-4d86-8fc8-cc9dce3672e6, reqId=3b5c20df-697e-423d-9208-6a02d536305b, initiatingNodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=1818609622, branchingType='New BaselineTopology', baselineNodes=[d0ce1c0d-ecf0-4f3f-a935-1081bd32d95b, 127.0.0.1,172.17.241.80,172.17.5.36:42502, 65e3ea66-d894-4978-a20c-30d1caffe6e4]], forceChangeBaselineTopology=false, timestamp=1577778880793], allowMerge=false]
2019-12-31 13:24:40.849 INFO 51682 --- [exchange-worker-#98] .c.d.d.p.GridDhtPartitionsExchangeFuture : Start activation process [nodeId=6ef2878f-6b05-4756-9417-8bcfcb9a1701, client=true, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:40.849 INFO 51682 --- [exchange-worker-#98] org.apache.ignite.cache.msg : Components activation performed in 0 ms.
2019-12-31 13:24:40.853 INFO 51682 --- [exchange-worker-#98] o.a.i.i.p.c.CacheAffinitySharedManager : Updating caches registry performed in 3 ms.
2019-12-31 13:24:40.999 INFO 51682 --- [exchange-worker-#98] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false], encryptionEnabled=false]
2019-12-31 13:24:41.000 INFO 51682 --- [exchange-worker-#98] o.a.i.i.p.c.CacheAffinitySharedManager : Caches starting performed in 147 ms.
2019-12-31 13:24:41.005 INFO 51682 --- [exchange-worker-#98] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity initialization for started caches performed in 4 ms.
2019-12-31 13:24:41.005 INFO 51682 --- [exchange-worker-#98] .c.d.d.p.GridDhtPartitionsExchangeFuture : Successfully activated caches [nodeId=6ef2878f-6b05-4756-9417-8bcfcb9a1701, client=true, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.009 INFO 51682 --- [exchange-worker-#98] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Single Message performed in 1 ms.
2019-12-31 13:24:41.009 INFO 51682 --- [exchange-worker-#98] o.apache.ignite.internal.exchange.time : Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], crd=false]
2019-12-31 13:24:41.524 WARN 51682 --- [disco-event-worker-#97] o.a.i.i.m.d.GridDiscoveryManager : Node FAILED: TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778880242, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false]
2019-12-31 13:24:41.525 INFO 51682 --- [disco-event-worker-#97] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=5, locNode=6ef2878f, servers=2, clients=1, state=ACTIVE, CPUs=32, offheap=1.0GB, heap=3.0GB]
2019-12-31 13:24:41.526 INFO 51682 --- [disco-event-worker-#97] o.a.i.i.m.d.GridDiscoveryManager : ^-- Baseline [id=0, size=3, online=2, offline=1]
2019-12-31 13:24:41.577 INFO 51682 --- [disco-notifier-worker-#96] o.a.i.i.p.c.GridClusterStateProcessor : Received state change finish message: true
2019-12-31 13:24:41.590 INFO 51682 --- [sys-#101] .i.p.c.GridCachePartitionExchangeManager : Delay process full message without exchange id (there is exchange in progress) [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20]
2019-12-31 13:24:41.593 INFO 51682 --- [sys-#102] .c.d.d.p.GridDhtPartitionsExchangeFuture : Received full message, will finish exchange [node=0f6adfb0-8641-487a-b8e9-ef0ace842b20, resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.596 INFO 51682 --- [sys-#102] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity applying from full message performed in 2 ms.
2019-12-31 13:24:41.596 INFO 51682 --- [sys-#102] .c.d.d.p.GridDhtPartitionsExchangeFuture : Affinity changes applied in 2 ms.
2019-12-31 13:24:41.598 INFO 51682 --- [sys-#102] .c.d.d.p.GridDhtPartitionsExchangeFuture : Full map updating for 1 groups performed in 1 ms.
2019-12-31 13:24:41.598 INFO 51682 --- [sys-#102] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], err=null]
2019-12-31 13:24:41.606 INFO 51682 --- [sys-#102] o.a.i.i.p.cache.GridCacheProcessor : Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=6ef2878f-6b05-4756-9417-8bcfcb9a1701
2019-12-31 13:24:41.613 INFO 51682 --- [sys-#102] .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange [localNode=6ef2878f-6b05-4756-9417-8bcfcb9a1701, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=0f6adfb0-8641-487a-b8e9-ef0ace842b20, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6/172.17.5.36:42500, /127.0.0.1:42500, hdpdev6_oob.nseroot.com/172.17.241.80:42500], discPort=42500, order=1, intOrder=1, lastExchangeTime=1577778880232, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], durationFromInit=766]
2019-12-31 13:24:41.613 INFO 51682 --- [exchange-worker-#98] .i.p.c.GridCachePartitionExchangeManager : Skipping rebalancing (no affinity changes) [top=AffinityTopologyVersion [topVer=4, minorTopVer=1], rebTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], evt=DISCOVERY_CUSTOM_EVT, evtNode=0f6adfb0-8641-487a-b8e9-ef0ace842b20, client=true]
2019-12-31 13:24:41.614 INFO 51682 --- [exchange-worker-#98] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=0f6adfb0-8641-487a-b8e9-ef0ace842b20, crdVer=1577778810538, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, customEvt=null, allowMerge=true]
2019-12-31 13:24:41.614 INFO 51682 --- [exchange-worker-#98] .c.d.d.p.GridDhtPartitionsExchangeFuture : Sending Single Message performed in 0 ms.
2019-12-31 13:24:41.615 INFO 51682 --- [exchange-worker-#98] o.apache.ignite.internal.exchange.time : Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false]
2019-12-31 13:24:41.622 INFO 51682 --- [sys-#103] o.a.i.i.p.c.GridClusterStateProcessor : Successfully performed final activation steps [nodeId=6ef2878f-6b05-4756-9417-8bcfcb9a1701, client=true, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1]]
2019-12-31 13:24:41.630 INFO 51682 --- [sys-#106] .c.d.d.p.GridDhtPartitionsExchangeFuture : Received full message, will finish exchange [node=0f6adfb0-8641-487a-b8e9-ef0ace842b20, resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]
2019-12-31 13:24:41.631 INFO 51682 --- [sys-#106] o.a.i.i.p.c.CacheAffinitySharedManager : Affinity applying from full message performed in 0 ms.
2019-12-31 13:24:41.631 INFO 51682 --- [sys-#106] .c.d.d.p.GridDhtPartitionsExchangeFuture : Affinity changes applied in 0 ms.
2019-12-31 13:24:41.632 INFO 51682 --- [sys-#106] .c.d.d.p.GridDhtPartitionsExchangeFuture : Full map updating for 1 groups performed in 0 ms.
2019-12-31 13:24:41.633 INFO 51682 --- [sys-#106] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null]
2019-12-31 13:24:41.635 INFO 51682 --- [sys-#106] .c.d.d.p.GridDhtPartitionsExchangeFuture : Detecting lost partitions performed in 2 ms.
2019-12-31 13:24:41.635 INFO 51682 --- [sys-#106] .c.d.d.p.GridDhtPartitionsExchangeFuture : Completed partition exchange [localNode=6ef2878f-6b05-4756-9417-8bcfcb9a1701, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, addrs=[127.0.0.1, 172.17.241.80, 172.17.5.36], sockAddrs=[hdpdev6_oob.nseroot.com/172.17.241.80:42502, hdpdev6/172.17.5.36:42502, /127.0.0.1:42502], discPort=42502, order=3, intOrder=3, lastExchangeTime=1577778880242, loc=false, ver=2.7.6#20190911-sha1:21f7ca41, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], durationFromInit=30]
2019-12-31 13:24:41.636 INFO 51682 --- [exchange-worker-#98] .i.p.c.GridCachePartitionExchangeManager : Skipping rebalancing (no affinity changes) [top=AffinityTopologyVersion [topVer=5, minorTopVer=0], rebTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], evt=NODE_FAILED, evtNode=51f0bedf-ffe8-4963-a2ae-6bed6955eb23, client=true]
2019-12-31 13:25:40.355 INFO 51682 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=6ef2878f, uptime=00:01:00.004]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.03%, avg=0.15%, GC=0%]
^-- PageMemory [pages=0]
^-- Heap [used=178MB, free=81.77%, comm=981MB]
^-- Off-heap [used=0MB, free=-1%, comm=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=2, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
2019-12-31 13:25:50.329 INFO 51682 --- [ignite-update-notifier-timer] o.a.i.i.p.cluster.GridUpdateNotifier : Update status is not available.
2019-12-31 13:26:40.351 INFO 51682 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=6ef2878f, uptime=00:02:00.005]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0.03%, avg=0.1%, GC=0%]
^-- PageMemory [pages=0]
^-- Heap [used=219MB, free=77.63%, comm=981MB]
^-- Off-heap [used=0MB, free=-1%, comm=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=0, qSize=0]
2019-12-31 13:27:40.355 INFO 51682 --- [grid-timeout-worker-#71] org.apache.ignite.internal.IgniteKernal :
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=6ef2878f, uptime=00:03:00.006]
^-- H/N/C [hosts=1, nodes=3, CPUs=32]
^-- CPU [cur=0%, avg=0.08%, GC=0%]
^-- PageMemory [pages=0]
^-- Heap [used=259MB, free=73.51%, comm=981MB]
^-- Off-heap [used=0MB, free=-1%, comm=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=0, qSize=0]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment