Skip to content

Instantly share code, notes, and snippets.

@darkn3rd
Created February 7, 2021 22:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save darkn3rd/71e4b8d7e600a854a1adf0a0b82a7540 to your computer and use it in GitHub Desktop.
Save darkn3rd/71e4b8d7e600a854a1adf0a0b82a7540 to your computer and use it in GitHub Desktop.
Dgraph Tchalla Liveliness Issue
++ awk '{gsub(/\.$/,""); print $0}'
++ hostname -f
+ dgraph alpha --my=t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080 --zero t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080,t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080,t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:23:33 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:23:33 Integration installed: Environment
[Sentry] 2021/02/07 21:23:33 Integration installed: Modules
[Sentry] 2021/02/07 21:23:33 Integration installed: IgnoreErrors
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:23:33 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:23:33 Integration installed: Environment
[Sentry] 2021/02/07 21:23:33 Integration installed: Modules
[Sentry] 2021/02/07 21:23:33 Integration installed: IgnoreErrors
I0207 21:23:33.648585 20 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
W0207 21:23:33.648886 20 run.go:573] --lru_mb is deprecated, use --cache_mb instead
I0207 21:23:33.780280 20 init.go:107]
Dgraph version : v20.11.1
Dgraph codename : tchalla-1
Dgraph SHA-256 : cefdcc880c0607a92a1d8d3ba0beb015459ebe216e79fdad613eb0d00d09f134
Commit SHA-1 : 7153d13fe
Commit timestamp : 2021-01-28 15:59:35 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit https://discuss.dgraph.io.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
I0207 21:23:33.780301 20 run.go:696] x.Config: {PortOffset:0 QueryEdgeLimit:1000000 NormalizeNodeLimit:10000 MutationsNQuadLimit:1000000 PollInterval:1s GraphqlExtension:true GraphqlDebug:false GraphqlLambdaUrl:}
I0207 21:23:33.780323 20 run.go:697] x.WorkerConfig: {TmpDir:t ExportPath:export NumPendingProposals:256 Tracing:0.01 MyAddr:t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080 ZeroAddr:[t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080 t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080 t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> RaftId:0 WhiteListedIPRanges:[] MaxRetries:-1 StrictMutations:false AclEnabled:false AbortOlderThan:5m0s SnapshotAfter:10000 ProposedGroupId:0 StartTime:2021-02-07 21:23:33.38287631 +0000 UTC m=+0.012889675 LudicrousMode:false LudicrousConcurrency:2000 EncryptionKey:**** LogRequest:0 HardSync:false}
I0207 21:23:33.780365 20 run.go:698] worker.Config: {PostingDir:p PostingDirCompression:1 PostingDirCompressionLevel:0 WALDir:w MutationsMode:0 AuthToken: PBlockCacheSize:1395864371 PIndexCacheSize:751619276 WalCache:0 HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35,0 CacheMb:0}
I0207 21:23:33.781685 20 storage.go:132] Init Raft Storage with snap: 0, first: 1, last: 0
I0207 21:23:33.781945 20 server_state.go:76] Setting Posting Dir Compression Level: 0
I0207 21:23:33.781960 20 server_state.go:120] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x2e0ddf8 Compression:1 InMemory:false MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 ValueThreshold:1024 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:1395864371 IndexCacheSize:751619276 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false managedTxns:false maxBatchCount:0 maxBatchSize:0}
I0207 21:23:33.797153 20 log.go:34] All 0 tables opened in 0s
I0207 21:23:33.798386 20 log.go:34] Discard stats nextEmptySlot: 0
I0207 21:23:33.798418 20 log.go:34] Set nextTxnTs to 0
E0207 21:23:33.799105 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": Unable to find any servers for group: 1. closer err: <nil>
I0207 21:23:33.799154 20 groups.go:99] Current Raft Id: 0x0
I0207 21:23:33.799281 20 worker.go:104] Worker listening at address: [::]:7080
I0207 21:23:33.800090 20 run.go:519] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
E0207 21:23:33.800089 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0207 21:23:33.800107 20 run.go:520] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
I0207 21:23:33.800156 20 run.go:552] gRPC server started. Listening on port 9080
I0207 21:23:33.800175 20 run.go:553] HTTP server started. Listening on port 8080
I0207 21:23:33.899495 20 pool.go:162] CONNECTING to t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
W0207 21:23:33.924843 20 pool.go:267] Connection lost with t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local: no such host"
I0207 21:23:34.124939 20 pool.go:162] CONNECTING to t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080
W0207 21:23:34.131937 20 pool.go:267] Connection lost with t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local: no such host"
I0207 21:23:34.532026 20 pool.go:162] CONNECTING to t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080
W0207 21:23:34.539048 20 pool.go:267] Connection lost with t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local: no such host"
E0207 21:23:34.799368 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": Unable to find any servers for group: 1. closer err: <nil>
E0207 21:23:34.800221 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0207 21:23:35.799485 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": Unable to find any servers for group: 1. closer err: <nil>
E0207 21:23:35.800339 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
E0207 21:23:36.799595 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": Unable to find any servers for group: 1. closer err: <nil>
E0207 21:23:36.800439 20 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0207 21:23:37.343090 20 groups.go:127] Connected to group zero. Assigned group: 1
I0207 21:23:37.343109 20 groups.go:129] Raft Id after connection to Zero: 0x1
I0207 21:23:37.343125 20 draft.go:230] Node ID: 0x1 with GroupID: 1
I0207 21:23:37.343193 20 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc00003e690 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x2e0ddf8 DisableProposalForwarding:false}
I0207 21:23:37.343275 20 node.go:326] Group 1 found 0 entries
I0207 21:23:37.343285 20 draft.go:1712] New Node for group: 1
I0207 21:23:37.343298 20 log.go:34] 1 became follower at term 0
I0207 21:23:37.343307 20 log.go:34] newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0207 21:23:37.343314 20 log.go:34] 1 became follower at term 1
I0207 21:23:37.343343 20 draft.go:180] Operation started with id: opRollup
I0207 21:23:37.343392 20 groups.go:807] Got address of a Zero leader: t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:23:37.343410 20 draft.go:1084] Found Raft progress: 0
I0207 21:23:37.343529 20 groups.go:821] Starting a new membership stream receive from t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080.
I0207 21:23:37.343523 20 node.go:189] Setting conf state to nodes:1
I0207 21:23:37.343630 20 log.go:34] 1 is starting a new election at term 1
I0207 21:23:37.343642 20 log.go:34] 1 became pre-candidate at term 1
I0207 21:23:37.343645 20 log.go:34] 1 received MsgPreVoteResp from 1 at term 1
I0207 21:23:37.343654 20 log.go:34] 1 became candidate at term 2
I0207 21:23:37.343657 20 log.go:34] 1 received MsgVoteResp from 1 at term 2
I0207 21:23:37.343664 20 log.go:34] 1 became leader at term 2
I0207 21:23:37.343668 20 log.go:34] raft.node: 1 elected leader 1 at term 2
I0207 21:23:37.344857 20 groups.go:838] Received first state update from Zero: counter:3 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080" > > > > zeros:<key:1 value:<id:1 addr:"t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080" leader:true > > maxRaftId:1
I0207 21:23:37.799823 20 pool.go:162] CONNECTING to t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:23:38.343492 20 groups.go:893] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0207 21:23:38.343516 20 groups.go:905] Got Zero leader: t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:23:38.347766 20 groups.go:487] Serving tablet for: dgraph.cors
I0207 21:23:38.347897 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.347941 20 index.go:889] Computing prefix index for attr dgraph.cors and tokenizers []
I0207 21:23:38.347948 20 index.go:907] Deleting index for attr dgraph.cors and tokenizers [exact]
I0207 21:23:38.347991 20 index.go:779] Deleting indexes for dgraph.cors
I0207 21:23:38.348002 20 log.go:34] DropPrefix called for [ dgraph.cors  dgraph.cors dgraph.cors  dgraph.cors]
I0207 21:23:38.348015 20 log.go:34] Writes flushed. Stopping compactions now...
I0207 21:23:38.348006 20 draft.go:124] Operation completed with id: opRollup
I0207 21:23:38.350018 20 log.go:34] DropPrefix done
I0207 21:23:38.350056 20 log.go:34] Resuming writes
I0207 21:23:38.364011 20 log.go:34] Rebuilding index for predicate dgraph.cors (1/2): Sent data of size 0 B
I0207 21:23:38.369593 20 index.go:942] Rebuilding index for attr dgraph.cors and tokenizers [exact]
I0207 21:23:38.374823 20 groups.go:487] Serving tablet for: dgraph.type
I0207 21:23:38.380508 20 log.go:34] Rebuilding index for predicate dgraph.cors (1/2): Sent data of size 0 B
I0207 21:23:38.384278 20 mutation.go:188] Done schema update predicate:"dgraph.cors" value_type:STRING directive:INDEX tokenizer:"exact" list:true upsert:true
I0207 21:23:38.384359 20 draft.go:124] Operation completed with id: opIndexing
E0207 21:23:38.384354 20 draft.go:714] Applying proposal. Error: operation opIndexing is already running. Proposal: {"group_id:1 start_ts:2 schema:<predicate:\"dgraph.type\" value_type:STRING directive:INDEX tokenizer:\"exact\" list:true > " [] "<nil>" "" "<nil>" "<nil>" '\t' '\x00' "<nil>" {} "" '\x00'}.
E0207 21:23:38.384483 20 groups.go:260] Error while proposing initial schema: operation opIndexing is already running
I0207 21:23:38.484708 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.484740 20 index.go:889] Computing prefix index for attr dgraph.type and tokenizers []
I0207 21:23:38.484745 20 index.go:907] Deleting index for attr dgraph.type and tokenizers [exact]
I0207 21:23:38.484774 20 index.go:779] Deleting indexes for dgraph.type
I0207 21:23:38.484781 20 log.go:34] DropPrefix called for [ dgraph.type  dgraph.type dgraph.type  dgraph.type]
I0207 21:23:38.484803 20 log.go:34] Writes flushed. Stopping compactions now...
I0207 21:23:38.499461 20 log.go:34] DropPrefix done
I0207 21:23:38.499479 20 log.go:34] Resuming writes
I0207 21:23:38.534448 20 log.go:34] Rebuilding index for predicate dgraph.type (1/2): Sent data of size 0 B
I0207 21:23:38.538463 20 index.go:942] Rebuilding index for attr dgraph.type and tokenizers [exact]
I0207 21:23:38.542661 20 groups.go:487] Serving tablet for: dgraph.drop.op
I0207 21:23:38.578209 20 log.go:34] Rebuilding index for predicate dgraph.type (1/2): Sent data of size 0 B
I0207 21:23:38.582031 20 mutation.go:188] Done schema update predicate:"dgraph.type" value_type:STRING directive:INDEX tokenizer:"exact" list:true
E0207 21:23:38.582082 20 draft.go:714] Applying proposal. Error: operation opIndexing is already running. Proposal: {"group_id:1 start_ts:3 schema:<predicate:\"dgraph.drop.op\" value_type:STRING > " [] "<nil>" "" "<nil>" "<nil>" '\r' '\x00' "<nil>" {} "" '\x00'}.
I0207 21:23:38.582110 20 draft.go:124] Operation completed with id: opIndexing
E0207 21:23:38.582154 20 groups.go:260] Error while proposing initial schema: operation opIndexing is already running
I0207 21:23:38.682369 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.682420 20 index.go:779] Deleting indexes for dgraph.drop.op
I0207 21:23:38.682477 20 draft.go:124] Operation completed with id: opIndexing
I0207 21:23:38.686663 20 groups.go:487] Serving tablet for: dgraph.graphql.schema
I0207 21:23:38.686748 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.686784 20 index.go:779] Deleting indexes for dgraph.graphql.schema
I0207 21:23:38.686841 20 draft.go:124] Operation completed with id: opIndexing
I0207 21:23:38.690560 20 groups.go:487] Serving tablet for: dgraph.graphql.xid
I0207 21:23:38.690650 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.690671 20 index.go:889] Computing prefix index for attr dgraph.graphql.xid and tokenizers []
I0207 21:23:38.690677 20 index.go:907] Deleting index for attr dgraph.graphql.xid and tokenizers [exact]
I0207 21:23:38.690724 20 index.go:779] Deleting indexes for dgraph.graphql.xid
I0207 21:23:38.690735 20 log.go:34] DropPrefix called for [dgraph.graphql.xid dgraph.graphql.xid dgraph.graphql.xid dgraph.graphql.xid]
I0207 21:23:38.690752 20 log.go:34] Writes flushed. Stopping compactions now...
I0207 21:23:38.712671 20 log.go:34] DropPrefix done
I0207 21:23:38.712689 20 log.go:34] Resuming writes
I0207 21:23:38.712728 20 index.go:942] Rebuilding index for attr dgraph.graphql.xid and tokenizers [exact]
I0207 21:23:38.716431 20 groups.go:487] Serving tablet for: dgraph.graphql.schema_history
I0207 21:23:38.748101 20 log.go:34] Rebuilding index for predicate dgraph.graphql.xid (1/2): Sent data of size 0 B
I0207 21:23:38.751769 20 mutation.go:188] Done schema update predicate:"dgraph.graphql.xid" value_type:STRING directive:INDEX tokenizer:"exact" upsert:true
E0207 21:23:38.751832 20 draft.go:714] Applying proposal. Error: operation opIndexing is already running. Proposal: {"group_id:1 start_ts:6 schema:<predicate:\"dgraph.graphql.schema_history\" value_type:STRING > " [] "<nil>" "" "<nil>" "<nil>" '\x16' '\x00' "<nil>" {} "" '\x00'}.
I0207 21:23:38.751908 20 draft.go:124] Operation completed with id: opIndexing
E0207 21:23:38.751968 20 groups.go:260] Error while proposing initial schema: operation opIndexing is already running
I0207 21:23:38.800219 20 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:23:38.852237 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.852278 20 index.go:779] Deleting indexes for dgraph.graphql.schema_history
I0207 21:23:38.852365 20 draft.go:124] Operation completed with id: opIndexing
I0207 21:23:38.856285 20 groups.go:487] Serving tablet for: dgraph.graphql.schema_created_at
I0207 21:23:38.856392 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.856418 20 index.go:779] Deleting indexes for dgraph.graphql.schema_created_at
I0207 21:23:38.856475 20 draft.go:124] Operation completed with id: opIndexing
I0207 21:23:38.860127 20 groups.go:487] Serving tablet for: dgraph.graphql.p_query
I0207 21:23:38.860224 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.860252 20 index.go:779] Deleting indexes for dgraph.graphql.p_query
I0207 21:23:38.860304 20 draft.go:124] Operation completed with id: opIndexing
I0207 21:23:38.863941 20 groups.go:487] Serving tablet for: dgraph.graphql.p_sha256hash
I0207 21:23:38.864032 20 draft.go:180] Operation started with id: opIndexing
I0207 21:23:38.864065 20 index.go:889] Computing prefix index for attr dgraph.graphql.p_sha256hash and tokenizers []
I0207 21:23:38.864072 20 index.go:907] Deleting index for attr dgraph.graphql.p_sha256hash and tokenizers [exact]
I0207 21:23:38.864097 20 index.go:779] Deleting indexes for dgraph.graphql.p_sha256hash
I0207 21:23:38.864106 20 log.go:34] DropPrefix called for [dgraph.graphql.p_sha256hash dgraph.graphql.p_sha256hash dgraph.graphql.p_sha256hash dgraph.graphql.p_sha256hash]
I0207 21:23:38.864115 20 log.go:34] Writes flushed. Stopping compactions now...
I0207 21:23:38.888350 20 log.go:34] DropPrefix done
I0207 21:23:38.888369 20 log.go:34] Resuming writes
I0207 21:23:38.888402 20 index.go:942] Rebuilding index for attr dgraph.graphql.p_sha256hash and tokenizers [exact]
I0207 21:23:38.894075 20 groups.go:159] Server is ready
I0207 21:23:38.894087 20 access_ee.go:390] ResetAcl closed
I0207 21:23:38.894091 20 access_ee.go:311] RefreshAcls closed
I0207 21:23:38.923634 20 log.go:34] Rebuilding index for predicate dgraph.graphql.p_sha256hash (1/2): Sent data of size 0 B
I0207 21:23:38.934744 20 mutation.go:188] Done schema update predicate:"dgraph.graphql.p_sha256hash" value_type:STRING directive:INDEX tokenizer:"exact"
I0207 21:23:38.934786 20 draft.go:124] Operation completed with id: opIndexing
I0207 21:23:39.399203 20 graphql.go:41] ResetCors closed
I0207 21:23:39.400784 20 run.go:813] Updating cors from subscription.
I0207 21:23:39.400798 20 run.go:837] Updating cors origins: [*]
I0207 21:23:43.802352 20 admin.go:697] No GraphQL schema in Dgraph; serving empty GraphQL API
E0207 21:23:48.222285 20 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader
I0207 21:23:48.343510 20 groups.go:893] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0207 21:23:48.343540 20 groups.go:905] Got Zero leader: t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:23:48.384459 20 draft.go:180] Operation started with id: opRollup
E0207 21:24:02.566368 20 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Unknown desc = Node is no longer leader
I0207 21:24:03.343490 20 groups.go:893] Leader idx=0x1 of group=1 is connecting to Zero for txn updates
I0207 21:24:03.343513 20 groups.go:905] Got Zero leader: t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:06.891295 20 pool.go:162] CONNECTING to t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:24:06.891308 20 node.go:586] Trying to add 0x2 to cluster. Addr: t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:24:06.891313 20 node.go:587] Current confstate at 0x1: nodes:1
I0207 21:24:06.891376 20 node.go:189] Setting conf state to nodes:1 nodes:2
I0207 21:24:06.891415 20 node.go:756] [0x2] Done joining cluster with err: <nil>
W0207 21:24:06.896482 20 pool.go:267] Connection lost with t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local: no such host"
W0207 21:24:07.943730 20 node.go:420] Unable to send message to peer: 0x2. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local: no such host"
W0207 21:24:18.043571 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:24:28.143649 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:24:35.893559 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:24:38.243647 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:24:48.343528 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:24:58.443559 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:25:03.893982 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:25:08.543566 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:25:18.643579 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:25:28.743565 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:25:31.894354 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:25:38.843558 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:25:48.943644 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:25:59.043642 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:25:59.894786 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:26:09.143500 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:26:19.143541 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:26:27.895295 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:26:29.243549 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:26:39.343529 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:26:49.443561 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:26:55.895593 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:26:59.543560 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:27:09.643563 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:27:19.743572 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:27:23.895987 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:27:29.843640 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:27:39.943647 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:27:50.043539 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:27:51.896409 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:28:00.143595 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:28:10.243640 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:28:19.896788 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:28:20.343567 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:28:30.443538 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:28:40.543648 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:28:47.897220 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:28:50.643557 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:29:00.743642 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:29:10.843561 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:29:15.897592 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:29:20.943535 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:29:31.043561 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:29:41.143557 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:29:43.897972 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:29:51.243563 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:30:01.343537 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:30:11.443626 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:30:11.898401 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:30:21.543558 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:30:31.643546 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:30:39.898790 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:30:41.743620 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:30:51.843507 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:31:01.843566 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:31:07.899249 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:31:11.943623 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:31:22.043542 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:31:32.043536 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:31:35.899686 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:31:42.143557 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:31:52.243641 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:32:02.343577 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:32:03.900086 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:32:12.443559 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:32:22.543541 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:32:31.900507 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:32:32.643534 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:32:42.743565 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:32:52.843564 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:32:59.900902 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:33:02.943564 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:33:13.043563 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:33:23.143582 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:33:27.901472 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:33:33.243640 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:33:43.343554 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:33:53.443647 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:33:55.901787 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:34:03.543652 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:34:13.643561 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:34:23.743544 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:34:23.902242 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:34:33.843521 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
W0207 21:34:43.943570 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
E0207 21:34:51.902668 20 groups.go:1035] While proposing delta with MaxAssigned: 18 and num txns: 0. Error=Server overloaded with pending proposals. Please retry later. Retrying...
W0207 21:34:54.043518 20 node.go:420] Unable to send message to peer: 0x2. Error: Unhealthy connection
++ hostname -f
++ awk '{gsub(/\.$/,""); print $0}'
+ dgraph alpha --my=t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080 --zero t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080,t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080,t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:24:06 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:24:06 Integration installed: Environment
[Sentry] 2021/02/07 21:24:06 Integration installed: Modules
[Sentry] 2021/02/07 21:24:06 Integration installed: IgnoreErrors
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:24:06 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:24:06 Integration installed: Environment
[Sentry] 2021/02/07 21:24:06 Integration installed: Modules
[Sentry] 2021/02/07 21:24:06 Integration installed: IgnoreErrors
I0207 21:24:06.629448 19 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
W0207 21:24:06.629742 19 run.go:573] --lru_mb is deprecated, use --cache_mb instead
I0207 21:24:06.764028 19 init.go:107]
Dgraph version : v20.11.1
Dgraph codename : tchalla-1
Dgraph SHA-256 : cefdcc880c0607a92a1d8d3ba0beb015459ebe216e79fdad613eb0d00d09f134
Commit SHA-1 : 7153d13fe
Commit timestamp : 2021-01-28 15:59:35 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit https://discuss.dgraph.io.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
I0207 21:24:06.764043 19 run.go:696] x.Config: {PortOffset:0 QueryEdgeLimit:1000000 NormalizeNodeLimit:10000 MutationsNQuadLimit:1000000 PollInterval:1s GraphqlExtension:true GraphqlDebug:false GraphqlLambdaUrl:}
I0207 21:24:06.764069 19 run.go:697] x.WorkerConfig: {TmpDir:t ExportPath:export NumPendingProposals:256 Tracing:0.01 MyAddr:t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080 ZeroAddr:[t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080 t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080 t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080] TLSClientConfig:<nil> TLSServerConfig:<nil> RaftId:0 WhiteListedIPRanges:[] MaxRetries:-1 StrictMutations:false AclEnabled:false AbortOlderThan:5m0s SnapshotAfter:10000 ProposedGroupId:0 StartTime:2021-02-07 21:24:06.359310961 +0000 UTC m=+0.013119979 LudicrousMode:false LudicrousConcurrency:2000 EncryptionKey:**** LogRequest:0 HardSync:false}
I0207 21:24:06.764097 19 run.go:698] worker.Config: {PostingDir:p PostingDirCompression:1 PostingDirCompressionLevel:0 WALDir:w MutationsMode:0 AuthToken: PBlockCacheSize:1395864371 PIndexCacheSize:751619276 WalCache:0 HmacSecret:**** AccessJwtTtl:0s RefreshJwtTtl:0s CachePercentage:0,65,35,0 CacheMb:0}
I0207 21:24:06.765512 19 storage.go:132] Init Raft Storage with snap: 0, first: 1, last: 0
I0207 21:24:06.765868 19 server_state.go:76] Setting Posting Dir Compression Level: 0
I0207 21:24:06.765881 19 server_state.go:120] Opening postings BadgerDB with options: {Dir:p ValueDir:p SyncWrites:false NumVersionsToKeep:2147483647 ReadOnly:false Logger:0x2e0ddf8 Compression:1 InMemory:false MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:10485760 LevelSizeMultiplier:10 TableSizeMultiplier:2 MaxLevels:7 ValueThreshold:1024 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:1395864371 IndexCacheSize:751619276 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:1000000 NumCompactors:4 CompactL0OnClose:false ZSTDCompressionLevel:0 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:false managedTxns:false maxBatchCount:0 maxBatchSize:0}
I0207 21:24:06.779352 19 log.go:34] All 0 tables opened in 0s
I0207 21:24:06.780615 19 log.go:34] Discard stats nextEmptySlot: 0
I0207 21:24:06.780645 19 log.go:34] Set nextTxnTs to 0
I0207 21:24:06.781395 19 groups.go:99] Current Raft Id: 0x0
E0207 21:24:06.781398 19 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\vdgraph.cors\x00": Unable to find any servers for group: 1. closer err: <nil>
I0207 21:24:06.781578 19 worker.go:104] Worker listening at address: [::]:7080
I0207 21:24:06.782351 19 run.go:519] Bringing up GraphQL HTTP API at 0.0.0.0:8080/graphql
I0207 21:24:06.782379 19 run.go:520] Bringing up GraphQL HTTP admin API at 0.0.0.0:8080/admin
E0207 21:24:06.782343 19 groups.go:1143] Error during SubscribeForUpdates for prefix "\x00\x00\x15dgraph.graphql.schema\x00": Unable to find any servers for group: 1. closer err: <nil>
I0207 21:24:06.782424 19 run.go:552] gRPC server started. Listening on port 9080
I0207 21:24:06.782438 19 run.go:553] HTTP server started. Listening on port 8080
I0207 21:24:06.881588 19 pool.go:162] CONNECTING to t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:06.886432 19 groups.go:127] Connected to group zero. Assigned group: 1
I0207 21:24:06.886448 19 groups.go:129] Raft Id after connection to Zero: 0x2
I0207 21:24:06.886494 19 pool.go:162] CONNECTING to t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:24:06.886575 19 pool.go:162] CONNECTING to t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:06.886596 19 pool.go:162] CONNECTING to t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:06.886608 19 draft.go:230] Node ID: 0x2 with GroupID: 1
I0207 21:24:06.886664 19 node.go:152] Setting raft.Config to: &{ID:2 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc000112f00 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x2e0ddf8 DisableProposalForwarding:false}
I0207 21:24:06.886803 19 node.go:326] Group 1 found 0 entries
I0207 21:24:06.886815 19 draft.go:1650] Calling IsPeer
I0207 21:24:06.890607 19 draft.go:1655] Done with IsPeer call
I0207 21:24:06.890620 19 draft.go:1712] New Node for group: 1
I0207 21:24:06.890628 19 draft.go:1719] Trying to join peers.
I0207 21:24:06.890635 19 draft.go:1633] Calling JoinCluster via leader: t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:24:06.892055 19 draft.go:1637] Done with JoinCluster call
I0207 21:24:06.892094 19 log.go:34] 2 became follower at term 0
I0207 21:24:06.892102 19 log.go:34] newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0207 21:24:06.892107 19 log.go:34] 2 became follower at term 1
I0207 21:24:06.892137 19 draft.go:180] Operation started with id: opRollup
I0207 21:24:06.892196 19 draft.go:1084] Found Raft progress: 0
I0207 21:24:06.892246 19 groups.go:807] Got address of a Zero leader: t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:06.892345 19 groups.go:821] Starting a new membership stream receive from t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080.
I0207 21:24:06.893370 19 groups.go:838] Received first state update from Zero: counter:21 groups:<key:1 value:<members:<key:1 value:<id:1 group_id:1 addr:"t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080" leader:true last_update:1612733017 > > members:<key:2 value:<id:2 group_id:1 addr:"t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080" > > tablets:<key:"dgraph.cors" value:<group_id:1 predicate:"dgraph.cors" > > tablets:<key:"dgraph.drop.op" value:<group_id:1 predicate:"dgraph.drop.op" > > tablets:<key:"dgraph.graphql.p_query" value:<group_id:1 predicate:"dgraph.graphql.p_query" > > tablets:<key:"dgraph.graphql.p_sha256hash" value:<group_id:1 predicate:"dgraph.graphql.p_sha256hash" > > tablets:<key:"dgraph.graphql.schema" value:<group_id:1 predicate:"dgraph.graphql.schema" > > tablets:<key:"dgraph.graphql.schema_created_at" value:<group_id:1 predicate:"dgraph.graphql.schema_created_at" > > tablets:<key:"dgraph.graphql.schema_history" value:<group_id:1 predicate:"dgraph.graphql.schema_history" > > tablets:<key:"dgraph.graphql.xid" value:<group_id:1 predicate:"dgraph.graphql.xid" > > tablets:<key:"dgraph.type" value:<group_id:1 predicate:"dgraph.type" > > checksum:12071872251239819759 > > zeros:<key:1 value:<id:1 addr:"t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080" leader:true > > zeros:<key:2 value:<id:2 addr:"t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080" > > zeros:<key:3 value:<id:3 addr:"t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080" > > maxLeaseId:10000 maxTxnTs:10000 maxRaftId:2 cid:"cd560ae5-1eaa-4650-bc41-d0c6e3afbbd6" license:<maxNodes:18446744073709551615 expiryTs:1615325020 enabled:true >
I0207 21:24:07.782573 19 pool.go:162] CONNECTING to t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:24:11.782481 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:16.782620 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:21.782748 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:26.782890 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:31.783017 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:36.783148 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:41.783290 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:46.783357 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:51.783441 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:24:56.783518 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:01.783647 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:06.783770 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:11.783894 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:16.784017 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:21.784159 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:26.784279 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:31.784362 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:36.784491 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:41.784552 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:46.784674 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:51.784802 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:25:56.784934 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:01.785069 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:06.785187 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:11.785286 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:16.785354 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:21.785476 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:26.785591 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:31.785718 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:36.785848 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:41.785980 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:46.786109 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:51.786233 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:26:56.786358 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:01.786477 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:06.786608 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:11.786736 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:16.786891 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:21.787019 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:26.787149 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:31.787270 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:36.787368 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:41.787495 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:46.787624 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:51.787751 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:27:56.787885 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:01.788016 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:06.788142 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:11.788272 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:16.788356 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:21.788493 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:26.788627 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:31.788767 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:36.788827 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:41.788945 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:46.789070 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:51.789201 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:28:56.789281 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:01.789363 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:06.789487 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:11.789614 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:16.789748 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:21.789872 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:26.790006 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:31.790142 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:36.790264 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:41.790351 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:46.790473 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:51.790600 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:29:56.790723 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:01.790849 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:06.790991 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:11.791136 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:16.791259 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:21.791358 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:26.791495 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:31.791626 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:36.791754 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:41.791883 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:46.792015 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:51.792148 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:30:56.792267 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:01.792354 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:06.792479 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:11.792602 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:16.792739 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:21.792862 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:26.792990 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:31.793108 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:36.793230 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:41.793364 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:46.793489 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:51.793602 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:31:56.793725 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:01.793848 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:06.793974 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:11.794117 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:16.794219 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:21.794349 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:26.794476 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:31.794616 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:36.794743 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:41.794885 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:46.795019 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:51.795159 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:32:56.795276 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:01.795362 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:06.795508 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:11.795628 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:16.795760 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:21.795901 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:26.796032 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:31.796166 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:36.796258 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:41.796357 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:46.796486 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:51.796610 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:33:56.796743 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:01.796884 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:06.797008 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:11.797146 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:16.797283 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:21.797424 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:26.797559 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:31.797692 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:36.797825 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:41.797959 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:46.798096 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:51.798224 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:34:56.798284 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:01.798360 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:06.798507 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:11.798648 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:16.798780 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:21.798916 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:26.799060 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:31.799225 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:36.799378 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:41.799514 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:46.799645 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
I0207 21:35:51.799777 19 admin.go:686] Error reading GraphQL schema: Please retry again, server is not ready to accept requests.
++ hostname
+ [[ t1-dgraph-zero-0 =~ -([0-9]+)$ ]]
+ ordinal=0
+ idx=1
+ [[ 0 -eq 0 ]]
++ awk '{gsub(/\.$/,""); print $0}'
++ hostname -f
+ exec dgraph zero --my=t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080 --idx 1 --replicas 5
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:23:33 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:23:33 Integration installed: Environment
[Sentry] 2021/02/07 21:23:33 Integration installed: Modules
[Sentry] 2021/02/07 21:23:33 Integration installed: IgnoreErrors
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:23:33 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:23:33 Integration installed: Environment
[Sentry] 2021/02/07 21:23:33 Integration installed: Modules
[Sentry] 2021/02/07 21:23:33 Integration installed: IgnoreErrors
I0207 21:23:33.989497 21 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0207 21:23:34.123959 21 init.go:107]
Dgraph version : v20.11.1
Dgraph codename : tchalla-1
Dgraph SHA-256 : cefdcc880c0607a92a1d8d3ba0beb015459ebe216e79fdad613eb0d00d09f134
Commit SHA-1 : 7153d13fe
Commit timestamp : 2021-01-28 15:59:35 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit https://discuss.dgraph.io.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
I0207 21:23:34.124018 21 run.go:185] Setting Config to: {bindall:true portOffset:0 nodeId:1 numReplicas:5 peer: w:zw rebalanceInterval:480000000000 tlsClientConfig:<nil>}
I0207 21:23:34.124051 21 run.go:98] Setting up grpc listener at: 0.0.0.0:5080
I0207 21:23:34.124212 21 run.go:98] Setting up http listener at: 0.0.0.0:6080
I0207 21:23:34.127239 21 storage.go:132] Init Raft Storage with snap: 0, first: 1, last: 0
I0207 21:23:34.127436 21 node.go:152] Setting raft.Config to: &{ID:1 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc000112410 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x2e0ddf8 DisableProposalForwarding:false}
I0207 21:23:34.130538 21 node.go:326] Group 0 found 0 entries
I0207 21:23:34.130578 21 raft.go:602] Starting a brand new node
I0207 21:23:34.130598 21 log.go:34] 1 became follower at term 0
I0207 21:23:34.130604 21 log.go:34] newRaft 1 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0207 21:23:34.130764 21 log.go:34] 1 became follower at term 1
I0207 21:23:34.131193 21 run.go:303] Running Dgraph Zero...
E0207 21:23:34.131832 21 raft.go:486] While proposing CID: Not Zero leader. Aborting proposal: cid:"63066c21-49bf-4685-b51a-d2d15540e5b2" . Retrying...
I0207 21:23:34.131576 21 node.go:189] Setting conf state to nodes:1
I0207 21:23:34.132202 21 raft.go:880] Done applying conf change at 0x1
I0207 21:23:35.340208 21 zero.go:432] Got connection request: cluster_info_only:true
I0207 21:23:35.340301 21 log.go:34] 1 no leader at term 1; dropping index reading msg
I0207 21:23:36.931256 21 log.go:34] 1 is starting a new election at term 1
I0207 21:23:36.931275 21 log.go:34] 1 became pre-candidate at term 1
I0207 21:23:36.931279 21 log.go:34] 1 received MsgPreVoteResp from 1 at term 1
I0207 21:23:36.931292 21 log.go:34] 1 became candidate at term 2
I0207 21:23:36.931296 21 log.go:34] 1 received MsgVoteResp from 1 at term 2
I0207 21:23:36.931304 21 log.go:34] 1 became leader at term 2
I0207 21:23:36.931309 21 log.go:34] raft.node: 1 elected leader 1 at term 2
I0207 21:23:36.931336 21 raft.go:845] I've become the leader, updating leases.
I0207 21:23:36.931349 21 assign.go:42] Updated Lease id: 1. Txn Ts: 1
E0207 21:23:37.132350 21 raft.go:486] While proposing CID: Not Zero leader. Aborting proposal: cid:"121154e7-04ef-48a5-bbb5-73ed4a919fa6" . Retrying...
W0207 21:23:37.340379 21 node.go:681] [0x1] Read index context timed out
I0207 21:23:37.340524 21 zero.go:450] Connected: cluster_info_only:true
I0207 21:23:37.342098 21 zero.go:432] Got connection request: addr:"t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080"
I0207 21:23:37.342259 21 pool.go:162] CONNECTING to t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:23:37.342394 21 zero.go:584] Connected: id:1 group_id:1 addr:"t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080"
W0207 21:23:37.346085 21 pool.go:267] Connection lost with t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local: no such host"
I0207 21:23:40.132615 21 raft.go:479] CID set for cluster: cd560ae5-1eaa-4650-bc41-d0c6e3afbbd6
I0207 21:23:40.132686 21 license_ee.go:45] Enterprise trial license proposed to the cluster: license:<maxNodes:18446744073709551615 expiryTs:1615325020 >
I0207 21:23:48.221353 21 pool.go:162] CONNECTING to t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:23:48.221370 21 node.go:586] Trying to add 0x2 to cluster. Addr: t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:23:48.221375 21 node.go:587] Current confstate at 0x1: nodes:1
I0207 21:23:48.221521 21 node.go:189] Setting conf state to nodes:1 nodes:2
I0207 21:23:48.221551 21 raft.go:880] Done applying conf change at 0x1
I0207 21:23:48.221577 21 node.go:756] [0x2] Done joining cluster with err: <nil>
W0207 21:23:48.223979 21 pool.go:267] Connection lost with t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local: no such host"
I0207 21:24:02.565043 21 pool.go:162] CONNECTING to t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:02.565059 21 node.go:586] Trying to add 0x3 to cluster. Addr: t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:02.565064 21 node.go:587] Current confstate at 0x1: nodes:1 nodes:2
I0207 21:24:02.565718 21 node.go:189] Setting conf state to nodes:1 nodes:2 nodes:3
I0207 21:24:02.565746 21 node.go:756] [0x3] Done joining cluster with err: <nil>
I0207 21:24:02.565764 21 raft.go:880] Done applying conf change at 0x1
W0207 21:24:02.567323 21 pool.go:267] Connection lost with t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local: no such host"
I0207 21:24:06.883308 21 zero.go:432] Got connection request: cluster_info_only:true
I0207 21:24:06.884012 21 zero.go:450] Connected: cluster_info_only:true
I0207 21:24:06.884768 21 zero.go:432] Got connection request: addr:"t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080"
I0207 21:24:06.885431 21 pool.go:162] CONNECTING to t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:24:06.886101 21 zero.go:584] Connected: id:2 group_id:1 addr:"t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080"
W0207 21:24:08.889270 21 pool.go:267] Connection lost with t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local: no such host"
I0207 21:24:34.131481 21 raft.go:719] Skipping creating a snapshot. Num groups: 1, Num checkpoints: 0
++ hostname
+ [[ t1-dgraph-zero-1 =~ -([0-9]+)$ ]]
+ ordinal=1
+ idx=2
+ [[ 1 -eq 0 ]]
++ awk '{gsub(/\.$/,""); print $0}'
++ hostname -f
+ exec dgraph zero --my=t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080 --peer t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080 --idx 2 --replicas 5
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:23:47 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:23:47 Integration installed: Environment
[Sentry] 2021/02/07 21:23:47 Integration installed: Modules
[Sentry] 2021/02/07 21:23:47 Integration installed: IgnoreErrors
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:23:47 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:23:47 Integration installed: Environment
[Sentry] 2021/02/07 21:23:47 Integration installed: Modules
[Sentry] 2021/02/07 21:23:47 Integration installed: IgnoreErrors
I0207 21:23:48.077596 21 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0207 21:23:48.213093 21 init.go:107]
Dgraph version : v20.11.1
Dgraph codename : tchalla-1
Dgraph SHA-256 : cefdcc880c0607a92a1d8d3ba0beb015459ebe216e79fdad613eb0d00d09f134
Commit SHA-1 : 7153d13fe
Commit timestamp : 2021-01-28 15:59:35 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit https://discuss.dgraph.io.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
I0207 21:23:48.213142 21 run.go:185] Setting Config to: {bindall:true portOffset:0 nodeId:2 numReplicas:5 peer:t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080 w:zw rebalanceInterval:480000000000 tlsClientConfig:<nil>}
I0207 21:23:48.213169 21 run.go:98] Setting up grpc listener at: 0.0.0.0:5080
I0207 21:23:48.213324 21 run.go:98] Setting up http listener at: 0.0.0.0:6080
I0207 21:23:48.214991 21 storage.go:132] Init Raft Storage with snap: 0, first: 1, last: 0
I0207 21:23:48.215214 21 node.go:152] Setting raft.Config to: &{ID:2 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc0000ccfa0 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x2e0ddf8 DisableProposalForwarding:false}
I0207 21:23:48.217978 21 node.go:326] Group 0 found 0 entries
I0207 21:23:48.218521 21 pool.go:162] CONNECTING to t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:23:48.222105 21 raft.go:598] [0x2] Starting node
I0207 21:23:48.222125 21 log.go:34] 2 became follower at term 0
I0207 21:23:48.222131 21 log.go:34] newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0207 21:23:48.222135 21 log.go:34] 2 became follower at term 1
I0207 21:23:48.222247 21 run.go:303] Running Dgraph Zero...
I0207 21:23:49.231713 21 log.go:34] 2 [term: 1] received a MsgHeartbeat message with higher term from 1 [term: 2]
I0207 21:23:49.231736 21 log.go:34] 2 became follower at term 2
I0207 21:23:49.231743 21 log.go:34] raft.node: 2 elected leader 1 at term 2
I0207 21:23:50.233424 21 node.go:189] Setting conf state to nodes:1
I0207 21:23:50.233496 21 raft.go:880] Done applying conf change at 0x2
I0207 21:23:50.233582 21 pool.go:162] CONNECTING to t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
W0207 21:23:50.233618 21 pool.go:204] Shutting down extra connection to t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:23:50.233763 21 node.go:189] Setting conf state to nodes:1 nodes:2
I0207 21:23:50.233807 21 raft.go:880] Done applying conf change at 0x2
W0207 21:23:50.235922 21 pool.go:267] Connection lost with t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080. Error: rpc error: code = Canceled desc = context canceled
I0207 21:24:02.565947 21 node.go:189] Setting conf state to nodes:1 nodes:2 nodes:3
I0207 21:24:02.565990 21 pool.go:162] CONNECTING to t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:02.565996 21 raft.go:880] Done applying conf change at 0x2
I0207 21:24:06.886367 21 pool.go:162] CONNECTING to t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
W0207 21:24:06.889129 21 pool.go:267] Connection lost with t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local: no such host"
++ hostname
+ [[ t1-dgraph-zero-2 =~ -([0-9]+)$ ]]
+ ordinal=2
+ idx=3
+ [[ 2 -eq 0 ]]
++ awk '{gsub(/\.$/,""); print $0}'
++ hostname -f
+ exec dgraph zero --my=t1-dgraph-zero-2.t1-dgraph-zero-headless.default.svc.cluster.local:5080 --peer t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080 --idx 3 --replicas 5
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:24:01 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:24:01 Integration installed: Environment
[Sentry] 2021/02/07 21:24:01 Integration installed: Modules
[Sentry] 2021/02/07 21:24:01 Integration installed: IgnoreErrors
[Decoder]: Using assembly version of decoder
Page Size: 4096
[Sentry] 2021/02/07 21:24:02 Integration installed: ContextifyFrames
[Sentry] 2021/02/07 21:24:02 Integration installed: Environment
[Sentry] 2021/02/07 21:24:02 Integration installed: Modules
[Sentry] 2021/02/07 21:24:02 Integration installed: IgnoreErrors
I0207 21:24:02.416894 21 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling.
I0207 21:24:02.552844 21 init.go:107]
Dgraph version : v20.11.1
Dgraph codename : tchalla-1
Dgraph SHA-256 : cefdcc880c0607a92a1d8d3ba0beb015459ebe216e79fdad613eb0d00d09f134
Commit SHA-1 : 7153d13fe
Commit timestamp : 2021-01-28 15:59:35 +0530
Branch : HEAD
Go version : go1.15.5
jemalloc enabled : true
For Dgraph official documentation, visit https://dgraph.io/docs/.
For discussions about Dgraph , visit https://discuss.dgraph.io.
Licensed variously under the Apache Public License 2.0 and Dgraph Community License.
Copyright 2015-2020 Dgraph Labs, Inc.
I0207 21:24:02.552890 21 run.go:185] Setting Config to: {bindall:true portOffset:0 nodeId:3 numReplicas:5 peer:t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080 w:zw rebalanceInterval:480000000000 tlsClientConfig:<nil>}
I0207 21:24:02.552924 21 run.go:98] Setting up grpc listener at: 0.0.0.0:5080
I0207 21:24:02.553132 21 run.go:98] Setting up http listener at: 0.0.0.0:6080
I0207 21:24:02.557303 21 storage.go:132] Init Raft Storage with snap: 0, first: 1, last: 0
I0207 21:24:02.557499 21 node.go:152] Setting raft.Config to: &{ID:3 peers:[] learners:[] ElectionTick:20 HeartbeatTick:1 Storage:0xc0002f0280 Applied:0 MaxSizePerMsg:262144 MaxCommittedSizePerReady:67108864 MaxUncommittedEntriesSize:0 MaxInflightMsgs:256 CheckQuorum:false PreVote:true ReadOnlyOption:0 Logger:0x2e0ddf8 DisableProposalForwarding:false}
I0207 21:24:02.560201 21 node.go:326] Group 0 found 0 entries
I0207 21:24:02.560443 21 pool.go:162] CONNECTING to t1-dgraph-zero-0.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:02.566625 21 raft.go:598] [0x3] Starting node
I0207 21:24:02.566649 21 log.go:34] 3 became follower at term 0
I0207 21:24:02.566655 21 log.go:34] newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
I0207 21:24:02.566660 21 log.go:34] 3 became follower at term 1
I0207 21:24:02.566757 21 run.go:303] Running Dgraph Zero...
I0207 21:24:03.632207 21 log.go:34] 3 [term: 1] received a MsgHeartbeat message with higher term from 1 [term: 2]
I0207 21:24:03.632233 21 log.go:34] 3 became follower at term 2
I0207 21:24:03.632240 21 log.go:34] raft.node: 3 elected leader 1 at term 2
I0207 21:24:04.635248 21 node.go:189] Setting conf state to nodes:1
I0207 21:24:04.635302 21 raft.go:880] Done applying conf change at 0x3
I0207 21:24:04.635448 21 pool.go:162] CONNECTING to t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
W0207 21:24:04.635471 21 pool.go:204] Shutting down extra connection to t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
I0207 21:24:04.635499 21 node.go:189] Setting conf state to nodes:1 nodes:2
I0207 21:24:04.635734 21 raft.go:880] Done applying conf change at 0x3
I0207 21:24:04.635523 21 pool.go:162] CONNECTING to t1-dgraph-zero-1.t1-dgraph-zero-headless.default.svc.cluster.local:5080
I0207 21:24:04.635903 21 node.go:189] Setting conf state to nodes:1 nodes:2 nodes:3
I0207 21:24:04.635945 21 raft.go:880] Done applying conf change at 0x3
W0207 21:24:04.637506 21 pool.go:267] Connection lost with t1-dgraph-alpha-0.t1-dgraph-alpha-headless.default.svc.cluster.local:7080. Error: rpc error: code = Canceled desc = context canceled
I0207 21:24:06.886725 21 pool.go:162] CONNECTING to t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080
W0207 21:24:06.891832 21 pool.go:267] Connection lost with t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local:7080. Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup t1-dgraph-alpha-1.t1-dgraph-alpha-headless.default.svc.cluster.local: no such host"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment