Skip to content

Instantly share code, notes, and snippets.

@therobot

therobot/node B.log Secret

Created July 22, 2013 16:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save therobot/ef85646d6e0092e494c4 to your computer and use it in GitHub Desktop.
Save therobot/ef85646d6e0092e494c4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
node B.log
Jul 22 15:27:06 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30211 exited with return code 0
Jul 22 15:27:16 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30250 exited with return code 0
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_replace_notify: Replaced: 0.198.8 -> 0.199.1 from <null>
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <cib admin_epoch="0" epoch="198" num_updates="8" >
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <configuration >
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <nodes >
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <node id="ip-10-33-193-70" type="normal" uname="ip-10-33-193-70" __crm_diff_marker__="removed:top" />
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </nodes>
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </configuration>
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </cib>
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <cib epoch="199" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-33-167-102" update-client="crmd" cib-last-written="Mon Jul 22 15:22:08 2013" have-quorum="0" dc-uuid="ip-10-33-167-102" />
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_replace for section 'all' (origin=local/cibadmin/2, version=0.199.1): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 attrd: [27044]: info: do_cib_replaced: Sending full refresh
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:124 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.199.1) : Non-status change
Jul 22 15:27:17 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 153: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ]
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: update_dc: Unset DC ip-10-33-167-102
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <cib admin_epoch="0" epoch="199" num_updates="1" />
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <cib epoch="200" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-33-167-102" update-client="cibadmin" cib-last-written="Mon Jul 22 15:27:17 2013" have-quorum="0" dc-uuid="ip-10-33-167-102" >
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <configuration >
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <nodes >
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <node id="ip-10-33-193-70" uname="ip-10-33-193-70" type="normal" __crm_diff_marker__="added:top" />
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </nodes>
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </configuration>
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </cib>
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/151, version=0.200.1): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_dc_takeover: Taking over DC status for this partition
Jul 22 15:27:17 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374503889)
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/154, version=0.200.4): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/155, version=0.200.5): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/157, version=0.200.7): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/159, version=0.200.8): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_offer_all: join-5: Waiting on 1 outstanding join acks
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Checking for expired actions every 900000ms
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Sending expected-votes=2 to corosync
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: ais_dispatch_message: Membership 19824: quorum still lost
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: ais_dispatch_message: Membership 19824: quorum still lost
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/162, version=0.200.9): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: Managed write_cib_contents process 30271 exited with return code 0.
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/164, version=0.200.10): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: update_dc: Set DC to ip-10-33-167-102 (3.0.5)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_finalize: join-5: Syncing the CIB from ip-10-33-167-102 to the rest of the cluster
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/166, version=0.200.10): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/167, version=0.200.11): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_ack: join-5: Updating node state to member for ip-10-33-167-102
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ip-10-33-167-102']/lrm (origin=local/crmd/168, version=0.200.12): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ip-10-33-167-102']/lrm": ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/170, version=0.200.14): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: crm_update_quorum: Updating quorum status to false (call=172)
Jul 22 15:27:17 ip-10-33-167-102 attrd: [27044]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/172, version=0.200.16): ok (rc=0)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: do_te_invoke:167 - Triggered transition abort (complete=1) : Peer Cancelled
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 173: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: WARN: match_down_event: No match for shutdown action on ip-10-33-193-70
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: te_update_diff: Stonith/shutdown of ip-10-33-193-70 not matched
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:215 - Triggered transition abort (complete=1, tag=node_state, id=ip-10-33-193-70, magic=NA, cib=0.200.15) : Node failure
Jul 22 15:27:17 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=173, ref=pe_calc-dc-1374506837-74, seq=19824, quorate=0
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 174: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=174, ref=pe_calc-dc-1374506837-75, seq=19824, quorate=0
Jul 22 15:27:17 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374503889)
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102
Jul 22 15:27:17 ip-10-33-167-102 cib: [27042]: info: Managed write_cib_contents process 30272 exited with return code 0.
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Started ip-10-33-167-102)
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Started ip-10-33-167-102)
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: handle_response: pe_calc calculation pe_calc-dc-1374506837-74 is obsolete
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 12: PEngine Input stored in: /var/lib/pengine/pe-input-442.bz2
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Started ip-10-33-167-102)
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Started ip-10-33-167-102)
Jul 22 15:27:17 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 13: PEngine Input stored in: /var/lib/pengine/pe-input-442.bz2
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: unpack_graph: Unpacked transition 13: 0 actions in 0 synapses
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_te_invoke: Processing graph 13 (ref=pe_calc-dc-1374506837-75) derived from /var/lib/pengine/pe-input-442.bz2
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: run_graph: ====================================================
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: notice: run_graph: Transition 13 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-442.bz2): Complete
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: te_graph_trigger: Transition 13 is now complete
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: notify_crmd: Transition 13 status: done - <null>
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 15:27:17 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Starting PEngine Recheck Timer
Jul 22 15:27:20 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30215 exited with return code 0
Jul 22 15:27:26 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30298 exited with return code 0
Jul 22 15:27:36 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30302 exited with return code 0
Jul 22 15:27:46 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30336 exited with return code 0
Jul 22 15:27:47 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30306 exited with return code 0
Jul 22 15:27:56 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30347 exited with return code 0
Jul 22 15:28:07 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30352 exited with return code 0
Jul 22 15:28:14 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30356 exited with return code 0
Jul 22 15:28:17 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30390 exited with return code 0
Jul 22 15:28:27 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30394 exited with return code 0
Jul 22 15:28:37 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30420 exited with return code 0
Jul 22 15:28:40 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30398 exited with return code 0
Jul 22 15:28:47 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30436 exited with return code 0
Jul 22 15:28:57 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30440 exited with return code 0
Jul 22 15:29:07 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30474 exited with return code 0
Jul 22 15:29:07 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30444 exited with return code 0
Jul 22 15:29:17 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30482 exited with return code 0
Jul 22 15:29:27 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30486 exited with return code 0
Jul 22 15:29:34 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30490 exited with return code 0
Jul 22 15:29:37 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30524 exited with return code 0
Jul 22 15:29:47 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30528 exited with return code 0
Jul 22 15:29:57 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30554 exited with return code 0
Jul 22 15:30:00 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30532 exited with return code 0
Jul 22 15:30:07 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30570 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19828: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19828: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:30:17 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30574 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19832: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19832: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:30:27 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30578 exited with return code 0
Jul 22 15:30:27 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30612 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19836: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19836: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jul 22 15:30:37 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30616 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19840: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19840: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:30:47 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30632 exited with return code 0
Jul 22 15:30:54 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30620 exited with return code 0
Jul 22 15:30:57 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30658 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19844: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19844: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:31:07 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30662 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19848: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19848: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jul 22 15:31:17 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30688 exited with return code 0
Jul 22 15:31:21 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30666 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19852: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19852: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:31:27 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30704 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19856: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19856: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:31:37 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30708 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19860: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19860: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:31:47 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30742 exited with return code 0
Jul 22 15:31:47 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30712 exited with return code 0
Jul 22 15:31:57 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30750 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19864: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19864: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:32:07 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30754 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19868: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19868: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:32:14 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30758 exited with return code 0
Jul 22 15:32:17 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30792 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19872: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19872: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:32:27 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30837 exited with return code 0
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib_replace_notify: Replaced: 0.200.18 -> 0.201.1 from <null>
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <cib admin_epoch="0" epoch="200" num_updates="18" >
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <configuration >
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <nodes >
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <node id="ip-10-33-193-70" uname="ip-10-33-193-70" type="normal" __crm_diff_marker__="removed:top" />
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </nodes>
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </configuration>
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </cib>
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <cib epoch="201" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-33-167-102" update-client="crmd" cib-last-written="Mon Jul 22 15:27:17 2013" have-quorum="0" dc-uuid="ip-10-33-167-102" />
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_replace for section 'all' (origin=local/cibadmin/2, version=0.201.1): ok (rc=0)
Jul 22 15:32:30 ip-10-33-167-102 attrd: [27044]: info: do_cib_replaced: Sending full refresh
Jul 22 15:32:30 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:124 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.201.1) : Non-status change
Jul 22 15:32:30 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jul 22 15:32:30 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jul 22 15:32:30 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <cib admin_epoch="0" epoch="201" num_updates="1" />
Jul 22 15:32:30 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374503889)
Jul 22 15:32:30 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 177: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <cib epoch="202" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-33-167-102" update-client="cibadmin" cib-last-written="Mon Jul 22 15:32:30 2013" have-quorum="0" dc-uuid="ip-10-33-167-102" >
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <configuration >
Jul 22 15:32:30 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ]
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <nodes >
Jul 22 15:32:30 ip-10-33-167-102 crmd: [27046]: info: update_dc: Unset DC ip-10-33-167-102
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <node id="ip-10-33-193-70" uname="ip-10-33-193-70" type="normal" __crm_diff_marker__="added:top" />
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </nodes>
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </configuration>
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </cib>
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/175, version=0.202.1): ok (rc=0)
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: Managed write_cib_contents process 30872 exited with return code 0.
Jul 22 15:32:30 ip-10-33-167-102 cib: [27042]: info: Managed write_cib_contents process 30874 exited with return code 0.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19876: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19876: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: do_dc_takeover: Taking over DC status for this partition
Jul 22 15:32:35 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/178, version=0.202.5): ok (rc=0)
Jul 22 15:32:35 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/179, version=0.202.6): ok (rc=0)
Jul 22 15:32:35 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/181, version=0.202.7): ok (rc=0)
Jul 22 15:32:35 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/183, version=0.202.8): ok (rc=0)
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_offer_all: join-6: Waiting on 1 outstanding join acks
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Checking for expired actions every 900000ms
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Sending expected-votes=2 to corosync
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: ais_dispatch_message: Membership 19876: quorum still lost
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:32:35 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/186, version=0.202.9): ok (rc=0)
Jul 22 15:32:35 ip-10-33-167-102 crmd: [27046]: info: ais_dispatch_message: Membership 19876: quorum still lost
Jul 22 15:32:36 ip-10-33-167-102 crmd: [27046]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:32:36 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/188, version=0.202.10): ok (rc=0)
Jul 22 15:32:38 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30898 exited with return code 0
Jul 22 15:32:41 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30876 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19880: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19880: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: update_dc: Set DC to ip-10-33-167-102 (3.0.5)
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: check_join_state: do_dc_join_filter_offer: Membership changed since join started: 19824 -> 19876
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: update_dc: Unset DC ip-10-33-167-102
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: join_make_offer: Making join offers based on membership 19876
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_offer_all: join-7: Waiting on 1 outstanding join acks
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: update_dc: Set DC to ip-10-33-167-102 (3.0.5)
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
Jul 22 15:32:48 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_finalize: join-7: Syncing the CIB from ip-10-33-167-102 to the rest of the cluster
Jul 22 15:32:48 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/191, version=0.202.10): ok (rc=0)
Jul 22 15:32:48 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/192, version=0.202.11): ok (rc=0)
Jul 22 15:32:48 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30939 exited with return code 0
Jul 22 15:32:54 ip-10-33-167-102 cib: [27042]: info: cib_stats: Processed 71 operations (3239.00us average, 0% utilization) in the last 10min
Jul 22 15:32:58 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30948 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19884: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19884: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_ack: join-7: Updating node state to member for ip-10-33-167-102
Jul 22 15:33:00 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ip-10-33-167-102']/lrm (origin=local/crmd/193, version=0.202.12): ok (rc=0)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ip-10-33-167-102']/lrm": ok (rc=0)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:33:00 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/195, version=0.202.14): ok (rc=0)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Jul 22 15:33:00 ip-10-33-167-102 attrd: [27044]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: crm_update_quorum: Updating quorum status to false (call=197)
Jul 22 15:33:00 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/197, version=0.202.16): ok (rc=0)
Jul 22 15:33:00 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: do_te_invoke:167 - Triggered transition abort (complete=1) : Peer Cancelled
Jul 22 15:33:00 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374503889)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 198: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: WARN: match_down_event: No match for shutdown action on ip-10-33-193-70
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: te_update_diff: Stonith/shutdown of ip-10-33-193-70 not matched
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:215 - Triggered transition abort (complete=1, tag=node_state, id=ip-10-33-193-70, magic=NA, cib=0.202.15) : Node failure
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=198, ref=pe_calc-dc-1374507180-83, seq=19876, quorate=0
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 199: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=199, ref=pe_calc-dc-1374507180-84, seq=19876, quorate=0
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Started ip-10-33-167-102)
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Started ip-10-33-167-102)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: handle_response: pe_calc calculation pe_calc-dc-1374507180-83 is obsolete
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 14: PEngine Input stored in: /var/lib/pengine/pe-input-443.bz2
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Started ip-10-33-167-102)
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Started ip-10-33-167-102)
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: unpack_graph: Unpacked transition 15: 0 actions in 0 synapses
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_te_invoke: Processing graph 15 (ref=pe_calc-dc-1374507180-84) derived from /var/lib/pengine/pe-input-444.bz2
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: run_graph: ====================================================
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: notice: run_graph: Transition 15 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-444.bz2): Complete
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: te_graph_trigger: Transition 15 is now complete
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: notify_crmd: Transition 15 status: done - <null>
Jul 22 15:33:00 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 15: PEngine Input stored in: /var/lib/pengine/pe-input-444.bz2
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 15:33:00 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Starting PEngine Recheck Timer
Jul 22 15:33:08 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30982 exited with return code 0
Jul 22 15:33:08 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30952 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19888: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19888: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:33:18 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30990 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19892: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19892: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:33:28 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 30994 exited with return code 0
Jul 22 15:33:35 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 30998 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19896: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19896: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:33:38 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31032 exited with return code 0
Jul 22 15:33:48 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31036 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19900: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19900: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:33:58 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31062 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19904: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19904: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:34:01 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31040 exited with return code 0
Jul 22 15:34:08 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31078 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19908: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19908: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:34:18 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31082 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19912: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19912: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:34:28 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31086 exited with return code 0
Jul 22 15:34:28 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31120 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19916: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19916: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:34:38 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31124 exited with return code 0
Jul 22 15:34:48 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31141 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19920: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19920: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:34:54 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31128 exited with return code 0
Jul 22 15:34:58 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31166 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19924: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19924: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:35:08 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31173 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19928: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19928: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:35:18 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31237 exited with return code 0
Jul 22 15:35:21 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31215 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19932: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19932: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:35:28 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31253 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19936: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19936: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:35:38 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31257 exited with return code 0
Jul 22 15:35:48 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31261 exited with return code 0
Jul 22 15:35:48 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31295 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19940: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19940: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:35:58 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31299 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19944: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19944: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:36:08 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31324 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19948: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19948: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:36:15 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31303 exited with return code 0
Jul 22 15:36:18 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31341 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19952: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19952: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:36:28 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31345 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19956: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19956: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:36:38 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31371 exited with return code 0
Jul 22 15:36:41 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31349 exited with return code 0
Jul 22 15:36:48 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31387 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19960: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19960: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:36:58 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31391 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19964: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19964: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:37:08 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31395 exited with return code 0
Jul 22 15:37:08 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31429 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19968: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19968: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:37:18 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31433 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19972: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19972: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:37:29 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31450 exited with return code 0
Jul 22 15:37:35 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31437 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19976: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19976: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:37:39 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31475 exited with return code 0
Jul 22 15:37:49 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31501 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19980: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19980: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:37:59 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31527 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19984: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19984: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:38:02 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31505 exited with return code 0
Jul 22 15:38:09 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31543 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19988: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19988: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jul 22 15:38:19 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31547 exited with return code 0
^Z
[1]+ Stopped tail -f /var/log/corosync.log | grep --color=auto -v debug
root@dev-lb04.bebanjo.net:/home/ubuntu# crm_status
crm_status: command not found
root@dev-lb04.bebanjo.net:/home/ubuntu# crm status
============
Last updated: Mon Jul 22 15:38:27 2013
Last change: Mon Jul 22 15:32:30 2013 via crmd on ip-10-33-167-102
Stack: openais
Current DC: ip-10-33-167-102 - partition WITHOUT quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ ip-10-33-167-102 ]
OFFLINE: [ ip-10-33-193-70 ]
Resource Group: ha
elasticip (ocf::aws:elasticip): Started ip-10-33-167-102
email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
root@dev-lb04.bebanjo.net:/home/ubuntu# fg
tail -f /var/log/corosync.log | grep --color=auto -v debug
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19992: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19992: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:38:29 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31602 exited with return code 0
Jul 22 15:38:29 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31551 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 19996: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 19996: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:38:39 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31610 exited with return code 0
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib_replace_notify: Replaced: 0.202.18 -> 0.203.1 from <null>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <cib admin_epoch="0" epoch="202" num_updates="18" >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <configuration >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <nodes >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <node id="ip-10-33-193-70" uname="ip-10-33-193-70" type="normal" __crm_diff_marker__="removed:top" />
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </nodes>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </configuration>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - </cib>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <cib epoch="203" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-33-167-102" update-client="crmd" cib-last-written="Mon Jul 22 15:32:30 2013" have-quorum="0" dc-uuid="ip-10-33-167-102" >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <configuration >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <nodes >
Jul 22 15:38:48 ip-10-33-167-102 attrd: [27044]: info: do_cib_replaced: Sending full refresh
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <node id="ip-10-34-184-243" type="normal" uname="ip-10-34-184-243" __crm_diff_marker__="added:top" />
Jul 22 15:38:48 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </nodes>
Jul 22 15:38:48 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:124 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.203.1) : Non-status change
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </configuration>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </cib>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_replace for section 'all' (origin=local/cibadmin/2, version=0.203.1): ok (rc=0)
Jul 22 15:38:48 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jul 22 15:38:48 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:38:48 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374503889)
Jul 22 15:38:48 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 202: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:38:48 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ]
Jul 22 15:38:48 ip-10-33-167-102 crmd: [27046]: info: update_dc: Unset DC ip-10-33-167-102
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: - <cib admin_epoch="0" epoch="203" num_updates="2" />
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <cib epoch="204" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-33-167-102" update-client="cibadmin" cib-last-written="Mon Jul 22 15:38:48 2013" have-quorum="0" dc-uuid="ip-10-33-167-102" >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <configuration >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <nodes >
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + <node id="ip-10-33-193-70" uname="ip-10-33-193-70" type="normal" __crm_diff_marker__="added:top" />
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </nodes>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </configuration>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib:diff: + </cib>
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/200, version=0.204.1): ok (rc=0)
Jul 22 15:38:48 ip-10-33-167-102 cib: [27042]: info: Managed write_cib_contents process 31646 exited with return code 0.
Jul 22 15:38:49 ip-10-33-167-102 cib: [27042]: info: Managed write_cib_contents process 31649 exited with return code 0.
Jul 22 15:38:49 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31650 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20000: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20000: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: do_dc_takeover: Taking over DC status for this partition
Jul 22 15:38:49 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/203, version=0.204.4): ok (rc=0)
Jul 22 15:38:49 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/204, version=0.204.5): ok (rc=0)
Jul 22 15:38:49 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/206, version=0.204.6): ok (rc=0)
Jul 22 15:38:49 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/208, version=0.204.7): ok (rc=0)
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_offer_all: join-8: Waiting on 1 outstanding join acks
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Checking for expired actions every 900000ms
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: config_query_callback: Sending expected-votes=2 to corosync
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: ais_dispatch_message: Membership 20000: quorum still lost
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:38:49 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/211, version=0.204.8): ok (rc=0)
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: ais_dispatch_message: Membership 20000: quorum still lost
Jul 22 15:38:49 ip-10-33-167-102 crmd: [27046]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:38:49 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/213, version=0.204.9): ok (rc=0)
Jul 22 15:38:56 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31654 exited with return code 0
Jul 22 15:38:59 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31688 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20004: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20004: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: update_dc: Set DC to ip-10-33-167-102 (3.0.5)
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: check_join_state: do_dc_join_filter_offer: Membership changed since join started: 19876 -> 20000
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: update_dc: Unset DC ip-10-33-167-102
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: join_make_offer: Making join offers based on membership 20000
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_offer_all: join-9: Waiting on 1 outstanding join acks
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: update_dc: Set DC to ip-10-33-167-102 (3.0.5)
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
Jul 22 15:39:01 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_finalize: join-9: Syncing the CIB from ip-10-33-167-102 to the rest of the cluster
Jul 22 15:39:01 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/216, version=0.204.9): ok (rc=0)
Jul 22 15:39:01 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/217, version=0.204.10): ok (rc=0)
Jul 22 15:39:09 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31741 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20008: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20008: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_ack: join-9: Updating node state to member for ip-10-33-167-102
Jul 22 15:39:13 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ip-10-33-167-102']/lrm (origin=local/crmd/218, version=0.204.11): ok (rc=0)
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ip-10-33-167-102']/lrm": ok (rc=0)
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:39:13 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/220, version=0.204.13): ok (rc=0)
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: crm_update_quorum: Updating quorum status to false (call=222)
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: do_te_invoke:167 - Triggered transition abort (complete=1) : Peer Cancelled
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 223: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:39:13 ip-10-33-167-102 attrd: [27044]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Jul 22 15:39:13 ip-10-33-167-102 cib: [27042]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/222, version=0.204.15): ok (rc=0)
Jul 22 15:39:13 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jul 22 15:39:13 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374503889)
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: WARN: match_down_event: No match for shutdown action on ip-10-33-193-70
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: te_update_diff: Stonith/shutdown of ip-10-33-193-70 not matched
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:215 - Triggered transition abort (complete=1, tag=node_state, id=ip-10-33-193-70, magic=NA, cib=0.204.14) : Node failure
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=223, ref=pe_calc-dc-1374507553-92, seq=20000, quorate=0
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 224: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=224, ref=pe_calc-dc-1374507553-93, seq=20000, quorate=0
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Started ip-10-33-167-102)
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Started ip-10-33-167-102)
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: handle_response: pe_calc calculation pe_calc-dc-1374507553-92 is obsolete
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 16: PEngine Input stored in: /var/lib/pengine/pe-input-445.bz2
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Started ip-10-33-167-102)
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Started ip-10-33-167-102)
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: unpack_graph: Unpacked transition 17: 0 actions in 0 synapses
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_te_invoke: Processing graph 17 (ref=pe_calc-dc-1374507553-93) derived from /var/lib/pengine/pe-input-446.bz2
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: run_graph: ====================================================
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: notice: run_graph: Transition 17 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-446.bz2): Complete
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: te_graph_trigger: Transition 17 is now complete
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: notify_crmd: Transition 17 status: done - <null>
Jul 22 15:39:13 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 17: PEngine Input stored in: /var/lib/pengine/pe-input-446.bz2
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 15:39:13 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Starting PEngine Recheck Timer
Jul 22 15:39:19 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31767 exited with return code 0
Jul 22 15:39:23 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31745 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20012: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20012: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:39:29 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31783 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20016: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20016: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:39:39 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31806 exited with return code 0
Jul 22 15:39:49 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31840 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20020: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20020: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:39:49 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31810 exited with return code 0
Jul 22 15:39:59 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31848 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20024: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20024: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:40:09 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31852 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20028: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20028: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:40:16 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31856 exited with return code 0
Jul 22 15:40:19 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31890 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20032: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20032: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:40:29 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31894 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20036: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20036: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:40:39 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31920 exited with return code 0
Jul 22 15:40:43 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31898 exited with return code 0
Jul 22 15:40:49 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31955 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20040: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20040: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:40:59 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31959 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20044: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20044: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:41:09 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 31993 exited with return code 0
Jul 22 15:41:10 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 31963 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20048: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20048: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:41:19 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32001 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20052: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20052: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:41:29 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32005 exited with return code 0
Jul 22 15:41:37 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32009 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20056: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20056: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:41:39 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32043 exited with return code 0
Jul 22 15:41:49 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32047 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20060: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20060: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:41:59 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32073 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20064: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20064: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:42:04 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32051 exited with return code 0
Jul 22 15:42:09 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32089 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20068: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20068: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:42:19 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32093 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20072: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20072: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:42:29 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32127 exited with return code 0
Jul 22 15:42:30 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32097 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20076: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20076: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:42:39 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32135 exited with return code 0
Jul 22 15:42:49 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32139 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20080: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20080: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:42:54 ip-10-33-167-102 cib: [27042]: info: cib_stats: Processed 57 operations (2982.00us average, 0% utilization) in the last 10min
Jul 22 15:42:57 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32143 exited with return code 0
Jul 22 15:42:59 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32177 exited with return code 0
Jul 22 15:43:09 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32200 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20084: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20084: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:43:20 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32226 exited with return code 0
Jul 22 15:43:24 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32204 exited with return code 0
Jul 22 15:43:30 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32242 exited with return code 0
Jul 22 15:43:40 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32267 exited with return code 0
Jul 22 15:43:50 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32301 exited with return code 0
Jul 22 15:43:51 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32271 exited with return code 0
Jul 22 15:44:00 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32309 exited with return code 0
Jul 22 15:44:10 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32313 exited with return code 0
Jul 22 15:44:18 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32317 exited with return code 0
Jul 22 15:44:20 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32351 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20088: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20088: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:44:30 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32355 exited with return code 0
Jul 22 15:44:40 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32399 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20092: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20092: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:44:44 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32378 exited with return code 0
Jul 22 15:44:50 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32435 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20096: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20096: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:45:00 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[25] on email for client 27046: pid 32440 exited with return code 0
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20100: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20100: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:45:09 ip-10-33-167-102 lrmd: [27043]: info: operation monitor[23] on elasticip for client 27046: pid 32447 exited with return code 7
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: process_lrm_event: LRM operation elasticip_monitor_20000 (call=23, rc=7, cib-update=225, confirmed=false) not running
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: process_graph_event: Detected action elasticip_monitor_20000 from a different transition: 11 vs. 17
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: process_graph_event:485 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=elasticip_monitor_20000, magic=0:7;6:11:0:f76ac39e-13f4-4516-b193-57880345216a, cib=0.204.18) : Old event
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: WARN: update_failcount: Updating failcount for elasticip on ip-10-33-167-102 after failed monitor: rc=7 (update=value++, time=1374507909)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:45:09 ip-10-33-167-102 attrd: [27044]: info: attrd_local_callback: Expanded fail-count-elasticip=value++ to 1
Jul 22 15:45:09 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-elasticip (1)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 226: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:45:09 ip-10-33-167-102 attrd: [27044]: notice: attrd_perform_update: Sent update 115: fail-count-elasticip=1
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=226, ref=pe_calc-dc-1374507909-94, seq=20000, quorate=0
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-ip-10-33-167-102-fail-count-elasticip, name=fail-count-elasticip, value=1, magic=NA, cib=0.204.19) : Transient attribute: update
Jul 22 15:45:09 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374507909)
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:45:09 ip-10-33-167-102 attrd: [27044]: notice: attrd_perform_update: Sent update 117: last-failure-elasticip=1374507909
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 227: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-ip-10-33-167-102-last-failure-elasticip, name=last-failure-elasticip, value=1374507909, magic=NA, cib=0.204.20) : Transient attribute: update
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_monitor_20000 on ip-10-33-167-102: not running (7)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=227, ref=pe_calc-dc-1374507909-95, seq=20000, quorate=0
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102 FAILED
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 228: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=228, ref=pe_calc-dc-1374507909-96, seq=20000, quorate=0
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: RecurringOp: Start recurring monitor (20s) for elasticip on ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Recover elasticip (Started ip-10-33-167-102)
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Restart email (Started ip-10-33-167-102)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: handle_response: pe_calc calculation pe_calc-dc-1374507909-94 is obsolete
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 18: PEngine Input stored in: /var/lib/pengine/pe-input-447.bz2
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_monitor_20000 on ip-10-33-167-102: not running (7)
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102 FAILED
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: get_failcount: elasticip has failed 1 times on ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: common_apply_stickiness: elasticip can fail 999999 more times on ip-10-33-167-102 before being forced off
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: RecurringOp: Start recurring monitor (20s) for elasticip on ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Recover elasticip (Started ip-10-33-167-102)
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Restart email (Started ip-10-33-167-102)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: handle_response: pe_calc calculation pe_calc-dc-1374507909-95 is obsolete
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 19: PEngine Input stored in: /var/lib/pengine/pe-input-448.bz2
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_monitor_20000 on ip-10-33-167-102: not running (7)
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102 FAILED
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Started ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: info: get_failcount: elasticip has failed 1 times on ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: common_apply_stickiness: elasticip can fail 999999 more times on ip-10-33-167-102 before being forced off
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: RecurringOp: Start recurring monitor (20s) for elasticip on ip-10-33-167-102
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Recover elasticip (Started ip-10-33-167-102)
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Restart email (Started ip-10-33-167-102)
Jul 22 15:45:09 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 20: PEngine Input stored in: /var/lib/pengine/pe-input-448.bz2
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: unpack_graph: Unpacked transition 20: 11 actions in 11 synapses
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_te_invoke: Processing graph 20 (ref=pe_calc-dc-1374507909-96) derived from /var/lib/pengine/pe-input-448.bz2
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: te_pseudo_action: Pseudo action 12 fired and confirmed
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: te_rsc_command: Initiating action 8: stop email_stop_0 on ip-10-33-167-102 (local)
Jul 22 15:45:09 ip-10-33-167-102 lrmd: [27043]: info: cancel_op: operation monitor[25] on email for client 27046, its parameters: crm_feature_set=[3.0.5] depth=[0] email=[devops@bebanjo.com] CRM_meta_name=[monitor] CRM_meta_interval=[10000] CRM_meta_timeout=[10000] CRM_meta_depth=[0] cancelled
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_lrm_rsc_op: Performing key=8:20:0:f76ac39e-13f4-4516-b193-57880345216a op=email_stop_0 )
Jul 22 15:45:09 ip-10-33-167-102 lrmd: [27043]: info: rsc:email stop[26] (pid 32480)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: process_lrm_event: LRM operation email_monitor_10000 (call=25, status=1, cib-update=0, confirmed=true) Cancelled
Jul 22 15:45:09 ip-10-33-167-102 lrmd: [27043]: info: Managed email:stop process 32480 exited with return code 0.
Jul 22 15:45:09 ip-10-33-167-102 lrmd: [27043]: info: operation stop[26] on email for client 27046: pid 32480 exited with return code 0
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: process_lrm_event: LRM operation email_stop_0 (call=26, rc=0, cib-update=229, confirmed=true) ok
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: match_graph_event: Action email_stop_0 (8) confirmed on ip-10-33-167-102 (rc=0)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: te_rsc_command: Initiating action 3: stop elasticip_stop_0 on ip-10-33-167-102 (local)
Jul 22 15:45:09 ip-10-33-167-102 lrmd: [27043]: info: cancel_op: operation monitor[23] on elasticip for client 27046, its parameters: crm_feature_set=[3.0.5] CRM_meta_on_fail=[restart] CRM_meta_name=[monitor] CRM_meta_interval=[20000] CRM_meta_timeout=[20000] cancelled
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: do_lrm_rsc_op: Performing key=3:20:0:f76ac39e-13f4-4516-b193-57880345216a op=elasticip_stop_0 )
Jul 22 15:45:09 ip-10-33-167-102 lrmd: [27043]: info: rsc:elasticip stop[27] (pid 32495)
Jul 22 15:45:09 ip-10-33-167-102 crmd: [27046]: info: process_lrm_event: LRM operation elasticip_monitor_20000 (call=23, status=1, cib-update=0, confirmed=true) Cancelled
Jul 22 15:45:14 ip-10-33-167-102 lrmd: [27043]: info: Managed elasticip:stop process 32495 exited with return code 0.
Jul 22 15:45:14 ip-10-33-167-102 lrmd: [27043]: info: operation stop[27] on elasticip for client 27046: pid 32495 exited with return code 0
Jul 22 15:45:14 ip-10-33-167-102 crmd: [27046]: info: process_lrm_event: LRM operation elasticip_stop_0 (call=27, rc=0, cib-update=230, confirmed=true) ok
Jul 22 15:45:14 ip-10-33-167-102 crmd: [27046]: info: match_graph_event: Action elasticip_stop_0 (3) confirmed on ip-10-33-167-102 (rc=0)
Jul 22 15:45:14 ip-10-33-167-102 crmd: [27046]: info: te_pseudo_action: Pseudo action 13 fired and confirmed
Jul 22 15:45:14 ip-10-33-167-102 crmd: [27046]: info: te_pseudo_action: Pseudo action 10 fired and confirmed
Jul 22 15:45:14 ip-10-33-167-102 crmd: [27046]: info: te_rsc_command: Initiating action 7: start elasticip_start_0 on ip-10-33-167-102 (local)
Jul 22 15:45:14 ip-10-33-167-102 crmd: [27046]: info: do_lrm_rsc_op: Performing key=7:20:0:f76ac39e-13f4-4516-b193-57880345216a op=elasticip_start_0 )
Jul 22 15:45:14 ip-10-33-167-102 lrmd: [27043]: info: rsc:elasticip start[28] (pid 32535)
Jul 22 15:45:14 ip-10-33-167-102 crmd: [27046]: info: te_pseudo_action: Pseudo action 4 fired and confirmed
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20104: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20104: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20108: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20108: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20112: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20112: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20116: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20116: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:46:12 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:start:stderr) Unknown problem connecting to host: 'https://eu-west-1.ec2.amazonaws.com'
Jul 22 15:46:12 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:start:stderr)
Jul 22 15:46:12 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:start:stderr) Unable to execute HTTP request: Connect to eu-west-1.ec2.amazonaws.com:443 timed out
Jul 22 15:46:12 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:start:stderr)
Jul 22 15:46:12 ip-10-33-167-102 lrmd: [27043]: WARN: Managed elasticip:start process 32535 exited with return code 1.
Jul 22 15:46:12 ip-10-33-167-102 lrmd: [27043]: info: operation start[28] on elasticip for client 27046: pid 32535 exited with return code 1
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: process_lrm_event: LRM operation elasticip_start_0 (call=28, rc=1, cib-update=231, confirmed=true) unknown error
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: WARN: status_from_rc: Action 7 (elasticip_start_0) on ip-10-33-167-102 failed (target: 0 vs. rc: 1): Error
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: WARN: update_failcount: Updating failcount for elasticip on ip-10-33-167-102 after failed start: rc=1 (update=INFINITY, time=1374507972)
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=elasticip_last_failure_0, magic=0:1;7:20:0:f76ac39e-13f4-4516-b193-57880345216a, cib=0.204.23) : Event failed
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: update_abort_priority: Abort priority upgraded from 0 to 1
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: update_abort_priority: Abort action done superceeded by restart
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: match_graph_event: Action elasticip_start_0 (7) confirmed on ip-10-33-167-102 (rc=4)
Jul 22 15:46:12 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-elasticip (INFINITY)
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: run_graph: ====================================================
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: notice: run_graph: Transition 20 (Complete=7, Pending=0, Fired=0, Skipped=4, Incomplete=0, Source=/var/lib/pengine/pe-input-448.bz2): Stopped
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: te_graph_trigger: Transition 20 is now complete
Jul 22 15:46:12 ip-10-33-167-102 attrd: [27044]: notice: attrd_perform_update: Sent update 119: fail-count-elasticip=INFINITY
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 15:46:12 ip-10-33-167-102 attrd: [27044]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-elasticip (1374507972)
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:46:12 ip-10-33-167-102 attrd: [27044]: notice: attrd_perform_update: Sent update 121: last-failure-elasticip=1374507972
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 232: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-ip-10-33-167-102-fail-count-elasticip, name=fail-count-elasticip, value=INFINITY, magic=NA, cib=0.204.24) : Transient attribute: update
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: abort_transition_graph: te_update_diff:164 - Triggered transition abort (complete=1, tag=nvpair, id=status-ip-10-33-167-102-last-failure-elasticip, name=last-failure-elasticip, value=1374507972, magic=NA, cib=0.204.25) : Transient attribute: update
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=232, ref=pe_calc-dc-1374507972-100, seq=20000, quorate=0
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 233: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 234: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=234, ref=pe_calc-dc-1374507972-101, seq=20000, quorate=0
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_monitor_20000 on ip-10-33-167-102: not running (7)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_last_failure_0 on ip-10-33-167-102: unknown error (1)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102 FAILED
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Stopped
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: get_failcount: elasticip has failed INFINITY times on ip-10-33-167-102
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: WARN: common_apply_stickiness: Forcing elasticip away from ip-10-33-167-102 after 1000000 failures (max=1000000)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: rsc_merge_weights: elasticip: Rolling back scores from email
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource elasticip cannot run anywhere
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource email cannot run anywhere
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Stop elasticip (ip-10-33-167-102)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Stopped)
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: handle_response: pe_calc calculation pe_calc-dc-1374507972-100 is obsolete
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 21: PEngine Input stored in: /var/lib/pengine/pe-input-449.bz2
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_monitor_20000 on ip-10-33-167-102: not running (7)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_last_failure_0 on ip-10-33-167-102: unknown error (1)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Started ip-10-33-167-102 FAILED
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Stopped
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: get_failcount: elasticip has failed INFINITY times on ip-10-33-167-102
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: WARN: common_apply_stickiness: Forcing elasticip away from ip-10-33-167-102 after 1000000 failures (max=1000000)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: rsc_merge_weights: elasticip: Rolling back scores from email
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource elasticip cannot run anywhere
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource email cannot run anywhere
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Stop elasticip (ip-10-33-167-102)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Stopped)
Jul 22 15:46:12 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 22: PEngine Input stored in: /var/lib/pengine/pe-input-449.bz2
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: unpack_graph: Unpacked transition 22: 4 actions in 4 synapses
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_te_invoke: Processing graph 22 (ref=pe_calc-dc-1374507972-101) derived from /var/lib/pengine/pe-input-449.bz2
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: te_pseudo_action: Pseudo action 7 fired and confirmed
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: te_rsc_command: Initiating action 1: stop elasticip_stop_0 on ip-10-33-167-102 (local)
Jul 22 15:46:12 ip-10-33-167-102 crmd: [27046]: info: do_lrm_rsc_op: Performing key=1:22:0:f76ac39e-13f4-4516-b193-57880345216a op=elasticip_stop_0 )
Jul 22 15:46:12 ip-10-33-167-102 lrmd: [27043]: info: rsc:elasticip stop[29] (pid 32596)
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20120: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20120: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20124: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20124: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20128: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20128: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20132: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20132: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:47:05 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:stop:stderr) Unknown problem connecting to host: 'https://eu-west-1.ec2.amazonaws.com'
Jul 22 15:47:05 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:stop:stderr)
Jul 22 15:47:05 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:stop:stderr) Unable to execute HTTP request: Connect to eu-west-1.ec2.amazonaws.com:443 timed out
Jul 22 15:47:05 ip-10-33-167-102 lrmd: [27043]: info: RA output: (elasticip:stop:stderr)
Jul 22 15:47:05 ip-10-33-167-102 lrmd: [27043]: info: Managed elasticip:stop process 32596 exited with return code 0.
Jul 22 15:47:05 ip-10-33-167-102 lrmd: [27043]: info: operation stop[29] on elasticip for client 27046: pid 32596 exited with return code 0
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: process_lrm_event: LRM operation elasticip_stop_0 (call=29, rc=0, cib-update=235, confirmed=true) ok
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: match_graph_event: Action elasticip_stop_0 (1) confirmed on ip-10-33-167-102 (rc=0)
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: te_pseudo_action: Pseudo action 8 fired and confirmed
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: te_pseudo_action: Pseudo action 2 fired and confirmed
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: run_graph: ====================================================
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: notice: run_graph: Transition 22 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-449.bz2): Complete
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: te_graph_trigger: Transition 22 is now complete
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: notify_crmd: Transition 22 status: done - <null>
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 15:47:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Starting PEngine Recheck Timer
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20136: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20136: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20140: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20140: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20144: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20144: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20148: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20148: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20152: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20152: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20156: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20156: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20160: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20160: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20164: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20164: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20168: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20168: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20172: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20172: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20176: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20176: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20180: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20180: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20184: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20184: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20188: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20188: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20192: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20192: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20196: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20196: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20200: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20200: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20204: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20204: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20208: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20208: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20212: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20212: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20216: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20216: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20220: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20220: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20224: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20224: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20228: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20228: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20232: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20232: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:52:54 ip-10-33-167-102 cib: [27042]: info: cib_stats: Processed 24 operations (2500.00us average, 0% utilization) in the last 10min
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20236: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20236: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20240: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20240: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20244: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20244: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20248: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20248: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20252: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20252: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20256: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20256: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20260: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20260: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20264: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20264: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20268: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20268: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20272: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20272: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20276: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20276: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20280: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20280: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20284: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20284: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20288: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20288: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20292: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20292: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20296: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20296: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20300: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20300: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20304: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20304: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20308: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20308: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20312: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20312: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20316: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20316: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20320: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20320: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20324: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20324: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20328: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20328: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20332: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20332: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20336: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20336: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20340: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20340: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20344: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20344: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20348: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20348: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20352: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20352: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20356: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20356: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20360: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20360: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20364: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20364: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20368: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20368: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20372: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20372: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20376: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20376: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20380: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20380: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20384: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20384: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20388: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20388: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20392: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20392: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20396: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20396: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20400: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20400: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 236: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=236, ref=pe_calc-dc-1374508925-103, seq=20000, quorate=0
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_monitor_20000 on ip-10-33-167-102: not running (7)
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_last_failure_0 on ip-10-33-167-102: unknown error (1)
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Stopped
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Stopped
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: get_failcount: elasticip has failed INFINITY times on ip-10-33-167-102
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: WARN: common_apply_stickiness: Forcing elasticip away from ip-10-33-167-102 after 1000000 failures (max=1000000)
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: rsc_merge_weights: elasticip: Rolling back scores from email
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource elasticip cannot run anywhere
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource email cannot run anywhere
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Stopped)
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Stopped)
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: unpack_graph: Unpacked transition 23: 0 actions in 0 synapses
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_te_invoke: Processing graph 23 (ref=pe_calc-dc-1374508925-103) derived from /var/lib/pengine/pe-input-450.bz2
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: run_graph: ====================================================
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: notice: run_graph: Transition 23 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-450.bz2): Complete
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: te_graph_trigger: Transition 23 is now complete
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: notify_crmd: Transition 23 status: done - <null>
Jul 22 16:02:05 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 23: PEngine Input stored in: /var/lib/pengine/pe-input-450.bz2
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 16:02:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Starting PEngine Recheck Timer
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20404: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20404: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20408: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20408: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20412: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20412: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 16:02:54 ip-10-33-167-102 cib: [27042]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20416: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20416: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20420: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20420: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20424: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20424: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20428: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20428: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20432: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20432: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20436: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20436: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20440: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20440: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20444: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20444: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20448: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20448: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20452: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20452: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20456: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20456: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20460: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20460: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20464: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20464: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20468: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20468: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20472: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20472: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20476: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20476: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20480: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20480: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20484: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20484: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20488: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20488: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20492: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20492: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20496: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20496: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20500: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20500: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20504: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20504: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20508: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20508: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20512: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20512: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20516: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20516: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20520: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20520: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20524: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20524: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20528: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20528: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20532: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20532: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20536: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20536: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20540: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20540: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20544: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20544: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20548: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20548: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20552: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20552: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20556: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20556: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20560: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20560: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20564: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20564: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20568: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20568: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20572: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20572: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20576: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20576: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20580: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20580: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20584: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20584: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20588: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20588: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20592: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20592: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 16:12:54 ip-10-33-167-102 cib: [27042]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20596: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20596: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20600: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20600: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20604: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20604: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20608: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20608: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20612: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20612: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20616: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20616: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20620: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20620: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20624: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20624: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20628: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20628: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20632: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20632: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20636: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20636: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20640: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20640: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20644: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20644: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20648: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20648: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20652: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20652: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20656: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20656: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20660: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20660: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20664: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20664: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20668: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20668: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke: Query 237: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_pe_invoke_callback: Invoking the PE: query=237, ref=pe_calc-dc-1374509825-104, seq=20000, quorate=0
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Startup probes: enabled
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: unpack_domains: Unpacking domains
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: determine_online_status: Node ip-10-33-167-102 is online
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_monitor_20000 on ip-10-33-167-102: not running (7)
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: WARN: unpack_rsc_op: Processing failed op elasticip_last_failure_0 on ip-10-33-167-102: unknown error (1)
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: group_print: Resource Group: ha
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: native_print: elasticip (ocf::aws:elasticip): Stopped
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: native_print: email (ocf::heartbeat:MailTo): Stopped
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: get_failcount: elasticip has failed INFINITY times on ip-10-33-167-102
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: WARN: common_apply_stickiness: Forcing elasticip away from ip-10-33-167-102 after 1000000 failures (max=1000000)
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: rsc_merge_weights: elasticip: Rolling back scores from email
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource elasticip cannot run anywhere
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: info: native_color: Resource email cannot run anywhere
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave elasticip (Stopped)
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: notice: LogActions: Leave email (Stopped)
Jul 22 16:17:05 ip-10-33-167-102 pengine: [27045]: notice: process_pe_message: Transition 24: PEngine Input stored in: /var/lib/pengine/pe-input-450.bz2
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: unpack_graph: Unpacked transition 24: 0 actions in 0 synapses
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_te_invoke: Processing graph 24 (ref=pe_calc-dc-1374509825-104) derived from /var/lib/pengine/pe-input-450.bz2
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: run_graph: ====================================================
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: notice: run_graph: Transition 24 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-450.bz2): Complete
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: te_graph_trigger: Transition 24 is now complete
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: notify_crmd: Transition 24 status: done - <null>
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 16:17:05 ip-10-33-167-102 crmd: [27046]: info: do_state_transition: Starting PEngine Recheck Timer
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20672: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20672: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20676: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20676: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20680: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20680: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20684: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20684: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20688: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20688: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 20692: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: memb: ip-10-33-167-102 1722229002
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 20692: memb=1, new=0, lost=0
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-33-167-102 1722229002
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.33.167.102) ; members(old:1 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Raw
nodeA.log (after boot)
Jul 22 15:35:15 ip-10-34-184-243 crmd: [878]: info: do_dc_join_offer_all: join-2: Waiting on 1 outstanding join acks
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib_replace_notify: Replaced: 0.196.6 -> 0.197.1 from <null>
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - <cib admin_epoch="0" epoch="196" num_updates="6" >
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - <configuration >
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - <nodes >
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - <node id="ip-10-33-193-70" type="normal" uname="ip-10-33-193-70" __crm_diff_marker__="removed:top" />
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - </nodes>
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - </configuration>
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - </cib>
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + <cib epoch="197" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-33-167-102" update-client="crmd" cib-last-written="Mon Jul 22 14:36:35 2013" have-quorum="1" />
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_replace for section 'all' (origin=local/cibadmin/2, version=0.197.1): ok (rc=0)
Jul 22 15:37:29 ip-10-34-184-243 attrd: [876]: info: do_cib_replaced: Sending full refresh
Jul 22 15:37:29 ip-10-34-184-243 crmd: [878]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ]
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: - <cib admin_epoch="0" epoch="197" num_updates="1" />
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + <cib epoch="198" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.5" update-origin="ip-10-34-184-243" update-client="cibadmin" cib-last-written="Mon Jul 22 15:37:29 2013" have-quorum="1" >
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + <configuration >
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + <nodes >
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + <node id="ip-10-34-184-243" uname="ip-10-34-184-243" type="normal" __crm_diff_marker__="added:top" />
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + </nodes>
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + </configuration>
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib:diff: + </cib>
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/16, version=0.198.1): ok (rc=0)
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: Managed write_cib_contents process 1976 exited with return code 0.
Jul 22 15:37:29 ip-10-34-184-243 cib: [874]: info: Managed write_cib_contents process 1979 exited with return code 0.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: ERROR: crm_timer_popped: Election Timeout (I_ELECTION_DC) just popped in state S_ELECTION! (120000ms)
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: do_dc_takeover: Taking over DC status for this partition
Jul 22 15:39:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/18, version=0.198.3): ok (rc=0)
Jul 22 15:39:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/19, version=0.198.4): ok (rc=0)
Jul 22 15:39:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/21, version=0.198.5): ok (rc=0)
Jul 22 15:39:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/23, version=0.198.6): ok (rc=0)
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: do_dc_join_offer_all: join-3: Waiting on 1 outstanding join acks
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: config_query_callback: Checking for expired actions every 900000ms
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: config_query_callback: Sending expected-votes=2 to corosync
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: ais_dispatch_message: Membership 0: quorum still lost
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:39:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/26, version=0.198.7): ok (rc=0)
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: ais_dispatch_message: Membership 0: quorum still lost
Jul 22 15:39:29 ip-10-34-184-243 crmd: [878]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:39:29 ip-10-34-184-243 cib: [874]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/28, version=0.198.8): ok (rc=0)
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Jul 22 15:39:53 ip-10-34-184-243 cib: [874]: info: cib_stats: Processed 37 operations (810.00us average, 0% utilization) in the last 10min
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: ERROR: crm_timer_popped: Integration Timer (I_INTEGRATED) just popped in state S_INTEGRATION! (180000ms)
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: info: crm_timer_popped: Welcomed: 1, Integrated: 0
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: WARN: do_state_transition: Progressed to state S_FINALIZE_JOIN after C_TIMER_POPPED
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: WARN: do_state_transition: 1 cluster nodes failed to respond to the join offer.
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: info: ghash_print_node: Welcome reply not received from: ip-10-34-184-243 3
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: WARN: do_log: FSA: Input I_ELECTION_DC from do_dc_join_finalize() received in state S_FINALIZE_JOIN
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_dc_join_finalize ]
Jul 22 15:42:29 ip-10-34-184-243 crmd: [878]: info: do_dc_join_offer_all: join-4: Waiting on 1 outstanding join acks
corosync [TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
corosync [MAIN ] Corosync Cluster Engine exiting with status 0 at main.c:1858.
Jul 22 15:43:00 ip-10-34-184-243 attrd: [876]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:43:00 ip-10-34-184-243 attrd: [876]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:43:00 ip-10-34-184-243 attrd: [876]: CRIT: attrd_ais_destroy: Lost connection to OpenAIS service!
Jul 22 15:43:00 ip-10-34-184-243 attrd: [876]: notice: main: Exiting...
Jul 22 15:43:00 ip-10-34-184-243 attrd: [876]: ERROR: attrd_cib_connection_destroy: Connection to the CIB terminated...
Jul 22 15:43:00 ip-10-34-184-243 stonith-ng: [873]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:43:00 ip-10-34-184-243 stonith-ng: [873]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:43:00 ip-10-34-184-243 stonith-ng: [873]: ERROR: stonith_peer_ais_destroy: AIS connection terminated
Jul 22 15:43:00 ip-10-34-184-243 cib: [874]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:43:00 ip-10-34-184-243 cib: [874]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:43:00 ip-10-34-184-243 cib: [874]: ERROR: cib_ais_destroy: AIS connection terminated
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: info: crmd_ais_destroy: connection closed
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: info: stonith_msgready: Lost connection to the STONITH service [873].
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: CRIT: stonith_dispatch_internal: Lost connection to the STONITH service [873/callback].
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: CRIT: stonith_dispatch_internal: Lost connection to the STONITH service [873/command].
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: CRIT: tengine_stonith_connection_destroy: Fencing daemon connection failed
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: info: cib_native_msgready: Lost connection to the CIB service [874].
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: CRIT: cib_native_dispatch: Lost connection to the CIB service [874/callback].
Jul 22 15:43:00 ip-10-34-184-243 crmd: [878]: CRIT: cib_native_dispatch: Lost connection to the CIB service [874/command].
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: ERROR: crmd_cib_connection_destroy: Connection to the CIB terminated...
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: ERROR: do_log: FSA: Input I_ERROR from crmd_cib_connection_destroy() received in state S_INTEGRATION
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_state_transition: State transition S_INTEGRATION -> S_RECOVERY [ input=I_ERROR cause=C_FSA_INTERNAL origin=crmd_cib_connection_destroy ]
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: ERROR: do_recover: Action A_RECOVER (0000000001000000) not supported
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: WARN: do_election_vote: Not voting in election, we're in state S_RECOVERY
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_dc_release: DC role released
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: pe_connection_destroy: Connection to the Policy Engine released
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_te_control: Transitioner is now inactive
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: ERROR: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_state_transition: State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_recover ]
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_shutdown: Disconnecting STONITH...
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_lrm_control: Disconnected from the LRM
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: notice: terminate_ais_connection: Disconnecting from AIS
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_ha_control: Disconnected from OpenAIS
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_cib_control: Disconnecting CIB
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: ERROR: do_exit: Could not recover from internal error
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: free_mem: Dropping I_PENDING: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_election_vote ]
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: free_mem: Dropping I_RELEASE_SUCCESS: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_dc_release ]
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: free_mem: Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_stop ]
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: crm_xml_cleanup: Cleaning up memory from libxml2
Jul 22 15:43:01 ip-10-34-184-243 crmd: [878]: info: do_exit: [crmd] stopped (2)
corosync [MAIN ] Corosync Cluster Engine ('1.4.2'): started and ready to provide service.
corosync [MAIN ] Corosync built-in features: nss
corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Set r/w permissions for uid=0, gid=0 on /var/log/corosync.log
corosync [TOTEM ] The network interface is down.
corosync [pcmk ] info: process_ais_conf: Reading configure
corosync [pcmk ] info: config_find_init: Local handle: 7739444317642555395 for logging
corosync [pcmk ] info: config_find_next: Processing additional logging options...
corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_logfile
corosync [pcmk ] info: get_config_opt: Found '/var/log/corosync.log' for option: logfile
corosync [pcmk ] info: get_config_opt: Defaulting to 'on' for option: to_syslog
corosync [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
corosync [pcmk ] info: config_find_init: Local handle: 5650605097994944516 for quorum
corosync [pcmk ] info: config_find_next: No additional configuration supplied for: quorum
corosync [pcmk ] info: get_config_opt: No default for option: provider
corosync [pcmk ] info: config_find_init: Local handle: 2730409743423111173 for service
corosync [pcmk ] info: config_find_next: Processing additional service options...
corosync [pcmk ] info: get_config_opt: Found '0' for option: ver
corosync [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd
corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
corosync [pcmk ] info: pcmk_startup: CRM: Initialized
corosync [pcmk ] Logging: Initialized pcmk_startup
corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
corosync [pcmk ] info: pcmk_startup: Service: 9
corosync [pcmk ] info: pcmk_startup: Local hostname: ip-10-34-184-243
corosync [pcmk ] info: pcmk_update_nodeid: Local node id: 16777343
corosync [pcmk ] info: update_member: Creating entry for node 16777343 born on 0
corosync [pcmk ] info: update_member: 0x17cf760 Node 16777343 now known as ip-10-34-184-243 (was: (null))
corosync [pcmk ] info: update_member: Node ip-10-34-184-243 now has 1 quorum votes (was 0)
corosync [pcmk ] info: update_member: Node 16777343/ip-10-34-184-243 is now: member
corosync [pcmk ] info: spawn_child: Forked child 2096 for process stonith-ng
corosync [pcmk ] info: spawn_child: Forked child 2097 for process cib
corosync [pcmk ] info: spawn_child: Forked child 2098 for process lrmd
corosync [pcmk ] info: spawn_child: Forked child 2099 for process attrd
corosync [pcmk ] info: spawn_child: Forked child 2100 for process pengine
corosync [pcmk ] info: spawn_child: Forked child 2101 for process crmd
corosync [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.6
corosync [SERV ] Service engine loaded: corosync extended virtual synchrony service
corosync [SERV ] Service engine loaded: corosync configuration service
corosync [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
corosync [SERV ] Service engine loaded: corosync cluster config database access v1.01
corosync [SERV ] Service engine loaded: corosync profile loading service
corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
corosync [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: Invoked: /usr/lib/heartbeat/stonithd
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/root
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
Jul 22 15:43:58 ip-10-34-184-243 lrmd: [2098]: info: Signal sent to pid=875, waiting for process to exit
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: Invoked: /usr/lib/heartbeat/attrd
Jul 22 15:43:58 ip-10-34-184-243 pengine: [2100]: info: Invoked: /usr/lib/heartbeat/pengine
Jul 22 15:43:58 ip-10-34-184-243 crmd: [2101]: info: Invoked: /usr/lib/heartbeat/crmd
Jul 22 15:43:58 ip-10-34-184-243 lrmd: [875]: info: lrmd is shutting down
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:43:58 ip-10-34-184-243 pengine: [2100]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:43:58 ip-10-34-184-243 crmd: [2101]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x17d8b50 for stonith-ng/2096
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: main: Starting up
Jul 22 15:43:58 ip-10-34-184-243 crmd: [2101]: info: main: CRM Hg Version: 9971ebba4494012a93c03b40a2c58ec0eb60f50c
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: get_ais_nodeid: Server details: id=16777343 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:43:58 ip-10-34-184-243 crmd: [2101]: info: crmd_init: Starting crmd
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:43:58 ip-10-34-184-243 pengine: [2100]: WARN: main: Terminating previous PE instance
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:43:58 ip-10-34-184-243 pengine: [877]: WARN: process_pe_message: Received quit message, terminating
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 16777343
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: crm_new_peer: Node 16777343 is now known as ip-10-34-184-243
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x17dceb0 for attrd/2099
Jul 22 15:43:58 ip-10-34-184-243 stonith-ng: [2096]: info: main: Starting stonith-ng mainloop
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: get_ais_nodeid: Server details: id=16777343 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 16777343
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: crm_new_peer: Node 16777343 is now known as ip-10-34-184-243
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: main: Cluster connection active
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: info: main: Accepting attribute updates
Jul 22 15:43:58 ip-10-34-184-243 attrd: [2099]: notice: main: Starting mainloop...
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: validate_with_relaxng: Creating RNG parser context
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: startCib: CIB Initialization completed successfully
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x17e1210 for cib/2097
corosync [pcmk ] info: update_member: Node ip-10-34-184-243 now has process list: 00000000000000000000000000111312 (1118994)
corosync [pcmk ] info: pcmk_ipc: Sending membership update 0 to cib
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: get_ais_nodeid: Server details: id=16777343 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 16777343
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: crm_new_peer: Node 16777343 is now known as ip-10-34-184-243
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: cib_init: Starting cib mainloop
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: ais_dispatch_message: Membership 0: quorum still lost
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: crm_update_peer: Node ip-10-34-184-243: id=16777343 state=member (new) addr=(null) votes=1 (new) born=0 seen=0 proc=00000000000000000000000000111312 (new)
Jul 22 15:43:58 ip-10-34-184-243 cib: [2097]: info: Managed write_cib_contents process 2105 exited with return code 0.
Jul 22 15:43:59 ip-10-34-184-243 lrmd: [2098]: info: enabling coredumps
Jul 22 15:43:59 ip-10-34-184-243 lrmd: [2098]: info: Started.
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: do_cib_control: CIB connection established
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x17e5a10 for crmd/2101
corosync [pcmk ] info: pcmk_ipc: Sending membership update 0 to crmd
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: get_ais_nodeid: Server details: id=16777343 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 16777343
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: crm_new_peer: Node 16777343 is now known as ip-10-34-184-243
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: ais_status_callback: status: ip-10-34-184-243 is now unknown
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: do_ha_control: Connected to the cluster
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: do_started: Delaying start, no membership data (0000000000100000)
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: crmd_init: Starting crmd's mainloop
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: ais_dispatch_message: Membership 0: quorum still lost
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: notice: crmd_peer_update: Status update: Client ip-10-34-184-243/crmd now has status [online] (DC=<null>)
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: ais_status_callback: status: ip-10-34-184-243 is now member (was unknown)
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: crm_update_peer: Node ip-10-34-184-243: id=16777343 state=member (new) addr=(null) votes=1 (new) born=0 seen=0 proc=00000000000000000000000000111312 (new)
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: do_started: Delaying start, Config not read (0000000000000040)
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: config_query_callback: Checking for expired actions every 900000ms
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: config_query_callback: Sending expected-votes=2 to corosync
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: do_started: The local CRM is operational
Jul 22 15:43:59 ip-10-34-184-243 crmd: [2101]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Jul 22 15:44:00 ip-10-34-184-243 pengine: [2100]: info: main: Starting pengine
Jul 22 15:44:00 ip-10-34-184-243 crmd: [2101]: info: ais_dispatch_message: Membership 0: quorum still lost
Jul 22 15:44:00 ip-10-34-184-243 crmd: [2101]: info: te_connect_stonith: Attempting connection to fencing daemon...
Jul 22 15:44:01 ip-10-34-184-243 crmd: [2101]: info: te_connect_stonith: Connected
Jul 22 15:44:03 ip-10-34-184-243 attrd: [2099]: info: cib_connect: Connected to the CIB after 1 signon attempts
Jul 22 15:44:03 ip-10-34-184-243 attrd: [2099]: info: cib_connect: Sending full refresh
corosync [MAIN ] Corosync Cluster Engine exiting with status 0 at main.c:1858.
Jul 22 15:44:09 ip-10-34-184-243 stonith-ng: [2096]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:44:09 ip-10-34-184-243 stonith-ng: [2096]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:44:09 ip-10-34-184-243 stonith-ng: [2096]: ERROR: stonith_peer_ais_destroy: AIS connection terminated
Jul 22 15:44:09 ip-10-34-184-243 attrd: [2099]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:44:09 ip-10-34-184-243 attrd: [2099]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:44:09 ip-10-34-184-243 attrd: [2099]: CRIT: attrd_ais_destroy: Lost connection to OpenAIS service!
Jul 22 15:44:09 ip-10-34-184-243 attrd: [2099]: notice: main: Exiting...
Jul 22 15:44:09 ip-10-34-184-243 attrd: [2099]: ERROR: attrd_cib_connection_destroy: Connection to the CIB terminated...
Jul 22 15:44:09 ip-10-34-184-243 cib: [2097]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:44:09 ip-10-34-184-243 cib: [2097]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:44:09 ip-10-34-184-243 cib: [2097]: ERROR: cib_ais_destroy: AIS connection terminated
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Resource temporarily unavailable (11)
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: ERROR: ais_dispatch: AIS connection failed
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: crmd_ais_destroy: connection closed
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: stonith_msgready: Lost connection to the STONITH service [2096].
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: CRIT: stonith_dispatch_internal: Lost connection to the STONITH service [2096/callback].
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: CRIT: stonith_dispatch_internal: Lost connection to the STONITH service [2096/command].
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: CRIT: tengine_stonith_connection_destroy: Fencing daemon connection failed
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: cib_native_msgready: Lost connection to the CIB service [2097].
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: CRIT: cib_native_dispatch: Lost connection to the CIB service [2097/callback].
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: CRIT: cib_native_dispatch: Lost connection to the CIB service [2097/command].
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: ERROR: crmd_cib_connection_destroy: Connection to the CIB terminated...
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: ERROR: do_log: FSA: Input I_ERROR from crmd_cib_connection_destroy() received in state S_PENDING
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_state_transition: State transition S_PENDING -> S_RECOVERY [ input=I_ERROR cause=C_FSA_INTERNAL origin=crmd_cib_connection_destroy ]
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: ERROR: do_recover: Action A_RECOVER (0000000001000000) not supported
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: ERROR: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_state_transition: State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_recover ]
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_shutdown: Disconnecting STONITH...
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_lrm_control: Disconnected from the LRM
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: notice: terminate_ais_connection: Disconnecting from AIS
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_ha_control: Disconnected from OpenAIS
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_cib_control: Disconnecting CIB
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: ERROR: do_exit: Could not recover from internal error
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: free_mem: Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_stop ]
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: crm_xml_cleanup: Cleaning up memory from libxml2
Jul 22 15:44:09 ip-10-34-184-243 crmd: [2101]: info: do_exit: [crmd] stopped (2)
corosync [MAIN ] Corosync Cluster Engine ('1.4.2'): started and ready to provide service.
corosync [MAIN ] Corosync built-in features: nss
corosync [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
corosync [TOTEM ] Initializing transport (UDP/IP Unicast).
corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Set r/w permissions for uid=0, gid=0 on /var/log/corosync.log
corosync [TOTEM ] The network interface [10.34.184.243] is now up.
corosync [pcmk ] info: process_ais_conf: Reading configure
corosync [pcmk ] info: config_find_init: Local handle: 7739444317642555395 for logging
corosync [pcmk ] info: config_find_next: Processing additional logging options...
corosync [pcmk ] info: get_config_opt: Found 'yes' for option: to_logfile
corosync [pcmk ] info: get_config_opt: Found '/var/log/corosync.log' for option: logfile
corosync [pcmk ] info: get_config_opt: Defaulting to 'on' for option: to_syslog
corosync [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
corosync [pcmk ] info: config_find_init: Local handle: 5650605097994944516 for quorum
corosync [pcmk ] info: config_find_next: No additional configuration supplied for: quorum
corosync [pcmk ] info: get_config_opt: No default for option: provider
corosync [pcmk ] info: config_find_init: Local handle: 2730409743423111173 for service
corosync [pcmk ] info: config_find_next: Processing additional service options...
corosync [pcmk ] info: get_config_opt: Found '0' for option: ver
corosync [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd
corosync [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
corosync [pcmk ] info: pcmk_startup: CRM: Initialized
corosync [pcmk ] Logging: Initialized pcmk_startup
corosync [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
corosync [pcmk ] info: pcmk_startup: Service: 9
corosync [pcmk ] info: pcmk_startup: Local hostname: ip-10-34-184-243
corosync [pcmk ] info: pcmk_update_nodeid: Local node id: 4088930826
corosync [pcmk ] info: update_member: Creating entry for node 4088930826 born on 0
corosync [pcmk ] info: update_member: 0x1878730 Node 4088930826 now known as ip-10-34-184-243 (was: (null))
corosync [pcmk ] info: update_member: Node ip-10-34-184-243 now has 1 quorum votes (was 0)
corosync [pcmk ] info: update_member: Node 4088930826/ip-10-34-184-243 is now: member
corosync [pcmk ] info: spawn_child: Forked child 2141 for process stonith-ng
corosync [pcmk ] info: spawn_child: Forked child 2142 for process cib
corosync [pcmk ] info: spawn_child: Forked child 2143 for process lrmd
corosync [pcmk ] info: spawn_child: Forked child 2144 for process attrd
corosync [pcmk ] info: spawn_child: Forked child 2145 for process pengine
corosync [pcmk ] info: spawn_child: Forked child 2146 for process crmd
corosync [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.6
corosync [SERV ] Service engine loaded: corosync extended virtual synchrony service
corosync [SERV ] Service engine loaded: corosync configuration service
corosync [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
corosync [SERV ] Service engine loaded: corosync cluster config database access v1.01
corosync [SERV ] Service engine loaded: corosync profile loading service
corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
corosync [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4: memb=0, new=0, lost=0
corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4: memb=1, new=1, lost=0
corosync [pcmk ] info: pcmk_peer_update: NEW: ip-10-34-184-243 4088930826
corosync [pcmk ] info: pcmk_peer_update: MEMB: ip-10-34-184-243 4088930826
corosync [pcmk ] info: update_member: Node ip-10-34-184-243 now has process list: 00000000000000000000000000111312 (1118994)
corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
corosync [CPG ] chosen downlist: sender r(0) ip(10.34.184.243) ; members(old:0 left:0)
corosync [MAIN ] Completed service synchronization, ready to provide service.
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: Invoked: /usr/lib/heartbeat/stonithd
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/root
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
Jul 22 15:44:29 ip-10-34-184-243 lrmd: [2143]: info: Signal sent to pid=2098, waiting for process to exit
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: Invoked: /usr/lib/heartbeat/attrd
Jul 22 15:44:29 ip-10-34-184-243 pengine: [2145]: info: Invoked: /usr/lib/heartbeat/pengine
Jul 22 15:44:29 ip-10-34-184-243 crmd: [2146]: info: Invoked: /usr/lib/heartbeat/crmd
Jul 22 15:44:29 ip-10-34-184-243 lrmd: [2098]: info: lrmd is shutting down
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:44:29 ip-10-34-184-243 pengine: [2145]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:44:29 ip-10-34-184-243 crmd: [2146]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x1881f30 for stonith-ng/2141
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: main: Starting up
Jul 22 15:44:29 ip-10-34-184-243 crmd: [2146]: info: main: CRM Hg Version: 9971ebba4494012a93c03b40a2c58ec0eb60f50c
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: get_ais_nodeid: Server details: id=4088930826 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:44:29 ip-10-34-184-243 crmd: [2146]: info: crmd_init: Starting crmd
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:44:29 ip-10-34-184-243 pengine: [2145]: WARN: main: Terminating previous PE instance
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:44:29 ip-10-34-184-243 pengine: [2100]: WARN: process_pe_message: Received quit message, terminating
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 4088930826
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: crm_new_peer: Node 4088930826 is now known as ip-10-34-184-243
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x1886720 for attrd/2144
Jul 22 15:44:29 ip-10-34-184-243 stonith-ng: [2141]: info: main: Starting stonith-ng mainloop
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: get_ais_nodeid: Server details: id=4088930826 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 4088930826
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: crm_new_peer: Node 4088930826 is now known as ip-10-34-184-243
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: main: Cluster connection active
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: info: main: Accepting attribute updates
Jul 22 15:44:29 ip-10-34-184-243 attrd: [2144]: notice: main: Starting mainloop...
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: validate_with_relaxng: Creating RNG parser context
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: startCib: CIB Initialization completed successfully
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x188af20 for cib/2142
corosync [pcmk ] info: pcmk_ipc: Sending membership update 4 to cib
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: get_ais_nodeid: Server details: id=4088930826 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 4088930826
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: crm_new_peer: Node 4088930826 is now known as ip-10-34-184-243
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: cib_init: Starting cib mainloop
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: ais_dispatch_message: Membership 4: quorum still lost
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: crm_update_peer: Node ip-10-34-184-243: id=4088930826 state=member (new) addr=r(0) ip(10.34.184.243) (new) votes=1 (new) born=0 seen=4 proc=00000000000000000000000000111312 (new)
Jul 22 15:44:29 ip-10-34-184-243 cib: [2142]: info: Managed write_cib_contents process 2150 exited with return code 0.
Jul 22 15:44:30 ip-10-34-184-243 lrmd: [2143]: info: enabling coredumps
Jul 22 15:44:30 ip-10-34-184-243 lrmd: [2143]: info: Started.
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: do_cib_control: CIB connection established
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: get_cluster_type: Cluster type is: 'openais'
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: init_ais_connection_classic: Creating connection to our Corosync plugin
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: init_ais_connection_classic: AIS connection established
corosync [pcmk ] info: pcmk_ipc: Recorded connection 0x188f740 for crmd/2146
corosync [pcmk ] info: pcmk_ipc: Sending membership update 4 to crmd
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: get_ais_nodeid: Server details: id=4088930826 uname=ip-10-34-184-243 cname=pcmk
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: crm_new_peer: Node ip-10-34-184-243 now has id: 4088930826
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: crm_new_peer: Node 4088930826 is now known as ip-10-34-184-243
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: ais_status_callback: status: ip-10-34-184-243 is now unknown
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: do_ha_control: Connected to the cluster
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: do_started: Delaying start, no membership data (0000000000100000)
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: crmd_init: Starting crmd's mainloop
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: ais_dispatch_message: Membership 4: quorum still lost
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: notice: crmd_peer_update: Status update: Client ip-10-34-184-243/crmd now has status [online] (DC=<null>)
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: ais_status_callback: status: ip-10-34-184-243 is now member (was unknown)
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: crm_update_peer: Node ip-10-34-184-243: id=4088930826 state=member (new) addr=r(0) ip(10.34.184.243) (new) votes=1 (new) born=0 seen=4 proc=00000000000000000000000000111312 (new)
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: do_started: Delaying start, Config not read (0000000000000040)
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: config_query_callback: Checking for expired actions every 900000ms
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: config_query_callback: Sending expected-votes=2 to corosync
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: do_started: The local CRM is operational
Jul 22 15:44:30 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Jul 22 15:44:31 ip-10-34-184-243 pengine: [2145]: info: main: Starting pengine
Jul 22 15:44:31 ip-10-34-184-243 crmd: [2146]: info: ais_dispatch_message: Membership 4: quorum still lost
Jul 22 15:44:31 ip-10-34-184-243 crmd: [2146]: info: te_connect_stonith: Attempting connection to fencing daemon...
Jul 22 15:44:32 ip-10-34-184-243 crmd: [2146]: info: te_connect_stonith: Connected
Jul 22 15:44:34 ip-10-34-184-243 attrd: [2144]: info: cib_connect: Connected to the CIB after 1 signon attempts
Jul 22 15:44:34 ip-10-34-184-243 attrd: [2144]: info: cib_connect: Sending full refresh
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_te_control: Registering TE UUID: 23cf5602-fed9-448e-bed1-8c29b280327c
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: set_graph_functions: Setting custom graph functions
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_dc_takeover: Taking over DC status for this partition
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_readwrite: We are now in R/W mode
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/5, version=0.198.1): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/6, version=0.198.2): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/8, version=0.198.3): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/10, version=0.198.4): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: join_make_offer: Making join offers based on membership 4
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: config_query_callback: Checking for expired actions every 900000ms
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: config_query_callback: Sending expected-votes=2 to corosync
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: ais_dispatch_message: Membership 4: quorum still lost
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/13, version=0.198.5): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: ais_dispatch_message: Membership 4: quorum still lost
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: crmd_ais_dispatch: Setting expected votes to 2
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/15, version=0.198.6): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: update_dc: Set DC to ip-10-34-184-243 (3.0.5)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: All 1 cluster nodes responded to the join offer.
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_dc_join_finalize: join-1: Syncing the CIB from ip-10-34-184-243 to the rest of the cluster
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/17, version=0.198.6): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/18, version=0.198.7): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: update_attrd: Connecting to attrd...
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ip-10-34-184-243']/transient_attributes (origin=local/crmd/19, version=0.198.8): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 attrd: [2144]: info: find_hash_entry: Creating hash entry for terminate
Jul 22 15:44:51 ip-10-34-184-243 attrd: [2144]: info: find_hash_entry: Creating hash entry for shutdown
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ip-10-34-184-243']/transient_attributes": ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_dc_join_ack: join-1: Updating node state to member for ip-10-34-184-243
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='ip-10-34-184-243']/lrm (origin=local/crmd/20, version=0.198.9): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: erase_xpath_callback: Deletion of "//node_state[@uname='ip-10-34-184-243']/lrm": ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/22, version=0.198.11): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date
Jul 22 15:44:51 ip-10-34-184-243 attrd: [2144]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: crm_update_quorum: Updating quorum status to false (call=24)
Jul 22 15:44:51 ip-10-34-184-243 cib: [2142]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/24, version=0.198.13): ok (rc=0)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: abort_transition_graph: do_te_invoke:167 - Triggered transition abort (complete=1) : Peer Cancelled
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_pe_invoke: Query 25: Requesting the current CIB: S_POLICY_ENGINE
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_pe_invoke_callback: Invoking the PE: query=25, ref=pe_calc-dc-1374507891-7, seq=4, quorate=0
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: info: unpack_config: Startup probes: enabled
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: info: unpack_domains: Unpacking domains
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: info: determine_online_status: Node ip-10-34-184-243 is online
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: info: group_print: Resource Group: ha
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: info: native_print: elasticip (ocf::aws:elasticip): Stopped
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: info: native_print: email (ocf::heartbeat:MailTo): Stopped
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: notice: RecurringOp: Start recurring monitor (20s) for elasticip on ip-10-34-184-243
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: notice: RecurringOp: Start recurring monitor (10s) for email on ip-10-34-184-243
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: notice: LogActions: Start elasticip (ip-10-34-184-243)
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: notice: LogActions: Start email (ip-10-34-184-243)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: unpack_graph: Unpacked transition 0: 10 actions in 10 synapses
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1374507891-7) derived from /var/lib/pengine/pe-input-246.bz2
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: te_pseudo_action: Pseudo action 10 fired and confirmed
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: te_rsc_command: Initiating action 4: monitor elasticip_monitor_0 on ip-10-34-184-243 (local)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_lrm_rsc_op: Performing key=4:0:7:23cf5602-fed9-448e-bed1-8c29b280327c op=elasticip_monitor_0 )
Jul 22 15:44:51 ip-10-34-184-243 lrmd: [2143]: info: rsc:elasticip probe[2] (pid 2153)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: te_rsc_command: Initiating action 5: monitor email_monitor_0 on ip-10-34-184-243 (local)
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: do_lrm_rsc_op: Performing key=5:0:7:23cf5602-fed9-448e-bed1-8c29b280327c op=email_monitor_0 )
Jul 22 15:44:51 ip-10-34-184-243 lrmd: [2143]: info: rsc:email probe[3] (pid 2154)
Jul 22 15:44:51 ip-10-34-184-243 pengine: [2145]: notice: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/pengine/pe-input-246.bz2
Jul 22 15:44:51 ip-10-34-184-243 lrmd: [2143]: WARN: Managed email:monitor process 2154 exited with return code 7.
Jul 22 15:44:51 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[3] on email for client 2146: pid 2154 exited with return code 7
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: process_lrm_event: LRM operation email_monitor_0 (call=3, rc=7, cib-update=26, confirmed=true) not running
Jul 22 15:44:51 ip-10-34-184-243 crmd: [2146]: info: match_graph_event: Action email_monitor_0 (5) confirmed on ip-10-34-184-243 (rc=0)
Jul 22 15:44:57 ip-10-34-184-243 lrmd: [2143]: WARN: Managed elasticip:monitor process 2153 exited with return code 7.
Jul 22 15:44:57 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[2] on elasticip for client 2146: pid 2153 exited with return code 7
Jul 22 15:44:57 ip-10-34-184-243 crmd: [2146]: info: process_lrm_event: LRM operation elasticip_monitor_0 (call=2, rc=7, cib-update=27, confirmed=true) not running
Jul 22 15:44:57 ip-10-34-184-243 crmd: [2146]: info: match_graph_event: Action elasticip_monitor_0 (4) confirmed on ip-10-34-184-243 (rc=0)
Jul 22 15:44:57 ip-10-34-184-243 crmd: [2146]: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on ip-10-34-184-243 (local) - no waiting
Jul 22 15:44:57 ip-10-34-184-243 crmd: [2146]: info: te_pseudo_action: Pseudo action 2 fired and confirmed
Jul 22 15:44:57 ip-10-34-184-243 crmd: [2146]: info: te_rsc_command: Initiating action 6: start elasticip_start_0 on ip-10-34-184-243 (local)
Jul 22 15:44:57 ip-10-34-184-243 attrd: [2144]: info: find_hash_entry: Creating hash entry for probe_complete
Jul 22 15:44:57 ip-10-34-184-243 crmd: [2146]: info: do_lrm_rsc_op: Performing key=6:0:0:23cf5602-fed9-448e-bed1-8c29b280327c op=elasticip_start_0 )
Jul 22 15:44:57 ip-10-34-184-243 attrd: [2144]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Jul 22 15:44:57 ip-10-34-184-243 lrmd: [2143]: info: rsc:elasticip start[4] (pid 2190)
Jul 22 15:44:57 ip-10-34-184-243 attrd: [2144]: notice: attrd_perform_update: Sent update 4: probe_complete=true
Jul 22 15:45:18 ip-10-34-184-243 lrmd: [2143]: info: Managed elasticip:start process 2190 exited with return code 0.
Jul 22 15:45:18 ip-10-34-184-243 lrmd: [2143]: info: operation start[4] on elasticip for client 2146: pid 2190 exited with return code 0
Jul 22 15:45:18 ip-10-34-184-243 crmd: [2146]: info: process_lrm_event: LRM operation elasticip_start_0 (call=4, rc=0, cib-update=28, confirmed=true) ok
Jul 22 15:45:18 ip-10-34-184-243 crmd: [2146]: info: match_graph_event: Action elasticip_start_0 (6) confirmed on ip-10-34-184-243 (rc=0)
Jul 22 15:45:18 ip-10-34-184-243 crmd: [2146]: info: te_rsc_command: Initiating action 7: monitor elasticip_monitor_20000 on ip-10-34-184-243 (local)
Jul 22 15:45:18 ip-10-34-184-243 crmd: [2146]: info: do_lrm_rsc_op: Performing key=7:0:0:23cf5602-fed9-448e-bed1-8c29b280327c op=elasticip_monitor_20000 )
Jul 22 15:45:18 ip-10-34-184-243 lrmd: [2143]: info: rsc:elasticip monitor[5] (pid 2286)
Jul 22 15:45:18 ip-10-34-184-243 crmd: [2146]: info: te_rsc_command: Initiating action 8: start email_start_0 on ip-10-34-184-243 (local)
Jul 22 15:45:18 ip-10-34-184-243 crmd: [2146]: info: do_lrm_rsc_op: Performing key=8:0:0:23cf5602-fed9-448e-bed1-8c29b280327c op=email_start_0 )
Jul 22 15:45:18 ip-10-34-184-243 lrmd: [2143]: info: rsc:email start[6] (pid 2287)
Jul 22 15:45:19 ip-10-34-184-243 lrmd: [2143]: info: Managed email:start process 2287 exited with return code 0.
Jul 22 15:45:19 ip-10-34-184-243 lrmd: [2143]: info: operation start[6] on email for client 2146: pid 2287 exited with return code 0
Jul 22 15:45:19 ip-10-34-184-243 crmd: [2146]: info: process_lrm_event: LRM operation email_start_0 (call=6, rc=0, cib-update=29, confirmed=true) ok
Jul 22 15:45:19 ip-10-34-184-243 crmd: [2146]: info: match_graph_event: Action email_start_0 (8) confirmed on ip-10-34-184-243 (rc=0)
Jul 22 15:45:19 ip-10-34-184-243 crmd: [2146]: info: te_pseudo_action: Pseudo action 11 fired and confirmed
Jul 22 15:45:19 ip-10-34-184-243 crmd: [2146]: info: te_rsc_command: Initiating action 9: monitor email_monitor_10000 on ip-10-34-184-243 (local)
Jul 22 15:45:19 ip-10-34-184-243 crmd: [2146]: info: do_lrm_rsc_op: Performing key=9:0:0:23cf5602-fed9-448e-bed1-8c29b280327c op=email_monitor_10000 )
Jul 22 15:45:19 ip-10-34-184-243 lrmd: [2143]: info: rsc:email monitor[7] (pid 2329)
Jul 22 15:45:20 ip-10-34-184-243 lrmd: [2143]: info: Managed email:monitor process 2329 exited with return code 0.
Jul 22 15:45:20 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[7] on email for client 2146: pid 2329 exited with return code 0
Jul 22 15:45:20 ip-10-34-184-243 crmd: [2146]: info: process_lrm_event: LRM operation email_monitor_10000 (call=7, rc=0, cib-update=30, confirmed=false) ok
Jul 22 15:45:20 ip-10-34-184-243 crmd: [2146]: info: match_graph_event: Action email_monitor_10000 (9) confirmed on ip-10-34-184-243 (rc=0)
Jul 22 15:45:25 ip-10-34-184-243 lrmd: [2143]: info: Managed elasticip:monitor process 2286 exited with return code 0.
Jul 22 15:45:25 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[5] on elasticip for client 2146: pid 2286 exited with return code 0
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: info: process_lrm_event: LRM operation elasticip_monitor_20000 (call=5, rc=0, cib-update=31, confirmed=false) ok
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: info: match_graph_event: Action elasticip_monitor_20000 (7) confirmed on ip-10-34-184-243 (rc=0)
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: info: run_graph: ====================================================
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: notice: run_graph: Transition 0 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-246.bz2): Complete
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: info: te_graph_trigger: Transition 0 is now complete
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: info: notify_crmd: Transition 0 status: done - <null>
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Jul 22 15:45:25 ip-10-34-184-243 crmd: [2146]: info: do_state_transition: Starting PEngine Recheck Timer
Jul 22 15:45:30 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[7] on email for client 2146: pid 2347 exited with return code 0
Jul 22 15:45:40 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[7] on email for client 2146: pid 2351 exited with return code 0
Jul 22 15:45:50 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[7] on email for client 2146: pid 2377 exited with return code 0
Jul 22 15:45:52 ip-10-34-184-243 lrmd: [2143]: info: operation monitor[5] on elasticip for client 2146: pid 2355 exited with return code 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment