Skip to content

Instantly share code, notes, and snippets.

@aboudreault

aboudreault/gist:8ed8ad8f4c7ba4f35a82

Last active June 8, 2018 01:11
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save aboudreault/8ed8ad8f4c7ba4f35a82 to your computer and use it in GitHub Desktop.
Save aboudreault/8ed8ad8f4c7ba4f35a82 to your computer and use it in GitHub Desktop.
Download ZIP
cassandra node balancing test
Raw
gistfile1.md

Procedures

  • ccm stop && ccm remove
  • ccm create -n 2 -v binary:2.1.3 demo ## or ccm create -n 2 --install-dir=path/to/cassandra-2.1/branch demo
  • ccm start
  • ccm node1 stress -- write n=1000000 -schema replication(factor=2) -rate threads=50
  • ccm node1 nodetool status
  • ccm add -i 127.0.0.3 -j 7400 node3 # no auto-boostrap
  • ccm node3 start
  • ccm node1 nodetool status
  • ccm node3 repair
  • ccm node3 nodetool status
  • ccm node1 nodetool cleanup
  • ccm node2 nodetool cleanup
  • ccm node3 nodetool cleanup
  • ccm node1 nodetool status
  • ccm node1 repair
  • ccm node1 stress -- read n=1000000 ## CRASH Data returned was not validated ?!?

Results branch cassandra-2.1

After stress

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  276.75 MB  1       ?       c93ecb01-0764-4ffc-a393-b42944f23bc5  rack1
UN  127.0.0.2  276.76 MB  1       ?       f5c17b2c-c146-427f-88ca-c19c476df899  rack1

After ccm add

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  276.75 MB  1       ?       c93ecb01-0764-4ffc-a393-b42944f23bc5  rack1
UN  127.0.0.2  276.73 MB  1       ?       f5c17b2c-c146-427f-88ca-c19c476df899  rack1
UN  127.0.0.3  83.7 KB    1       ?       1f6984c1-af27-41aa-8923-d50cb6f3c1b5  rack1

After new node repair

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  292.36 MB  1       ?       c93ecb01-0764-4ffc-a393-b42944f23bc5  rack1
UN  127.0.0.2  292.36 MB  1       ?       f5c17b2c-c146-427f-88ca-c19c476df899  rack1
UN  127.0.0.3  71.8 MB    1       ?       1f6984c1-af27-41aa-8923-d50cb6f3c1b5  rack1

After nodes cleanup

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  91.76 KB   1       ?       c93ecb01-0764-4ffc-a393-b42944f23bc5  rack1
UN  127.0.0.2  292.36 MB  1       ?       f5c17b2c-c146-427f-88ca-c19c476df899  rack1
UN  127.0.0.3  71.8 MB    1       ?       1f6984c1-af27-41aa-8923-d50cb6f3c1b5  rack1

After node1 repair

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  74.59 MB   1       ?       c93ecb01-0764-4ffc-a393-b42944f23bc5  rack1
UN  127.0.0.2  146.87 MB  1       ?       f5c17b2c-c146-427f-88ca-c19c476df899  rack1
UN  127.0.0.3  71.8 MB    1       ?       1f6984c1-af27-41aa-8923-d50cb6f3c1b5  rack1

Stress output

Failed to connect over JMX; not collecting these stats
java.io.IOException: Operation x0 on key(s) [3938324b4c4e4d334e30]: Data returned was not validated

        at org.apache.cassandra.stress.Operation.error(Operation.java:211)
        at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:183)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:107)
        at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:259)
        at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:307)
java.io.IOException: Operation x0 on key(s) [34313239335034344b30]: Data returned was not validated

Results cassandra 2.1.3

After Stress

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  281.74 MB  1       ?       56226e2a-9098-4176-b913-4788401071fe  rack1
UN  127.0.0.2  277.91 MB  1       ?       cf9aa429-a580-442e-b11e-f00740a0791c  rack1

After ccm add

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  281.74 MB  1       ?       56226e2a-9098-4176-b913-4788401071fe  rack1
UN  127.0.0.2  277.91 MB  1       ?       cf9aa429-a580-442e-b11e-f00740a0791c  rack1
UN  127.0.0.3  46.24 KB   1       ?       27b35f2a-5cdc-4f6e-aaa8-49db35d49f61  rack1

After new node repair

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  281.74 MB  1       ?       56226e2a-9098-4176-b913-4788401071fe  rack1
UN  127.0.0.2  277.91 MB  1       ?       cf9aa429-a580-442e-b11e-f00740a0791c  rack1
UN  127.0.0.3  229.97 MB  1       ?       27b35f2a-5cdc-4f6e-aaa8-49db35d49f61  rack1

After nodes cleanup

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  208.92 MB  1       ?       56226e2a-9098-4176-b913-4788401071fe  rack1
UN  127.0.0.2  146.86 MB  1       ?       cf9aa429-a580-442e-b11e-f00740a0791c  rack1
UN  127.0.0.3  229.97 MB  1       ?       27b35f2a-5cdc-4f6e-aaa8-49db35d49f61  rack1

After node1 repair

Datacenter datacenter1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.0.1  208.92 MB  1       ?       56226e2a-9098-4176-b913-4788401071fe  rack1
UN  127.0.0.2  146.86 MB  1       ?       cf9aa429-a580-442e-b11e-f00740a0791c  rack1
UN  127.0.0.3  229.97 MB  1       ?       27b35f2a-5cdc-4f6e-aaa8-49db35d49f61  rack1

Stress output: success

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment