Skip to content

Instantly share code, notes, and snippets.

/Initial setup Secret

Created January 8, 2013 12:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save anonymous/1e74ff02c2d4f622ce8f to your computer and use it in GitHub Desktop.
Save anonymous/1e74ff02c2d4f622ce8f to your computer and use it in GitHub Desktop.
Download ZIP
Apache Cassandra 1.2.0 - Nodes can't rejoin after stopping when using GossipingPropertyFileSnitch Creating a keyspace before killing a node doesn't help. Setting initial token doesn't help either (0..2^64). Using the RandomPartitioner didn't help either. This procedure works when using SimpleSnitch or PropertyFileSnitch instead of GossipingPrope…
Raw
Initial setup
Spin up 3 ec2 instances (CentOS 6.3; java 1.7.0_05)
# 10.32.146.208
cd ~
rm -rf apache-cassandra-* /var/lib/cassandra /var/log/cassandra
wget http://mirror.symnds.com/software/Apache/cassandra/1.2.0/apache-cassandra-1.2.0-bin.tar.gz
tar -xf apache-cassandra-1.2.0-bin.tar.gz
cd apache-cassandra-1.2.0
mkdir /var/log/cassandra /var/lib/cassandra
sed -i 's/seeds: "127.0.0.1"/seeds: "10.32.146.208,10.114.18.51,10.203.86.83"/g' conf/cassandra.yaml
sed -i 's/listen_address: localhost/listen_address: 10.32.146.208/g' conf/cassandra.yaml
sed -i 's/rpc_address: localhost/rpc_address: 0.0.0.0/g' conf/cassandra.yaml
sed -i 's/endpoint_snitch: SimpleSnitch/endpoint_snitch: GossipingPropertyFileSnitch/g' conf/cassandra.yaml
./bin/cassandra -f
# 10.114.18.51
cd ~
rm -rf apache-cassandra-* /var/lib/cassandra /var/log/cassandra
wget http://mirror.symnds.com/software/Apache/cassandra/1.2.0/apache-cassandra-1.2.0-bin.tar.gz
tar -xf apache-cassandra-1.2.0-bin.tar.gz
cd apache-cassandra-1.2.0
mkdir /var/log/cassandra /var/lib/cassandra
sed -i 's/seeds: "127.0.0.1"/seeds: "10.32.146.208,10.114.18.51,10.203.86.83"/g' conf/cassandra.yaml
sed -i 's/listen_address: localhost/listen_address: 10.114.18.51/g' conf/cassandra.yaml
sed -i 's/rpc_address: localhost/rpc_address: 0.0.0.0/g' conf/cassandra.yaml
sed -i 's/endpoint_snitch: SimpleSnitch/endpoint_snitch: GossipingPropertyFileSnitch/g' conf/cassandra.yaml
./bin/cassandra -f
# 10.203.86.83
cd ~
rm -rf apache-cassandra-* /var/lib/cassandra /var/log/cassandra
wget http://mirror.symnds.com/software/Apache/cassandra/1.2.0/apache-cassandra-1.2.0-bin.tar.gz
tar -xf apache-cassandra-1.2.0-bin.tar.gz
cd apache-cassandra-1.2.0
mkdir /var/log/cassandra /var/lib/cassandra
sed -i 's/seeds: "127.0.0.1"/seeds: "10.32.146.208,10.114.18.51,10.203.86.83"/g' conf/cassandra.yaml
sed -i 's/listen_address: localhost/listen_address: 10.203.86.83/g' conf/cassandra.yaml
sed -i 's/rpc_address: localhost/rpc_address: 0.0.0.0/g' conf/cassandra.yaml
sed -i 's/endpoint_snitch: SimpleSnitch/endpoint_snitch: GossipingPropertyFileSnitch/g' conf/cassandra.yaml
./bin/cassandra -f
Confirm that the ring is intact:
[root@ip-10-32-146-208 apache-cassandra-1.2.0]# ./bin/nodetool ring
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: DC1
==========
Address Rack Status State Load Owns Token
5225213360801090766
10.114.18.51 RAC1 Up Normal 15.52 KB 37.83% -6242616909889239630
10.203.86.83 RAC1 Up Normal 6.19 KB 44.62% 1989247850266406247
10.32.146.208 RAC1 Up Normal 24.87 KB 17.54% 5225213360801090766
Raw
Run 1 + Run 2 output
[root@ip-10-32-146-208 apache-cassandra-1.2.0]# ./bin/cassandra -f
xss = -ea -javaagent:./bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms827M -Xmx827M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
INFO 05:42:41,606 Logging initialized
INFO 05:42:41,625 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_05
INFO 05:42:41,626 Heap size: 856686592/857735168
INFO 05:42:41,626 Classpath: ./bin/../conf:./bin/../build/classes/main:./bin/../build/classes/thrift:./bin/../lib/antlr-3.2.jar:./bin/../lib/apache-cassandra-1.2.0.jar:./bin/../lib/apache-cassandra-clientutil-1.2.0.jar:./bin/../lib/apache-cassandra-thrift-1.2.0.jar:./bin/../lib/avro-1.4.0-fixes.jar:./bin/../lib/avro-1.4.0-sources-fixes.jar:./bin/../lib/commons-cli-1.1.jar:./bin/../lib/commons-codec-1.2.jar:./bin/../lib/commons-lang-2.6.jar:./bin/../lib/compress-lzf-0.8.4.jar:./bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:./bin/../lib/guava-13.0.1.jar:./bin/../lib/high-scale-lib-1.1.2.jar:./bin/../lib/jackson-core-asl-1.9.2.jar:./bin/../lib/jackson-mapper-asl-1.9.2.jar:./bin/../lib/jamm-0.2.5.jar:./bin/../lib/jline-1.0.jar:./bin/../lib/json-simple-1.1.jar:./bin/../lib/libthrift-0.7.0.jar:./bin/../lib/log4j-1.2.16.jar:./bin/../lib/metrics-core-2.0.3.jar:./bin/../lib/netty-3.5.9.Final.jar:./bin/../lib/servlet-api-2.5-20081211.jar:./bin/../lib/slf4j-api-1.7.2.jar:./bin/../lib/slf4j-log4j12-1.7.2.jar:./bin/../lib/snakeyaml-1.6.jar:./bin/../lib/snappy-java-1.0.4.1.jar:./bin/../lib/snaptree-0.1.jar:./bin/../lib/jamm-0.2.5.jar
INFO 05:42:41,628 JNA not found. Native methods will be disabled.
INFO 05:42:41,644 Loading settings from file:/root/apache-cassandra-1.2.0/conf/cassandra.yaml
INFO 05:42:43,455 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 05:42:43,456 disk_failure_policy is stop
INFO 05:42:43,462 Global memtable threshold is enabled at 272MB
INFO 05:42:43,473 Unable to load cassandra-topology.properties; compatibility mode disabled
INFO 05:42:45,189 Initializing key cache with capacity of 40 MBs.
INFO 05:42:45,262 Scheduling key cache save to each 14400 seconds (going to save all keys).
INFO 05:42:45,263 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
INFO 05:42:45,273 Scheduling row cache save to each 0 seconds (going to save all keys).
INFO 05:42:46,368 Couldn't detect any schema definitions in local storage.
INFO 05:42:46,369 Found table data in data directories. Consider using the CLI to define your schema.
INFO 05:42:46,676 No commitlog files found; skipping replay
INFO 05:42:47,661 Cassandra version: 1.2.0
INFO 05:42:47,661 Thrift API version: 19.35.0
INFO 05:42:47,661 CQL supported versions: 2.0.0,3.0.0 (default: 3.0.0)
INFO 05:42:47,772 Loading persisted ring state
INFO 05:42:47,778 Starting up server gossip
WARN 05:42:47,788 No host ID found, created 9f65b63f-2054-4fd1-a986-e8a7e335d493 (Note: This should happen exactly once per node).
INFO 05:42:48,061 Enqueuing flush of Memtable-local@1019368344(352/352 serialized/live bytes, 15 ops)
INFO 05:42:48,070 Writing Memtable-local@1019368344(352/352 serialized/live bytes, 15 ops)
INFO 05:42:48,178 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-1-Data.db (354 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=49260)
INFO 05:42:48,489 Starting Messaging Service on port 7000
WARN 05:42:48,525 Generated random token [5225213360801090766]. Random tokens will result in an unbalanced ring; see http://wiki.apache.org/cassandra/Operations
INFO 05:42:48,528 Enqueuing flush of Memtable-local@1355131965(84/84 serialized/live bytes, 4 ops)
INFO 05:42:48,528 Writing Memtable-local@1355131965(84/84 serialized/live bytes, 4 ops)
INFO 05:42:48,535 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-2-Data.db (120 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=49535)
INFO 05:42:48,543 Enqueuing flush of Memtable-local@974339150(50/50 serialized/live bytes, 2 ops)
INFO 05:42:48,543 Writing Memtable-local@974339150(50/50 serialized/live bytes, 2 ops)
INFO 05:42:48,553 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-3-Data.db (109 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=49709)
INFO 05:42:48,682 Node /10.32.146.208 state jump to normal
INFO 05:42:48,701 Enqueuing flush of Memtable-local@794520776(50/50 serialized/live bytes, 2 ops)
INFO 05:42:48,701 Writing Memtable-local@794520776(50/50 serialized/live bytes, 2 ops)
INFO 05:42:48,768 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-4-Data.db (109 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=49883)
INFO 05:42:48,775 Startup completed! Now serving reads.
INFO 05:42:48,802 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-4-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-2-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-1-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-3-Data.db')]
INFO 05:42:49,085 Not starting native transport as requested. Use JMX (StorageService->startNativeTransport()) to start it
INFO 05:42:49,091 Binding thrift service to /0.0.0.0:9160
INFO 05:42:49,120 Compacted to [/var/lib/cassandra/data/system/local/system-local-ia-5-Data.db,]. 692 to 463 (~66% of original) bytes for 1 keys at 0.001452MB/s. Time: 304ms.
INFO 05:42:49,228 Using TFramedTransport with a max frame size of 15728640 bytes.
INFO 05:42:49,238 Using synchronous/threadpool thrift server on 0.0.0.0 : 9160
INFO 05:42:49,239 Listening for thrift clients...
INFO 05:43:49,917 Node /10.114.18.51 is now part of the cluster
INFO 05:43:49,924 InetAddress /10.114.18.51 is now UP
INFO 05:43:49,938 Enqueuing flush of Memtable-peers@391325684(227/227 serialized/live bytes, 14 ops)
INFO 05:43:49,939 Writing Memtable-peers@391325684(227/227 serialized/live bytes, 14 ops)
INFO 05:43:49,947 Completed flushing /var/lib/cassandra/data/system/peers/system-peers-ia-1-Data.db (288 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=50960)
INFO 05:43:49,959 Enqueuing flush of Memtable-local@1401778222(50/50 serialized/live bytes, 2 ops)
INFO 05:43:49,959 Writing Memtable-local@1401778222(50/50 serialized/live bytes, 2 ops)
INFO 05:43:49,966 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-6-Data.db (109 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=51134)
INFO 05:43:53,055 Node /10.203.86.83 is now part of the cluster
INFO 05:43:53,056 InetAddress /10.203.86.83 is now UP
INFO 05:43:53,131 Enqueuing flush of Memtable-peers@661836864(226/226 serialized/live bytes, 14 ops)
INFO 05:43:53,132 Writing Memtable-peers@661836864(226/226 serialized/live bytes, 14 ops)
INFO 05:43:53,138 Completed flushing /var/lib/cassandra/data/system/peers/system-peers-ia-2-Data.db (292 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=52083)
INFO 05:43:53,145 Enqueuing flush of Memtable-local@1894469408(50/50 serialized/live bytes, 2 ops)
INFO 05:43:53,146 Writing Memtable-local@1894469408(50/50 serialized/live bytes, 2 ops)
INFO 05:43:53,162 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-7-Data.db (109 bytes) for commitlog position ReplayPosition(segmentId=1357641766337, position=52257)
^C INFO 05:45:15,019 Stop listening to thrift clients
INFO 05:45:15,026 Announcing shutdown
INFO 05:45:16,028 Waiting for messaging service to quiesce
INFO 05:45:16,031 MessagingService shutting down server thread.
[root@ip-10-32-146-208 apache-cassandra-1.2.0]# ./bin/cassandra -f
xss = -ea -javaagent:./bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms827M -Xmx827M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
INFO 05:45:31,226 Logging initialized
INFO 05:45:31,306 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_05
INFO 05:45:31,307 Heap size: 856686592/857735168
INFO 05:45:31,307 Classpath: ./bin/../conf:./bin/../build/classes/main:./bin/../build/classes/thrift:./bin/../lib/antlr-3.2.jar:./bin/../lib/apache-cassandra-1.2.0.jar:./bin/../lib/apache-cassandra-clientutil-1.2.0.jar:./bin/../lib/apache-cassandra-thrift-1.2.0.jar:./bin/../lib/avro-1.4.0-fixes.jar:./bin/../lib/avro-1.4.0-sources-fixes.jar:./bin/../lib/commons-cli-1.1.jar:./bin/../lib/commons-codec-1.2.jar:./bin/../lib/commons-lang-2.6.jar:./bin/../lib/compress-lzf-0.8.4.jar:./bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:./bin/../lib/guava-13.0.1.jar:./bin/../lib/high-scale-lib-1.1.2.jar:./bin/../lib/jackson-core-asl-1.9.2.jar:./bin/../lib/jackson-mapper-asl-1.9.2.jar:./bin/../lib/jamm-0.2.5.jar:./bin/../lib/jline-1.0.jar:./bin/../lib/json-simple-1.1.jar:./bin/../lib/libthrift-0.7.0.jar:./bin/../lib/log4j-1.2.16.jar:./bin/../lib/metrics-core-2.0.3.jar:./bin/../lib/netty-3.5.9.Final.jar:./bin/../lib/servlet-api-2.5-20081211.jar:./bin/../lib/slf4j-api-1.7.2.jar:./bin/../lib/slf4j-log4j12-1.7.2.jar:./bin/../lib/snakeyaml-1.6.jar:./bin/../lib/snappy-java-1.0.4.1.jar:./bin/../lib/snaptree-0.1.jar:./bin/../lib/jamm-0.2.5.jar
INFO 05:45:31,310 JNA not found. Native methods will be disabled.
INFO 05:45:31,325 Loading settings from file:/root/apache-cassandra-1.2.0/conf/cassandra.yaml
INFO 05:45:33,086 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 05:45:33,086 disk_failure_policy is stop
INFO 05:45:33,092 Global memtable threshold is enabled at 272MB
INFO 05:45:33,103 Unable to load cassandra-topology.properties; compatibility mode disabled
INFO 05:45:35,337 Initializing key cache with capacity of 40 MBs.
INFO 05:45:35,349 Scheduling key cache save to each 14400 seconds (going to save all keys).
INFO 05:45:35,351 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
INFO 05:45:35,420 Scheduling row cache save to each 0 seconds (going to save all keys).
INFO 05:45:36,061 Opening /var/lib/cassandra/data/system/peers/system-peers-ia-1 (288 bytes)
INFO 05:45:36,094 Opening /var/lib/cassandra/data/system/peers/system-peers-ia-2 (292 bytes)
INFO 05:45:36,206 Opening /var/lib/cassandra/data/system/local/system-local-ia-6 (109 bytes)
INFO 05:45:36,210 Opening /var/lib/cassandra/data/system/local/system-local-ia-7 (109 bytes)
INFO 05:45:36,212 Opening /var/lib/cassandra/data/system/local/system-local-ia-5 (463 bytes)
INFO 05:45:36,498 Couldn't detect any schema definitions in local storage.
INFO 05:45:36,499 Found table data in data directories. Consider using the CLI to define your schema.
INFO 05:45:36,937 completed pre-loading (3 keys) key cache.
INFO 05:45:37,056 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357641766337.log, /var/lib/cassandra/commitlog/CommitLog-2-1357641766338.log
INFO 05:45:37,079 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357641766337.log
INFO 05:45:37,830 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357641766337.log
INFO 05:45:37,830 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357641766338.log
INFO 05:45:37,842 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357641766338.log
INFO 05:45:37,967 Enqueuing flush of Memtable-users@1648762822(28/28 serialized/live bytes, 2 ops)
INFO 05:45:37,975 Enqueuing flush of Memtable-local@565849070(52/52 serialized/live bytes, 2 ops)
INFO 05:45:37,978 Writing Memtable-users@1648762822(28/28 serialized/live bytes, 2 ops)
INFO 05:45:38,085 Enqueuing flush of Memtable-schema_keyspaces@2135911387(389/389 serialized/live bytes, 12 ops)
INFO 05:45:38,087 Enqueuing flush of Memtable-schema_columns@926038057(21317/21317 serialized/live bytes, 328 ops)
INFO 05:45:38,087 Enqueuing flush of Memtable-schema_columnfamilies@1107634190(20754/20754 serialized/live bytes, 344 ops)
INFO 05:45:38,100 Completed flushing /var/lib/cassandra/data/system_auth/users/system_auth-users-ia-1-Data.db (72 bytes) for commitlog position ReplayPosition(segmentId=1357641936742, position=142)
INFO 05:45:38,133 Writing Memtable-local@565849070(52/52 serialized/live bytes, 2 ops)
INFO 05:45:38,143 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-8-Data.db (84 bytes) for commitlog position ReplayPosition(segmentId=1357641936742, position=142)
INFO 05:45:38,206 Writing Memtable-schema_keyspaces@2135911387(389/389 serialized/live bytes, 12 ops)
INFO 05:45:38,214 Completed flushing /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-1-Data.db (264 bytes) for commitlog position ReplayPosition(segmentId=1357641936742, position=142)
INFO 05:45:38,216 Writing Memtable-schema_columns@926038057(21317/21317 serialized/live bytes, 328 ops)
INFO 05:45:38,296 Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-1-Data.db (3776 bytes) for commitlog position ReplayPosition(segmentId=1357641936742, position=142)
INFO 05:45:38,297 Writing Memtable-schema_columnfamilies@1107634190(20754/20754 serialized/live bytes, 344 ops)
INFO 05:45:38,315 Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ia-1-Data.db (4437 bytes) for commitlog position ReplayPosition(segmentId=1357641936742, position=142)
INFO 05:45:38,316 Log replay complete, 19 replayed mutations
INFO 05:45:39,092 Cassandra version: 1.2.0
INFO 05:45:39,092 Thrift API version: 19.35.0
INFO 05:45:39,092 CQL supported versions: 2.0.0,3.0.0 (default: 3.0.0)
INFO 05:45:39,204 Loading persisted ring state
ERROR 05:45:39,305 Exception encountered during startup
java.lang.RuntimeException: Could not retrieve DC for /10.114.18.51 from gossip and PFS compatibility is disabled
at org.apache.cassandra.locator.GossipingPropertyFileSnitch.getDatacenter(GossipingPropertyFileSnitch.java:109)
at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:127)
at org.apache.cassandra.locator.TokenMetadata$Topology.addEndpoint(TokenMetadata.java:1040)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:185)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:157)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:441)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:397)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:309)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:397)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:440)
java.lang.RuntimeException: Could not retrieve DC for /10.114.18.51 from gossip and PFS compatibility is disabled
at org.apache.cassandra.locator.GossipingPropertyFileSnitch.getDatacenter(GossipingPropertyFileSnitch.java:109)
at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:127)
at org.apache.cassandra.locator.TokenMetadata$Topology.addEndpoint(TokenMetadata.java:1040)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:185)
at org.apache.cassandra.locator.TokenMetadata.updateNormalTokens(TokenMetadata.java:157)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:441)
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:397)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:309)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:397)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:440)
Exception encountered during startup: Could not retrieve DC for /10.114.18.51 from gossip and PFS compatibility is disabled
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment