Skip to content

Instantly share code, notes, and snippets.

[ec2-user@ip-10-150-96-149 apache-drill-1.11.0-SNAPSHOT]$ bin/drill-embedded
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Jul 24, 2017 6:34:27 PM org.glassfish.jersey.server.ApplicationHandler initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.11.0-SNAPSHOT
"drill baby drill"
0: jdbc:drill:zk=local> use kudu;
+-------+-----------------------------------+
| ok | summary |
#!/bin/bash
#
# userdata for Kudu tablet server colocated with spark
#
echo '* - nofile 32768' >> /etc/security/limits.conf
echo '* - nproc 65536' >> /etc/security/limits.conf
sysctl -w fs.file-max=32768
yum remove -y java-1.7.0-openjdk
yum install -y sysstat java-1.8.0-openjdk java-1.8.0-openjdk-devel
@cresny
cresny / gist:0e109f843730b64d3a330f8fb06bb8a6
Created June 23, 2017 17:22
1 successful run and cp after resume, next 2 fail
2017-06-23 16:59:56.580 [ip-10-150-96-68] INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Removing web dashboard root cache directory /tmp/flink-web-dbb2f241-ccfd-4a51-a8b3-de92d1b5f72d
2017-06-23 17:00:08.453 [ip-10-150-96-68] INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE)
2017-06-23 17:00:10.124 [ip-10-150-96-68] INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-b9a67936-f87e-459e-b32a-a7c352f61ec6
2017-06-23 17:00:10.126 [ip-10-150-96-68] INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:45857 - max concurrent requests: 50 - max backlog: 1000
2017-06-23 17:00:10.193 [ip-10-150-96-68] INFO org.apache.flink.runtime.metrics.MetricRegistry - Configuring JMXReporter with {port=8789, class=org.apache.flink.metrics.jmx.JMXReporter}.
2017-06-23 17:00:10.258 [ip-10-150-96-68] INFO org.apache.flink.runtime.metrics.MetricRegistry - Reporting metrics for reporter
@cresny
cresny / checkpoint-hang.log
Created June 23, 2017 16:51
cp1-cp3 (success), cancel -> resume cp3, cp4 (hang...)
2017-06-23 16:43:44.907 [ip-10-150-96-68] INFO org.apache.flink.runtime.blob.BlobClient - Blob client connecting to akka://flink/user/jobmanager
2017-06-23 16:43:45.235 [ip-10-150-96-68] INFO org.apache.flink.runtime.client.JobClient - Checking and uploading JAR files
2017-06-23 16:43:45.235 [ip-10-150-96-68] INFO org.apache.flink.runtime.blob.BlobClient - Blob client connecting to akka://flink/user/jobmanager
2017-06-23 16:43:45.570 [ip-10-150-96-68] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart
2017-06-23 16:43:45.573 [ip-10-150-96-68] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Persisting periodic checkpoints externally at s3://mm-flink-checkpoints//cliff.
2017-06-23 16:43:45.573 [ip-10-150-96-68] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Persisting periodic checkpoints externally at s3://mm-flink-checkpoints//cliff.
2017-06-23 16:43:45.574 [ip-10-150-96-68] INFO org.apache.flink.
@cresny
cresny / savepoint-fail.log
Created June 20, 2017 21:48
restart after two external checkpoints, then fail on first subsequent checkpoint
2017-06-20 21:39:04.768 [ip-10-150-96-228] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Persisting periodic checkpoints externally at s3://mm-flink-checkpoints//cliff.
2017-06-20 21:39:07.727 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/test/state/c5e1089f1647880d96ed2d46b8409020.
2017-06-20 21:39:07.732 [ip-10-150-96-53] INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Attempting to load RocksDB native library and store it under '/media/flink/tmp0'
2017-06-20 21:39:07.733 [ip-10-150-96-53] DEBUG org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Attempting to create RocksDB native library folder /media/flink/tmp0/rocksdb-lib-d91dd1136bd29e4ab106d848dbc56d4a
2017-06-20 21:39:07.752 [ip-10-150-96-53] DEBUG o.a.flink.runtime.state.filesystem.FsCheckpointStreamFactory - Initialed file stream factory to URI s3://mm-prod-stats-pulse-streaming/te
@cresny
cresny / savepoint-incremental-fail
Created June 20, 2017 19:39
Seven incremental savepoints with intermittent restores, last one fails
2017-06-20 18:44:39.376 [ip-10-150-96-228] INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Attempting to load RocksDB native library and store it under '/media/flink/tmp0'
2017-06-20 18:44:39.378 [ip-10-150-96-228] DEBUG org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Attempting to create RocksDB native library folder /media/flink/tmp0/rocksdb-lib-18151a61e3774f0bcd2b1adeed79010e
2017-06-20 18:44:39.476 [ip-10-150-96-228] INFO org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Successfully loaded RocksDB native library
2017-06-20 18:44:39.521 [ip-10-150-96-228] INFO o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Initializing RocksDB keyed state backend from snapshot.
2017-06-20 18:44:39.522 [ip-10-150-96-228] DEBUG o.a.flink.contrib.streaming.state.RocksDBKeyedStateBackend - Restoring snapshot from state handles: null.
2017-06-20 18:45:45.067 [ip-10-150-96-228] INFO com.mediamath.reporting.combiner.Combiner - start combiner snapshot for id