Skip to content

Instantly share code, notes, and snippets.

View klion26's full-sized avatar

Congxian Qiu klion26

126 followers · 654 following
View GitHub Profile
@klion26
klion26 / makefile.md
Last active September 4, 2023 02:26
makefile question
  1. $(JAVA_HOME)/include/darwin 是啥
  2. 下面的 makefile 逻辑是不是类似替换后的那样
  3. 如果用 docker（linux64）的编译，能否生成 linux64 的 so 呢？

原始 makefile

ifeq ($(PLATFORM), OS_MACOSX)
ifeq (,$(findstring lib$(PROJECT_NAME)jni-osx,$(JNILIBNAME)))
ifeq ($(MACHINE),arm64)
        JNILIBNAME = lib$(PROJECT_NAME)jni-osx-arm64.jnilib
@klion26
klion26 / gist:026a79897334fdeefec381cf7cdd5d93
Last active July 5, 2020 03:24
Dependency for release-1.11.0-RC4

result generated using mvn dependency:tree -Dverbos|grep -v ":test"|grep -v ":provided"

flink-connector-elasticsearch6

[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ flink-connector-elasticsearch6_2.11 ---
[INFO] org.apache.flink:flink-connector-elasticsearch6_2.11:jar:1.11.0
[INFO] +- org.apache.flink:flink-connector-elasticsearch-base_2.11:jar:1.11.0:compile
[INFO] +- org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:6.3.1:compile
[INFO] |  +- org.elasticsearch:elasticsearch:jar:6.3.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-core:jar:6.3.1:compile
@klion26
klion26 / rocksdb_single_cf
Created May 21, 2020 03:08
RocksDB 单 CF 相关结果
1 拉取修改过后的 flink 代码
`git clone https://github.com/klion26/flink.git -b single_cf`
2 install flink 到本地
`mvn clean install -Denforcer.skip -Dcheckstyle.skip -DskipTests`
3 拉取 flink-benchmark 代码到本地
`git clone https://github.com/dataArtisans/flink-benchmarks.git`
4 切换到 flink-benchmark 目录，执行 flink-benchmark 跑结果，结果会输出到目录下的 jmh-result.csv 中，在服务器上跑时，建议使用 nohup 启动，防止跑到一般退出的问题
`mvn clean package exec:exec -Dexec.executable=java -Dexec.args="-jar target/benchmarks.jar -rf csv org.apache.flink.state.benchmark.*"`
@klion26
klion26 / jmh-result.csv
Last active May 20, 2020 02:04
JMH Result of RocksDB with Single CF(serialized state name in the byte key)
jmh-result_statename_20 $(stateName)=20
Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: backendType
org.apache.flink.state.benchmark.ListStateBenchmark.listAdd thrpt 1 30 453.769667 8.444930 ops/ms ROCKSDB
org.apache.flink.state.benchmark.ListStateBenchmark.listAddAll thrpt 1 30 223.345186 12.710213 ops/ms ROCKSDB
org.apache.flink.state.benchmark.ListStateBenchmark.listAppend thrpt 1 30 443.680643 6.849584 ops/ms ROCKSDB
org.apache.flink.state.benchmark.ListStateBenchmark.listGet thrpt 1 30 88.892699 2.259862 ops/ms ROCKSDB
org.apache.flink.state.benchmark.ListStateBenchmark.listGetAndIterate thrpt 1 30 88.998718 2.213051 ops/ms ROCKSDB
org.apache.flink.state.benchmark.ListStateBenchmark.listUpdate thrpt 1 30 456.334396 13.475957 ops/ms ROCKSDB
org.apache.flink.state.benchmark.MapStateBenchmark.mapAdd thrpt 1 30 433.497316 11.128228 ops/ms ROCKSDB
@klion26
klion26 / gist:f3ddb3fe839901aa56b4862caaa72498
Created April 23, 2019 06:07
/var/flink/data/localState/aid_AllocationID{37a99d74a8e452ff06257c61ab13a3c8}/jid_6241b30b0adb82bd50cd5d37aa6128d1/vtx_54b7f0cbe350c274d942032aa504dbdd_sti_1/chk_21244/rocks_db CoStreamFlatMap_54b7f0cbe350c274d942032aa504dbdd__2_90__
/var/flink/data/localState/aid_AllocationID{37a99d74a8e452ff06257c61ab13a3c8}/jid_6241b30b0adb82bd50cd5d37aa6128d1/vtx_567adb020dcc57a12c17bd43c00b0f55_sti_53/chk_21244/rocks_db CoBroadcastWithKeyedOperator_567adb020dcc57a12c17bd43c00b0f55__54_90__
/var/flink/data/localState/aid_AllocationID{37a99d74a8e452ff06257c61ab13a3c8}/jid_6241b30b0adb82bd50cd5d37aa6128d1/vtx_567adb020dcc57a12c17bd43c00b0f55_sti_53/chk_21244/rocks_db StreamMap_3c5866a6cc097b462de842b2ef91910d__54_90__
/var/flink/data/localState/aid_AllocationID{37a99d74a8e452ff06257c61ab13a3c8}/jid_6241b30b0adb82bd50cd5d37aa6128d1/vtx_bc2936094388a70852534bd6c0fce178_sti_77/chk_21244/rocks_db WindowOperator_bc2936094388a70852534bd6c0fce178__78_90__
/var/flink/data/localState/aid_AllocationID{5cde66b8a81c5202f7685928bb1
@klion26
klion26 / POJOMigration
Last active March 11, 2019 01:26
public static void main(String[] args) throws Exception {
// Checking input parameters
final ParameterTool params = ParameterTool.fromArgs(args);
// set up the execution environment
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// make parameters available in the web interface
env.getConfig().setGlobalJobParameters(params);
@klion26
klion26 / testRecoveryFailureLog.java
Last active January 29, 2019 09:41
testCheckpointRecoveryFailure
public void testRecoveryFailureLog() throws Exception {
final Time timeout = Time.seconds(30L);
final File zookeeperStoragePath = temporaryFolder.newFolder();
// Config
final int numberOfJobManagers = 2;
final int numberOfTaskManagers = 2;
final int numberOfSlotsPerTaskManager = 2;
assertEquals(PARALLELISM, numberOfTaskManagers * numberOfSlotsPerTaskManager);
@klion26
klion26 / FailTest
Created August 5, 2018 13:31
FailTest
/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/bin/java -ea -Dlog.level=WARN -DforkNumber=01 -Dlog4j.configuration=log4j-test.properties -Dcodebase=new -Xms256m -Xmx2048m -Dmvn.forkNumber=1 -XX:+UseG1GC -Didea.test.cyclic.buffer.size=1048576 "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64939:/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/
@klion26
klion26 / FailTest
Created August 5, 2018 13:31
FailTest
/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/bin/java -ea -Dlog.level=WARN -DforkNumber=01 -Dlog4j.configuration=log4j-test.properties -Dcodebase=new -Xms256m -Xmx2048m -Dmvn.forkNumber=1 -XX:+UseG1GC -Didea.test.cyclic.buffer.size=1048576 "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64939:/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/
@klion26
klion26 / FailTest
Created August 5, 2018 13:31
FailTest
/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/bin/java -ea -Dlog.level=WARN -DforkNumber=01 -Dlog4j.configuration=log4j-test.properties -Dcodebase=new -Xms256m -Xmx2048m -Dmvn.forkNumber=1 -XX:+UseG1GC -Didea.test.cyclic.buffer.size=1048576 "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64939:/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/