framiere

for epub in $(ls *epub| grep -vE "\(\d+\)\.epub$") ; do
    content=$(unzip -p $epub OEBPS/content.opf)
    title=$(echo $content | xpath "(//dc:title/text())[1]")
    identifier=$(echo $content | xpath "(//dc:identifier/text())[1]")
    mv $epub "$title ($identifier).epub"
done

framiere

sh
ZOOKEEPER_NODES=$(grep "zookeeper.connect=" /etc/kafka/server.properties | cut -d '=' -f 2)

zk_nodes=$(echo $ZOOKEEPER_NODES | tr "," "\n")
for zk_node in $zk_nodes
do
   zk_host=$(echo $zk_node | cut -d':' -f1)
   zk_port=$(echo $zk_node | cut -d':' -f2)
   zk_node_type=$(echo 'mntr' | nc $zk_host $zk_port | grep 'zk_server_state'| cut -f2)
   echo "$zk_node is $zk_node_type"

framiere

Architectures trade offs

For the following architecture we suppose that you are using the acks=all property.

Note: even if you did explicitly set acks=all or set something different thanall if you are using EOS, idempotency or Transaction, the producer will set it to all in the background.

Note: when choosing anything different than acks=all you are guaranteed to lose messages regardless of the failure.

Note: when using more brokers than the replication factor, you have less possibility of data loss but you are less available when using quorum min.insync.replicas as partitions are scattered around brokers. Please set the replication factor and min.insync.replicas adequately.

framiere

https://www.confluent.io/blog/taking-ksql-spin-using-real-time-device-data/
https://gist.github.com/rmoff/7efa882dfd808dbab4eb7b8e6f9eda16
https://asciinema.org/a/nNfYKsVFwzOKMIWfdj5L16uzW
https://github.com/confluentinc/ksql/blob/master/ksql-examples/src/main/java/io/confluent/ksql/datagen/DataGen.java
https://github.com/confluentinc/ksql/blob/master/ksql-parser/src/main/antlr4/io/confluent/ksql/parser/SqlBase.g4
https://github.com/confluentinc/ksql/blob/v5.1.0-beta201806200051/ksql-engine/src/main/java/io/confluent/ksql/KsqlContext.java
https://github.com/mmolimar/ksql-jdbc-driver
https://twitter.com/alexott_en/status/1008384806693097472
https://github.com/framiere/a-kafka-story
https://docs.confluent.io/current/ksql/docs/syntax-reference.html

framiere

## Consumer Throughput: Single consumer thread, no compression
## Consumer Throughput: 3 consumer thread, no compression

bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \
--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \
--messages 15000000 \
--threads 1

framiere

graph TB
network --> network_threads((network_threads))
network_threads --> request_queue
request_queue --> io_threads((io_threads))
io_threads --> page_cache
io_threads --> purgatory
purgatory --> other_brokers
purgatory --> response_queue
response_queue --> network

framiere

Apple d'offre Composants cités hadoop, hdfs, hbase, kafka, flume, spark, hbase, avro, orc, sqoop, oozie

Connecteurs cités sgbd, SSAS, SSRS, excel, PowerBI, trigger db

Objectif

Est-ce que l'objectif est de TOUS les tester ?

framiere

root@quest-test:/home/framiere# curl -sSL https://get.docker.com/ | sh
Warning: the "docker" command appears to already exist on this system.

If you already have Docker installed, this script can cause trouble, which is
why we're displaying this warning and provide the opportunity to cancel the
installation.

If you installed the current Docker package using this script and are using it
again to update Docker, you can safely ignore this message.

framiere

12:53 $ http --verify no https://www.hopwork.fr/
HTTP/1.1 500 Internal Server Error
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: keep-alive
Content-Language: fr-FR
Content-Type: application/json;charset=UTF-8
Date: Wed, 14 Oct 2015 10:53:33 GMT
Expires: 0
Pragma: no-cache

framiere

<dependency>
    <groupId>net.sf.squirrel-sql</groupId>
    <artifactId>squirrel-sql</artifactId>
    <version>3.7.0</version>
    <scope>system</scope>
    <systemPath>${project.basedir}/src/main/lib/squirrel-sql-3.7.0.jar</systemPath>
</dependency>