Skip to content

Instantly share code, notes, and snippets.

Avatar

Kiollpt

View GitHub Profile
@Kiollpt
Kiollpt / flow.md
Last active July 14, 2021 02:25
#airflow
View flow.md

DD-DataDictionary

  • description: monitor the AWS RDBMS schema change

  • create new column to Summary or Detail:

    1. add COL into google sheet (fill default value it can decrease amount of updating time)
    2. orginzie sql in Summary or Detail func.(fetch table schema from inforamtion_schema table)
    3. col = a.columns[:NUM] in check_cell func -> NUM+=1
@Kiollpt
Kiollpt / aa
Created July 13, 2021 05:49
airflow test
View aa
ss
@Kiollpt
Kiollpt / # airflow
Created July 13, 2021 05:48
View # airflow
‎‎​
@Kiollpt
Kiollpt / bash
Last active July 19, 2021 08:26
#Linux
View bash
LS_COLORS=$LS_COLORS:'di=0;35:' ; export LS_COLORS; ls
export LS_OPTIONS='--color=auto'
#eval "$(dircolors -b)"
alias ls='ls $LS_OPTIONS'
# tmux
@Kiollpt
Kiollpt / start
Last active January 25, 2021 13:24
#tmux
View start
setpup:
~/.vimrc
set mouse=a
watch tail logfile1 logfile2
shortcut : tmux a -t #num
@Kiollpt
Kiollpt / Event-stream based GraphQL subscriptions.md
Created December 13, 2020 03:32 — forked from OlegIlyenko/Event-stream based GraphQL subscriptions.md
Event-stream based GraphQL subscriptions for real-time updates
View Event-stream based GraphQL subscriptions.md

In this gist I would like to describe an idea for GraphQL subscriptions. It was inspired by conversations about subscriptions in the GraphQL slack channel and different GH issues, like #89 and #411.

Conceptual Model

At the moment GraphQL allows 2 types of queries:

  • query
  • mutation

Reference implementation also adds the third type: subscription. It does not have any semantics yet, so here I would like to propose one possible semantics interpretation and the reasoning behind it.

@Kiollpt
Kiollpt / Stateful.scala
Last active November 20, 2020 10:14
#Spark
View Stateful.scala
val output = ds.groupByKey(_.`pickup zone`).mapGroupsWithState[State,Result](GroupStateTimeout.NoTimeout()){
case (key:String,values:Iterator[order],state: GroupState[State]) => {
val data = values.toSeq
println(data)
val size = data.size
print()
val updateState = if(state.exists) {
@Kiollpt
Kiollpt / array.md
Last active October 26, 2020 09:11
#Leetcode
View array.md

Array

  • * 27 Remove Element 视频讲解
  • * 26 Remove Duplicates from Sorted Array 视频讲解 L1:match condition | L2
  • ? 80 Remove Duplicates from Sorted Array II 视频讲解
  • 277 Find the Celebrity 视频讲解
  • * 189 Rotate Array 视频讲解
  • * 41 First Missing Positive 视频讲解
  • *? 299 Bulls and Cows 视频讲解
  • 134 Gas Station 视频讲解
@Kiollpt
Kiollpt / csv to kafka.scala
Last active November 5, 2020 09:57
#spark
View csv to kafka.scala
def main(args: Array[String]): Unit = {
val ss = SparkSession.builder().master("local[*]").appName("app1").getOrCreate()
import ss.implicits._
val filePath = "src/resource"
val schema = Encoders.product[order].schema
val ds = ss.readStream.option("header","true").schema(schema).csv(filePath)
val ds1 = ds.select( lit("1") as "key",to_json(struct("*"))as "value")
@Kiollpt
Kiollpt / main.py
Last active September 12, 2020 05:20
main
View main.py
if __name__ == "__main__":