Skip to content

Instantly share code, notes, and snippets.

View naturalett's full-sized avatar

Lidor Ettinger naturalett

6 followers · 1 following
View GitHub Profile
@naturalett
naturalett / values.yaml
Created November 18, 2019 09:27
Workers configuration
workers:
  enabled: true
  replicas: 3
 nodeSelector:
 kops.k8s.io/instancegroup: airflow-workers
@naturalett
naturalett / values.yaml
Created November 18, 2019 09:32
Storage configuration for DAGs
persistence:
enabled: true
storageClass: efs-airflow
accessMode: ReadWriteMany
size: 1Gi
@naturalett
naturalett / airflow_cfg.csv
Created November 21, 2019 15:00
Airflow Configuration
Parameter Explanation
core.parallelism Maximum number of tasks running across an entire Airflow installation
core.dag_concurrency The number of task instances allowed to run concurrently by the scheduler
core.non_pooled_task_slot_count Number of task slots allocated to tasks not running in a pool
core.max_active_runs_per_dag Maximum number of active DAG runs - per DAG
scheduler.max_threads How many threads the scheduler process should use to use to schedule DAGs
celery.worker_concurrency Number of task instances that a worker will take if using CeleryExecutor
@naturalett
naturalett / dags_ci_cd.yaml
Created January 24, 2020 11:02
dags ci cd
pipeline {
agent {
kubernetes {
label podLabel
yaml pod
}
}
stages {
stage('Checkout') {
steps {
@naturalett
naturalett / dag.py.mustache
Created January 24, 2020 11:08
dag template
# !! AUTO-GENERATED !!
# application_id: {{{ ni_application_id }}}
# version: {{{ version }}}
import airflow
from airflow import DAG
from airflow.contrib.operators.emr_add_steps_operator \
import EmrAddStepsOperator
from operators.emr_cluster_name_to_id import EmrClusterNameToIdOperator
from airflow.contrib.sensors.emr_step_sensor import EmrStepSensor
@naturalett
naturalett / dag_properties.yml
Created January 24, 2020 11:11
dag properties
---
ni_application_id: ${project.name}
version: ${project.version}
main_class: com.naturalint.SparkExample
schedule: None
timeout_minutes: 45
jar_location: s3://<bucket-name>//${gitRepositoryName}/${s3ArtifactsDir}/${project.version}${gitCommitIdStr}/${project.build.finalName}.jar
emr_cluster_name: ${gitRepositoryName}-ci
properties:
- property:
@naturalett
naturalett / getJsonWebToken.groovy
Created February 23, 2020 22:09
getJsonWebToken
APP_ID = <APP_ID>
def getJsonWebToken(privateKey) {
try {
privateCrtKey = getRSAPrivateKey(privateKey)
time = accessTime()
def jsonWebToken = Jwts.builder()
.setSubject("RS256")
.signWith(RS256, privateCrtKey)
.setExpiration(time['expirationTime'])
@naturalett
naturalett / validateAuth.groovy
Created February 23, 2020 22:19
validateAuth
def validateAuth(jsonWebToken) {
try {
def httpConn = new URL("https://api.github.com/app").openConnection();
httpConn.setRequestProperty( 'Authorization', "Bearer ${jsonWebToken}" )
httpConn.setRequestProperty( 'Accept', 'application/vnd.github.machine-man-preview+json' )
return httpConn.getResponseCode();
} catch(Exception e){
echo "Exception: ${e}"
error "Authentication request failed"
}
@naturalett
naturalett / getToken.groovy
Created February 23, 2020 22:27
getToken
INSTALLATION_ID = <INSTALLATION_ID>
def getToken(jsonWebToken) {
try {
def httpConn = new URL("https://api.github.com/app/installations/${INSTALLATION_ID}/access_tokens").openConnection();
httpConn.setRequestProperty( 'Authorization', "Bearer ${jsonWebToken}" )
httpConn.setRequestProperty( 'Accept', 'application/vnd.github.machine-man-preview+json' )
httpConn.setRequestMethod("POST");
def responseText = httpConn.getInputStream().getText()
def slurper = new JsonSlurper()
@naturalett
naturalett / getPreviousCheckNameRunID.groovy
Created February 23, 2020 22:30
getPreviousCheckNameRunID
ORGANIZATION_NAME = <ORGANIZATION_NAME>
def getPreviousCheckNameRunID(repository, commitID, token, checkName) {
try {
def httpConn = new URL("https://api.github.com/repos/${ORGANIZATION_NAME}/${repository}/commits/${commitID}/check-runs").openConnection();
httpConn.setDoOutput(true)
httpConn.setRequestProperty( 'Authorization', "token ${token}" )
httpConn.setRequestProperty( 'Accept', 'application/vnd.github.antiope-preview+json' )
checkRuns = httpConn.getInputStream().getText();
def slurperCheckRun = new JsonSlurper()