This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
workers: | |
enabled: true | |
replicas: 3 | |
nodeSelector: | |
kops.k8s.io/instancegroup: airflow-workers |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
persistence: | |
enabled: true | |
storageClass: efs-airflow | |
accessMode: ReadWriteMany | |
size: 1Gi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Parameter | Explanation | |
---|---|---|
core.parallelism | Maximum number of tasks running across an entire Airflow installation | |
core.dag_concurrency | The number of task instances allowed to run concurrently by the scheduler | |
core.non_pooled_task_slot_count | Number of task slots allocated to tasks not running in a pool | |
core.max_active_runs_per_dag | Maximum number of active DAG runs - per DAG | |
scheduler.max_threads | How many threads the scheduler process should use to use to schedule DAGs | |
celery.worker_concurrency | Number of task instances that a worker will take if using CeleryExecutor |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pipeline { | |
agent { | |
kubernetes { | |
label podLabel | |
yaml pod | |
} | |
} | |
stages { | |
stage('Checkout') { | |
steps { |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# !! AUTO-GENERATED !! | |
# application_id: {{{ ni_application_id }}} | |
# version: {{{ version }}} | |
import airflow | |
from airflow import DAG | |
from airflow.contrib.operators.emr_add_steps_operator \ | |
import EmrAddStepsOperator | |
from operators.emr_cluster_name_to_id import EmrClusterNameToIdOperator | |
from airflow.contrib.sensors.emr_step_sensor import EmrStepSensor |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
--- | |
ni_application_id: ${project.name} | |
version: ${project.version} | |
main_class: com.naturalint.SparkExample | |
schedule: None | |
timeout_minutes: 45 | |
jar_location: s3://<bucket-name>//${gitRepositoryName}/${s3ArtifactsDir}/${project.version}${gitCommitIdStr}/${project.build.finalName}.jar | |
emr_cluster_name: ${gitRepositoryName}-ci | |
properties: | |
- property: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
APP_ID = <APP_ID> | |
def getJsonWebToken(privateKey) { | |
try { | |
privateCrtKey = getRSAPrivateKey(privateKey) | |
time = accessTime() | |
def jsonWebToken = Jwts.builder() | |
.setSubject("RS256") | |
.signWith(RS256, privateCrtKey) | |
.setExpiration(time['expirationTime']) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def validateAuth(jsonWebToken) { | |
try { | |
def httpConn = new URL("https://api.github.com/app").openConnection(); | |
httpConn.setRequestProperty( 'Authorization', "Bearer ${jsonWebToken}" ) | |
httpConn.setRequestProperty( 'Accept', 'application/vnd.github.machine-man-preview+json' ) | |
return httpConn.getResponseCode(); | |
} catch(Exception e){ | |
echo "Exception: ${e}" | |
error "Authentication request failed" | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
INSTALLATION_ID = <INSTALLATION_ID> | |
def getToken(jsonWebToken) { | |
try { | |
def httpConn = new URL("https://api.github.com/app/installations/${INSTALLATION_ID}/access_tokens").openConnection(); | |
httpConn.setRequestProperty( 'Authorization', "Bearer ${jsonWebToken}" ) | |
httpConn.setRequestProperty( 'Accept', 'application/vnd.github.machine-man-preview+json' ) | |
httpConn.setRequestMethod("POST"); | |
def responseText = httpConn.getInputStream().getText() | |
def slurper = new JsonSlurper() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ORGANIZATION_NAME = <ORGANIZATION_NAME> | |
def getPreviousCheckNameRunID(repository, commitID, token, checkName) { | |
try { | |
def httpConn = new URL("https://api.github.com/repos/${ORGANIZATION_NAME}/${repository}/commits/${commitID}/check-runs").openConnection(); | |
httpConn.setDoOutput(true) | |
httpConn.setRequestProperty( 'Authorization', "token ${token}" ) | |
httpConn.setRequestProperty( 'Accept', 'application/vnd.github.antiope-preview+json' ) | |
checkRuns = httpConn.getInputStream().getText(); | |
def slurperCheckRun = new JsonSlurper() |
OlderNewer