Skip to content

Instantly share code, notes, and snippets.

@garystafford
Last active December 15, 2018 16:32
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save garystafford/98cbaf3a03041edd5974528a24d18ac8 to your computer and use it in GitHub Desktop.
createTime: '2018-12-15T16:31:21.779Z'
id: template-demo-1
jobs:
- sparkJob:
jarFileUris:
- gs://dataproc-demo-bucket/dataprocJavaDemo-1.0-SNAPSHOT.jar
mainClass: org.example.dataproc.InternationalLoansAppDataprocSmall
stepId: ibrd-small-spark
- sparkJob:
jarFileUris:
- gs://dataproc-demo-bucket/dataprocJavaDemo-1.0-SNAPSHOT.jar
mainClass: org.example.dataproc.InternationalLoansAppDataprocLarge
stepId: ibrd-large-spark
- pysparkJob:
args:
- gs://dataproc-demo-bucket
- ibrd-statement-of-loans-historical-data.csv
- ibrd-summary-large-python
mainPythonFileUri: gs://dataproc-demo-bucket/international_loans_dataproc.py
stepId: ibrd-large-pyspark
name: projects/dataproc-demo-224523/regions/us-east1/workflowTemplates/template-demo-1
placement:
managedCluster:
clusterName: three-node-cluster
config:
gceClusterConfig:
zoneUri: us-east1-b
masterConfig:
diskConfig:
bootDiskSizeGb: 500
machineTypeUri: n1-standard-4
softwareConfig:
imageVersion: 1.3-deb9
workerConfig:
diskConfig:
bootDiskSizeGb: 500
machineTypeUri: n1-standard-4
numInstances: 2
updateTime: '2018-12-15T16:32:06.508Z'
version: 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment