Skip to content

Instantly share code, notes, and snippets.

@DharmitD
Created November 13, 2024 05:15
Show Gist options
  • Save DharmitD/e1711ddc9fe9e3e26f977a0bcfd72d24 to your computer and use it in GitHub Desktop.
Save DharmitD/e1711ddc9fe9e3e26f977a0bcfd72d24 to your computer and use it in GitHub Desktop.
Semaphores and Mutexes PipelineSpec IR
# PIPELINE DEFINITION
# Name: hello-world
# Description: A simple intro pipeline
# Inputs:
# text: str [Default: 'hi there']
components:
comp-hello-world:
executorLabel: exec-hello-world
inputDefinitions:
parameters:
text:
parameterType: STRING
outputDefinitions:
parameters:
Output:
parameterType: STRING
deploymentSpec:
executors:
exec-hello-world:
container:
args:
- --executor_input
- '{{$}}'
- --function_to_execute
- hello_world
command:
- sh
- -c
- "\nif ! [ -x \"$(command -v pip)\" ]; then\n python3 -m ensurepip ||\
\ python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1\
\ python3 -m pip install --quiet --no-warn-script-location 'kfp==2.9.0'\
\ '--no-deps' 'typing-extensions>=3.7.4,<5; python_version<\"3.9\"' && \"\
$0\" \"$@\"\n"
- sh
- -ec
- 'program_path=$(mktemp -d)
printf "%s" "$0" > "$program_path/ephemeral_component.py"
_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main --component_module_path "$program_path/ephemeral_component.py" "$@"
'
- "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import\
\ *\n\ndef hello_world(text: str) -> str:\n print(text)\n return text\n\
\n"
image: python:3.9
pipelineInfo:
description: A simple intro pipeline
name: hello-world
root:
dag:
tasks:
hello-world:
cachingOptions:
enableCache: true
componentRef:
name: comp-hello-world
inputs:
parameters:
text:
componentInputParameter: text
taskInfo:
name: hello-world
inputDefinitions:
parameters:
text:
defaultValue: hi there
isOptional: true
parameterType: STRING
schemaVersion: 2.1.0
sdkVersion: kfp-2.9.0
---
platforms:
kubernetes:
pipelineConfig:
mutexName: mutex
semaphoreKey: semaphore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment