Skip to content

Instantly share code, notes, and snippets.

View EngHabu's full-sized avatar
💻
Hacking...

Haytham Abuelfutuh EngHabu

💻
Hacking...
View GitHub Profile
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
# Example 1
@task
def task1() -> Union[int, str]:
return "hello world"
--> Outputs literal:
{
"union": {
val: <literal scalar string "hello world">
index: 1

Performance Optimized Configs

Load Optimized

  1. Make sure Propeller's pod has at least 8Gi of memory and 4CPUs
  2. Increase Propeller's throughput:
    propeller:
      workers: 400
    
@EngHabu
EngHabu / side_car.py
Created July 9, 2021 16:28
Example of mounting a shared volume
import os
import time
from typing import List
from flytekit import task, workflow
from flytekitplugins.pod import Pod
from kubernetes.client.models import (
V1Container,
V1EmptyDirVolumeSource,
V1PodSpec,
apiVersion: v1
kind: Namespace
metadata:
name: flyte
---
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
---
# This is a system-wide configuration file used to
# keep track of registries for various container backends.
# It adheres to TOML format and does not support recursive
# lists of registries.
# The default location for this configuration file is /etc/containers/registries.conf.
# The only valid categories are: 'registries.search', 'registries.insecure',
# and 'registries.block'.
@inputs(in_str=Types.String)
@dynamic_task
def dynamic_wf_task(wf_params, in_str):
wf_params.logging.info("Running dynamic task... yielding a code generated sub workflow")
res = []
nodes = {}
previous_node = None
i = 0
for s in in_str:
@EngHabu
EngHabu / override_node_timeout_flytekit.py
Created June 15, 2020 18:57
Example for how to override node timeout in FlyteKit
def with_timeout(node, time_out):
"""
:param SdkNode node: The node to override timeout for.
:param datetime.timedelta time_out: New Timeout for the node.
"""
node.metadata._timeout = time_out
return node
@workflow_class