Skip to content

Instantly share code, notes, and snippets.

View skritch's full-sized avatar

Sam Kritchevsky skritch

View GitHub Profile
@skritch
skritch / kedro_f_to_node.py
Last active September 6, 2022 17:49
kedro_f_to_node.py
import inspect
import re
from typing import Callable, Union, Optional
import types
from kedro.pipeline import Pipeline, node
from kedro.pipeline.node import Node
CAMEL_TO_SNAKE_RE = re.compile(r'(?<!^)(?=[A-Z])')
@skritch
skritch / ListFilesJob.scala
Last active February 5, 2021 22:26
List S3 files fast with Flink
import scala.collection.JavaConverters._
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
import com.amazonaws.services.s3.model.{ListObjectsV2Request, ListObjectsV2Result}
import com.amazonaws.services.s3.{AmazonS3, AmazonS3ClientBuilder}
import org.apache.flink.api.common.serialization.SimpleStringEncoder
import org.apache.flink.api.scala._
import org.apache.flink.core.fs.Path
import org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink
import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner
@skritch
skritch / flink_avro_parquet.scala
Last active September 1, 2020 17:18
A partial implementation of a way to write a batch table to Parquet at a path in Flink, tested on 1.10

Keybase proof

I hereby claim:

  • I am skritch on github.
  • I am skritch (https://keybase.io/skritch) on keybase.
  • I have a public key ASDQcppHLNIVBjh50QMaxC1GxHr4d5Zt-CVuZec556sgSAo

To claim this, I am signing this object:

from itertools import imap
import json
class Map(object):
def __init__(self, iterable, steps=[]):
"""
Construct an iterator that looks up items in the given iterable by index, key, field name, or calls
a function on them.
:param iterable: An iterable of "objects"
from alex import coliseum
c = coliseum.Coliseum(radius=50)
Cpillar=coliseum.Pillar(textures=basalt) # these pillars are all exactly the same object
# silly version
map(c.append(Cpillar, _), [
(-40, 0),
(-30, -30),
(0, -40)
]