pryce-turner

Processing Files with Dynamic Container Tasks in Flyte

Motivation

I'm still very new to Flyte but it's quickly becoming one of my favorite tools. Between the modern architectural decisions and a welcoming and extremely helpful community, what's not to love?

Some of the abstractions however, do require a slight shift in assumptions compared to more localized workflow frameworks. Additionally, while the documentation is excellent at explaining Flyte's different features in isolation, there aren't as many posts on how they work together like there are for more established projects. This short piece is my attempt to capture my journey so far with a rather specific usecase that ties together a number of notable features. This is mostly to solidify my own understanding, but hopefully also a way to pay forward all the wonderful support I've received from the team.

Setup

• Everything you need to run the code below is captured in the [Envi

pimentel

catDf <- data.frame(meow = rnorm(100), purr = rpois(100, 3))
aList <- list(aVector = 1:20, aDf = catDf, anotherList = list(1:200, 1:20))

# annoying as hell, right?
head(aList)

#' Return the first or last part of a list
#'
#' Returns the first or last part of a list. Instead of returning the first
#' n entries as the standard head() does, it attempts to call head()

davetang

#!/usr/bin/env perl

use strict;
use warnings;

=head1 NAME

This scripts converts a psl file into a bed file written by Dave Tang.

=head1 SYNOPSIS