Skip to content

Instantly share code, notes, and snippets.

id:
type: integer
nullable: false
some_text_field:
type: string
nullable: false
some_date_field:
type: datetime
class Example1(AbstractClassBasedFlow):
...
class Example2(AbstractClassBasedFlow):
...
flow1 = Example1.to_prefect_flow()
flow2 = Example2.to_prefect_flow()
from prefect.flows import Flow
class AbstractClassBasedFlow:
"""
An example abstract class that shows how you can dynamically create prefect
flows.
"""
@classmethod
name: sample_etl_configuration
description: Here's some example of an ETL that serves as an input for data processing
tasks:
extract_from_sql:
template_file: extract.sql
validation_file: extraction_contract.yml
transform:
functions: [add_total_fees, add_total_payout, agg_by_day_and_platform]
validation_file: transformed_contract.yml
load_from_sql:
@jonaslsl
jonaslsl / Spark+ipython_on_MacOS.md
Created January 22, 2018 18:35 — forked from ololobus/Spark+ipython_on_MacOS.md
Apache Spark installation + ipython/jupyter notebook integration guide for macOS

Apache Spark installation + ipython/jupyter notebook integration guide for macOS

Tested with Apache Spark 2.1.0, Python 2.7.13 and Java 1.8.0_112

For older versions of Spark and ipython, please, see also previous version of text.

Install Java Development Kit

@jonaslsl
jonaslsl / conversor_bdi.awk
Last active August 25, 2023 00:59
Conversor BDI Bovespa em script AWK
#!/usr/bin/awk -f
#Autor: Jonas Scherer
#Data: Jan/2016
#Uso: ./conversor_bdi.awk COTAHIST_A2015.TXT
#Detalhes: Conversor do formato de cotacoes historicas disponiveis no site da Bovespa.
#O conversor podera ser usado para exportar os dados para excel para serem trabalhados de uma forma mais simples.
#Util para quem faz day-trades e deseja criar seus proprios dashboards ou fazer backtests.
#OBS: Remover a primeira e ultima linha para remover o cabecalho e o rodape que vem da Bovespa
### init.sh
#!/bin/bash
echo -n "Username: "
read username
export username
echo -n "Password: "
read -s password
export password
echo "\r"
#!/usr/bin/env ruby
user_id = 3
puts "Start"
Dir["./reports/*"].each do |file_name|
file_name.gsub!(" ", "\\ ")
puts "./arachni_web_scan_import #{file_name} #{user_id}"
system("./arachni_web_scan_import #{file_name} #{user_id}")
end
puts "End"
puts "Start"
input = File.read("./input.txt")
input.gsub!(/\r\n?/, "\n")
input.each_line do |line|
s = line.strip
puts "./arachni #{s} --report-save-path ./reports"
system("./arachni #{s} --report-save-path ./reports")
end
puts "End"
.element {
position: relative;
top: 50%;
transform: translateY(-50%);
}