Skip to content

Instantly share code, notes, and snippets.

View joao-parana's full-sized avatar

João Antonio Ferreira joao-parana

40 followers · 134 following
View GitHub Profile
@joao-parana
joao-parana / TestAppendWithPartitioning.java
Created October 30, 2022 21:26
TestAppendWithPartitioning for test identity transform partitioning in Iceberg version 1.0.0
import org.apache.hadoop.conf.Configuration;
import org.apache.iceberg.*;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.catalog.TableIdentifier;
import org.apache.iceberg.data.GenericRecord;
import org.apache.iceberg.data.IcebergGenerics;
import org.apache.iceberg.data.Record;
import org.apache.iceberg.data.parquet.GenericParquetWriter;
import org.apache.iceberg.hadoop.HadoopCatalog;
import org.apache.iceberg.io.CloseableIterable;
@treuille
treuille / caching_DAG_example.py
Last active May 30, 2023 17:18
This demonstrates how piping cached functions into one another automatically sets up an efficient directed acyclic computational graph.
import streamlit as st
import pandas as pd
@st.cache
def load_metadata():
DATA_URL = "https://streamlit-self-driving.s3-us-west-2.amazonaws.com/labels.csv.gz"
return pd.read_csv(DATA_URL, nrows=1000)
@st.cache
def create_summary(metadata, summary_type):
@HoverBaum
HoverBaum / create-blogposts.js
Created March 22, 2018 13:12
Creating blogposts in Contentful.
const createBlogPosts = async (posts, assets, categories, managementToken, spaceId, simpleLog = console.log) => {
const client = contentful.createClient({
accessToken: managementToken,
logHandler: (level, data) => simpleLog(`${level} | ${data}`)
})
const space = await client.getSpace(spaceId)
const linkMap = new Map()
assets.forEach(asset => linkMap.set(asset.wpAsset.link, asset.fields.file['en-US'].url))
@joao-parana
joao-parana / hello.html
Last active December 18, 2017 19:04
SOMA 01 - Hello World
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="description" content="SOMA 01 - Hello World">
<title>SOMA 01 - Hello World</title>
</head>
<body>
<script src="https://d3js.org/d3.v4.min.js" charset="utf-8"></script>
<script>
@andrearota
andrearota / example.scala
Created October 18, 2016 08:40
Creating Spark UDF with extra parameters via currying
// Problem: creating a Spark UDF that take extra parameter at invocation time.
// Solution: using currying
// http://stackoverflow.com/questions/35546576/how-can-i-pass-extra-parameters-to-udfs-in-sparksql
// We want to create hideTabooValues, a Spark UDF that set to -1 fields that contains any of given taboo values.
// E.g. forbiddenValues = [1, 2, 3]
// dataframe = [1, 2, 3, 4, 5, 6]
// dataframe.select(hideTabooValues(forbiddenValues)) :> [-1, -1, -1, 4, 5, 6]
//
// Implementing this in Spark, we find two major issues:
@evanwill
evanwill / gitBash_windows.md
Last active April 26, 2024 03:58
how to add more utilities to git bash for windows, wget, make

How to add more to Git Bash on Windows

Git for Windows comes bundled with the "Git Bash" terminal which is incredibly handy for unix-like commands on a windows machine. It is missing a few standard linux utilities, but it is easy to add ones that have a windows binary available.

The basic idea is that C:\Program Files\Git\mingw64\ is your / directory according to Git Bash (note: depending on how you installed it, the directory might be different. from the start menu, right click on the Git Bash icon and open file location. It might be something like C:\Users\name\AppData\Local\Programs\Git, the mingw64 in this directory is your root. Find it by using pwd -W). If you go to that directory, you will find the typical linux root folder structure (bin, etc, lib and so on).

If you are missing a utility, such as wget, track down a binary for windows and copy the files to the corresponding directories. Sometimes the windows binary have funny prefixes, so

@voxxit
voxxit / USING-VAULT.md
Last active July 7, 2022 03:02
Consul + Vault + MySQL = <3 
git clone https://gist.github.com/dd6f95398c1bdc9f1038.git vault
cd vault
docker-compose up -d
export VAULT_ADDR=http://192.168.99.100:8200

Initializing a vault:

vault init