Skip to content

Instantly share code, notes, and snippets.

View jacopotagliabue's full-sized avatar

Jacopo Tagliabue jacopotagliabue

View GitHub Profile
import bauplan
import boto3
from datetime import datetime
def main():
try:
# Initialize clients
bpln = bauplan.Client()
username = bpln.info().user.username
@jacopotagliabue
jacopotagliabue / python_dag_plan.py
Last active November 11, 2024 16:32
Graph inference for FaaS planning (sample, generated Python code, before Kuzu)
from typing import List, Optional, Set
class Column:
def __init__(self, name: str):
self.name = name
def __repr__(self):
return f"Column(name={self.name})"
@jacopotagliabue
jacopotagliabue / test_reading_arrow_from_sources.py
Created August 28, 2024 21:01
Benchmark time to get Arrow tables from IPC, local parquet, S3 parquet.
"""
Standalone script to write the same dataframe in different storage formats and then
measure the time it takes to read it into the user process.
Setup:
* create a virtualenv with pyarrow==17.0.0, boto3==1.35.7, posix-ipc==1.1.1
* make sure you have an S3 bucket and appropriate AWS credentials to write / read from it
@jacopotagliabue
jacopotagliabue / gist:3fc8aaf95ac2caf99e96bf5dadd7df05
Created August 12, 2024 23:30
Query 13 from TPC-H on parquet files stored in S3
SELECT
c_count,
count(*) AS custdist
FROM (
SELECT
c_custkey,
count(o_orderkey)
FROM
read_parquet('s3://bucket/customer.parquet') AS customer
LEFT OUTER JOIN
@jacopotagliabue
jacopotagliabue / hello_reclist.py
Created December 28, 2021 12:00
Running the CoveoCartRecList on CoveoDataset
from reclist.datasets import CoveoDataset
from reclist.recommenders.prod2vec import CoveoP2VRecModel
from reclist.reclist import CoveoCartRecList
if __name__ == "__main__":
# get the coveo data challenge dataset as a RecDataset object
coveo_dataset = CoveoDataset()
# re-use a skip-gram model from reclist to train a latent product space, to be used
@jacopotagliabue
jacopotagliabue / my_reclist.py
Created December 28, 2021 11:50
Creating a new RecList
class MyRecList(RecList):
@rec_test(test_type='stats')
def basic_stats(self):
"""
Basic statistics on training, test and prediction data
"""
from reclist.metrics.standard_metrics import statistics
return statistics(self._x_train,
self._y_train,
@jacopotagliabue
jacopotagliabue / Generating expressions from DBPedia facts
Last active June 7, 2019 21:53
WebPPL example for Medium post 'How Much is a Billion Dollar'
// max expression length (i.e. how many atomic facts can be considered)
var MAX_EXP_LENGTH = 5;
var TARGET_EXP = {
'q': 1000000000,
'd': 'target',
'u': '$'
}
var topK = 100;
@jacopotagliabue
jacopotagliabue / Evaluating the likelihood of competing hypotheses
Created February 9, 2019 00:46
WebPPL example for Medium post '(Robot) data scientists as a service'
// specify how much data to use
var N = 25; // you can try 25, 50, 100
var dataType = 'exp'; // linear, constant, exp
///fold: load observed data
print("Using " + N + " data points for data type: " + dataType);
var dataString = {
'exp': '[{"x": 8.528104691935328, "y": 369.4385603078311}, {"x": 5.800314416734446, "y": 56.17361673332166}, {"x": 6.957475968211479, "y": 124.0710506473754}, {"x": 9.481786398402916, "y": 715.0458977557804}, {"x": 8.735115980299934, "y": 426.23403384951274}, {"x": 3.045444240247178, "y": 8.356747238145411}, {"x": 6.9001768350511785, "y": 119.71324924833438}, {"x": 4.697285583404605, "y": 25.034180007597616}, {"x": 4.793562296412884, "y": 27.708924544629504}, {"x": 5.821197003876745, "y": 56.65940024103008}, {"x": 5.288087142321756, "y": 38.57248324353289}, {"x": 7.908547013925951, "y": 241.1127051063566}, {"x": 6.522075450293987, "y": 91.98603466343532}, {"x": 5.243350032985656, "y": 38.66132447348228}, {"x": 5.887726465490851, "y": 58.81299495311647}, {"x": 5.
@jacopotagliabue
jacopotagliabue / Generating mathematical expressions for symbolic regression
Created February 3, 2019 00:49
WebPPL example for Medium post '(Robot) data scientists as a service'
var unaryOps = ['log', 'round', 'sqrt'];
var binaryOps = ['add', 'sub', 'mul', 'div'];
var constants = ['x', 'z', 'u', 'q'];
var randomNumber = function() {
return [uniformDraw(_.range(10))];
}
var randomConstant = function() {
return [uniformDraw(constants)];
@jacopotagliabue
jacopotagliabue / A visual generative model for polygons
Created October 11, 2018 01:41
WebPPL tutorial for Medium post 'Fluid concepts and creative probabilities'
var drawLines = function(drawObj, lines, strokeW){
var line = lines[0];
drawObj.line(line[0], line[1], line[2], line[3], strokeW);
if (lines.length > 1) {
drawLines(drawObj, lines.slice(1), strokeW);
}
}
var drawSamples = function(sam){
drawLines(Draw(50, 50, true), sam);