Skip to content

Instantly share code, notes, and snippets.

View arielshaqed's full-sized avatar
💭
🦎 lakeFSing FTW 🏁

Ariel Shaqed (Scolnicov) arielshaqed

💭
🦎 lakeFSing FTW 🏁
8 followers · 4 following
View GitHub Profile
@arielshaqed
arielshaqed / oc-tasks.md
Last active October 26, 2022 12:25
remaining tasks for LakeFSOutputCommitter. Basis for the execution plan, which is what you _really_ want to read.
  • Integrate to write single format (text?) in one mode (append?) M1
  • Implement abort M1
  • Testing (component test?) M1
  • Decide whether we need 1 branch per task? M1
  • Write all formats M2
    • Text
    • CSV
    • Parquet *
    • ORC *?
  • JSON
@arielshaqed
arielshaqed / prs-with-participants.gql
Last active November 4, 2021 16:39
Useful GitHub GraphQL
{
repository(owner: "treeverse", name: "lakefs") {
pullRequests(first: 100, states: CLOSED, orderBy: {field: CREATED_AT, direction: DESC}) {
edges {
node {
title
author {
login
}
mergedAt
@arielshaqed
arielshaqed / Dockerfile
Last active May 24, 2021 10:46
Dockerfile for debug minio container that outputs logs
FROM minio/mc:latest AS mc
FROM minio/minio:edge
COPY --from=mc /bin/mc /bin/
ENTRYPOINT ["sh", "-c", "minio server /data & (sleep 1; until mc alias set root http://localhost:9000 ${MINIO_ROOT_USER:-minioadmin} ${MINIO_ROOT_PASSWORD:-minioadmin}; do sleep 1; done; mc admin trace root)"]
@arielshaqed
arielshaqed / log4j.properties
Created April 21, 2021 13:12
log4j.properties for debugging clients. Place inside clients/spark/examples/src/main/resources/
# Define the root logger with appender file
log = /tmp/log4j
log4j.rootLogger = console, FILE
# Define the file appender
log4j.appender.FILE=org.apache.log4j.FileAppender
log4j.appender.FILE.File=${log}/log.out
# Define the layout for file appender
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
@arielshaqed
arielshaqed / concurrent-ops-in-alternatives.md
Last active March 16, 2021 15:30
Description of concurrent operations by underlying object store (mostly S3) in lakeFS alternatives

How concurrent are lakeFS alternatives?

DeltaLake

DeltaLake concurrency uses LogStore. On S3 it is single-writer concurrency (a writer is a Spark cluster):

Delta Lake supports concurrent reads from multiple clusters, but concurrent writes to S3 must originate from a single Spark driver in order for Delta Lake to provide transactional guarantees.

@arielshaqed
arielshaqed / run-with-httpie
Created March 8, 2021 13:52
redirects take headers with them :-(
ariels@redqueen:~/dev/Tests/manualweb$ http -v --follow get localhost:8888/redirect/foo authorization:yes-i-can
GET /redirect/foo HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: localhost:8888
User-Agent: HTTPie/2.2.0
authorization: yes-i-can
@arielshaqed
arielshaqed / magit-origin-master.el
Created February 2, 2021 07:54
add "r o" and "d o" to magit: diff and rebase with origin/master
;;; Define "r o" (an "o" suffix) for "magit rebase onto origin/master"
(require 'magit)
(transient-define-suffix as-magit-rebase-onto-origin-master (args)
"Rebase the current branch onto origin/master.
TODO(ariels): Make it better and also document it."
:if 'magit-get-current-branch
:description "origin/master"
(interactive (list (magit-rebase-arguments)))
(magit-git-rebase "origin/master" args))
@arielshaqed
arielshaqed / sim.go
Last active January 12, 2021 11:44
simulate range sizes with a log-normal path size distribution
package main
import (
"fmt"
"math"
"time"
"golang.org/x/exp/rand"
"gonum.org/v1/gonum/stat/distuv"
@arielshaqed
arielshaqed / method-keys.ts
Last active October 26, 2022 12:23
Extract keys of methods of TypeScript type
// https://www.typescriptlang.org/play/#code/C4TwDgpgBAshwAsD2ATA0hEBnA8gJwDkIA3CPAHgBUA+KAXigG8AoKNqAbTSgEsA7KAGtMSAGZRKAXQBcErpKgQAHsAh8UWKADEArnwDGwHkgEB+KN1l8SZANzMAvs1CRY8ZOkxYqtBnESoGNj4RKQUNBzCIGISkvbM-Kp4ogCG+tAAkkys7HxWOgC2AEZ2OWyiABQp+cVkAJSyWMB4-ADm9uxQrRUNUMRIPCj2Ti7QWvRuAZ7Y5BnUtmwA9ItQAESiq1AAPmutq8xAA
type MethodKeysOrNever<T> = {
[K in keyof T]: T[K] extends Function ? K : never;
}
type MethodKeys<T> = MethodKeysOrNever<T>[keyof T];
@arielshaqed
arielshaqed / Happy New Decade, 2267 different ways!
Created January 1, 2020 08:47
Happy New 2020!
((10*(9*8*(7-6)-5)+4)*3-2)*1
((10*(9*8*(7-6)-5)+4)*3-2)/1
((10*(9*8-(7-6)*5)+4)*3-2)*1
((10*(9*8-(7-6)*5)+4)*3-2)/1
((10*(9*8/(7-6)-5)+4)*3-2)*1
((10*(9*8/(7-6)-5)+4)*3-2)/1
((10*9+8)*7-(6+5*4/3))*(2+1)
((10*9+8)*7-6-5*4/3)*(2+1)
((10*9+8)/7+6)*(5*4*(3+2)+1)
((10*9+8*(7*(6+5)-4))*3-2)*1