Skip to content

Instantly share code, notes, and snippets.

View dataders's full-sized avatar

Anders dataders

View GitHub Profile
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@dataders
dataders / dbt-synapse_on_M1_macbook.md
Created March 16, 2023 18:11
brain dump on working with dbt, MSODBC driver on an M1 macbook

Setup dbt-sqlserver / dbt-synapse on M1

Install pyodbc

Info gotten from here

brew install unixodbc
export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/2.3.9_1/lib"
export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/2.3.9_1/include"

12 Principles of Agile Software Development

background

source: agilemanifesto.org/principles

Writing these up in a numbered, markdown-friendly list, because I'm nitpicky

principles

@dataders
dataders / substrait_is_now.md
Created January 24, 2023 18:31
thinking out loud about substrait

working title

Codd, Chomsky, McKinney, and Wickham walk into a bar… (maybe Chamberlain and Wittgenstein should also be included?)

background

  • the dream of substrait is true separation b/w query engines and transformation APIs
  • previously, particular APIs would give better performance due to their inextricable link to the architecture of the underlying compute engine
  • if the above benefit is removed, folks could use the API with which they are most familiar
  • given this, we could see an industry consolidation around the “best” transformation API.

Data + AI Summit Call for Presentations

Data teams overcome challenges by building data pipelines, using advanced analytics and developing machine learning models. These challenges often span across disciplines to incorporate multiple data types, technologies, and tools — this is the driver of data lakehouse adoption.

Are you a practitioner solving data, analytics and AI challenges using Apache Spark™, Delta Lake, MLflow, TensorFlow, PyTorch, Scikit-learn, BI and SQL analytics, real time streaming, deep learning and machine learning frameworks? If so, we invite you to share your experience with our global Summit community.

Draft your proposal for a 15-minute lightning talk, 40-minute session or 90-minute technical deep dive about how you are simplifying data, analytics and AI challenges. Share your expertise with the largest gathering of data and AI professionals.

Please visit the Data + AI Summit 2023 website to learn more about this year's Call for Presen

@dataders
dataders / bulk_delete_users_databricks_account.ipynb
Created November 4, 2022 20:05
"Bulk" DELETE users from Databricks Account
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

NARRATOR NATHAN: ... So what was Angela actually doing here?

NATHAN: I'm just so confused. You won't let me have the experience I want. But I can't even tell if you're invested in this yourself.

ACTRESS PLAYING ANGELA: I'm invested as I can be but this all sort of absurd right? Swapping out the kids through the window, aging them three years at a time.

NATHAN: It won't work if you don't try. You have to want it to work. Are you mad that I... have been caught up in my work and haven't been prioritizing our family life --

ACTRESS: Nathan! What family life?

gource \
--s 0.1 \
--auto-skip-seconds 0.05 \
--key \
--title "mart_core: 2021" \
--highlight-users \
--max-file-lag 0.1 \
--font-scale 1.5 \
--filename-time 2 \
-1920x1080 -o - \
gource \
--s 0.1 \
--auto-skip-seconds 0.05 \
--key \
--title "dbt: the 'beta' era" \
--highlight-users \
--max-file-lag 0.1 \
--caption-file caption_versions.psv \
--caption-duration 5 \
--caption-size 80 \