Skip to content

Instantly share code, notes, and snippets.

View amotl's full-sized avatar

Andreas Motl amotl

  • $PYTHONPATH
View GitHub Profile
@amotl
amotl / cratedb-ecosystem-build-failures.sh
Last active December 7, 2024 05:07
Enumerate recent build failures in repositories relevant for CrateDB's ecosystem
# GitHub Actions Workflow Failures Digest for CrateDB's ecosystem
# https://github.com/crate/crate-clients-tools/issues/148
# Optionally use authentication when running into rate limits.
# It is an (here invalidated) personal access token (classic),
# using the "workflow" scope.
# GITHUB_TOKEN=ghp_00r4G0tGxLDT5RGOwWNw7tZhFnK5fT0uMfoo
# HTTPIE_OPTIONS="--auth-type bearer --auth ${GITHUB_TOKEN}"
@amotl
amotl / cratedb-sqlalchemy-active-servers.py
Created July 1, 2024 16:24
Access list of active server connections to CrateDB when using SQLAlchemy
# Connect using SQLAlchemy Core.
import sqlalchemy as sa
from pprint import pp
dburi = "crate://localhost:4200"
query = "SELECT country, mountain, coordinates, height FROM sys.summits ORDER BY country;"
def print_active_servers(connection: sa.engine.Connection):
print("Active Servers:", connection.connection.dbapi_connection.client.active_servers)
@amotl
amotl / cratedb_heap_exhaust_weird_error.py
Last active June 11, 2024 23:55
Attempt to trip a low-memory condition in CrateDB, resulting in a weird error message.
"""
Attempt to trip a low-memory condition in CrateDB, resulting in a weird error message
like `SQLParseException[ShardCollectContext for 0 already added]`.
This program tries to emulate the MLflow test case `test_search_runs_returns_expected_results_with_large_experiment`,
succeeded by a `DELETE FROM` table truncation operation.
Remark: It did not work out well. This program trips `OutOfMemoryError[Java heap space]`
right away. Please use the MLflow test case reproducer demonstrated at:
https://github.com/crate-workbench/mlflow-cratedb/issues/53#issuecomment-1927234463
@amotl
amotl / cratedb_regression_15488.py
Created January 30, 2024 00:38
CrateDB error: The assembled list of ParameterSymbols is invalid. Missing parameters.
"""
## About
Regression with CrateDB nightly-5.7.0-2024-01-26-00-02.
The assembled list of ParameterSymbols is invalid. Missing parameters.
-- https://github.com/crate/crate/issues/15488
## Setup
@amotl
amotl / update-from-merges.sql
Created December 16, 2023 22:05
Demo: SQL "UPDATE ... FROM" for upsert/merge operations. Reflecting Meltano's PostgreSQL data loader (target) adapter.
-- Demo: SQL "UPDATE ... FROM" for upsert/merge operations.
-- Reflecting Meltano's PostgreSQL data loader (target) adapter.
--
-- Usage:
--
-- psql postgresql://postgres@localhost:5432/ < update-from-merges.sql
-- crash --host http://crate@localhost:4200/ < update-from-merges.sql
-- Make a blank slate.
DROP TABLE IF EXISTS main;
@amotl
amotl / cratedb-vector-knn-exercise.sql
Created December 12, 2023 21:53
CrateDB: Exercise storing and searching by vectors using its "FLOAT_VECTOR" and "KNN_MATCH".
/**
* CrateDB: Exercise storing and searching by vectors using its "FLOAT_VECTOR" and "KNN_MATCH".
* The example uses euclidean distance for vector similarity search.
*
* Synopsis::
*
* docker run --rm -it --publish=4200:4200 crate/crate:nightly -Cdiscovery.type=single-node
* crash < cratedb-vector-knn-exercise.sql
*
* Resources:
@amotl
amotl / pgvector-vector-knn-exercise.sql
Created December 12, 2023 21:53
PostgreSQL/pgvector: Exercise storing and searching by vectors using its operators.
/**
* PostgreSQL/pgvector: Exercise storing and searching by vectors using its operators.
* The example uses euclidean distance for vector similarity search.
*
* Synopsis::
*
* docker run --rm -it --publish=5432:5432 --env "POSTGRES_HOST_AUTH_METHOD=trust" ankane/pgvector postgres -c log_statement=all
* psql postgresql://postgres@localhost < pgvector-vector-knn-exercise.sql
*
* Resources:
"""
pip install requests 'requests-cache<2'
"""
import os
import requests_cache
import typing as t
from langchain.embeddings import OpenAIEmbeddings
import typing as t
import pytest
def monkeypatch_pytest_notebook_treat_cell_exit_as_notebook_skip():
"""
Patch `pytest-notebook`, in fact `nbclient.client.NotebookClient`,
to propagate cell-level `pytest.exit()` invocations as signals
to mark the whole notebook as skipped.
import logging
import sys
import requests
from langchain.schema import Document
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import CrateDBVectorSearch
from langchain.embeddings import OpenAIEmbeddings