Skip to content

Instantly share code, notes, and snippets.

View yeus's full-sized avatar

Thomas Meschede yeus

View GitHub Profile
# flake.nix
{
inputs = {
flake-utils.url = "github:numtide/flake-utils";
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
/*poetry2nix = {
url = "github:nix-community/poetry2nix";
inputs.nixpkgs.follows = "nixpkgs";
};*/
};
@yeus
yeus / automated_blog_writing.md
Last active May 30, 2023 00:26
automated_blog_writing.md

May 29, 2023

An example of using pydoxtools for LLM based article writing and file/directory-based information retrieval

When Python Libraries Talk: Pydoxtools Writes Blog Post About Itself in Fewer Than 100 Lines of Code!

We wrote the first blogpost about Pydoxtools completely automatically - using Pydoxtools! Here is the code how this was done.

:::info This was the prompt used to generate the first blogpost:

@yeus
yeus / sql_nlp_ingesting.py
Created May 10, 2023 16:01
doing NLP calculations on SQL db
"""
In this example we will extract information from
an SQL database and inject it into chroma db.
"""
import logging
from pathlib import Path
import chromadb
import dask
@yeus
yeus / test.py
Last active May 20, 2023 20:17
playing with pydoxtools
"""
In this example we will extract information from
an SQL database and inject it into chroma db.
"""
import logging
import uuid
from pathlib import Path
import chromadb
"""
In this example we will extract information from
an SQL database and inject it into chroma db.
"""
import logging
from pathlib import Path
import chromadb
import dask
@yeus
yeus / spacy_trf_vecs.py
Created April 5, 2022 21:06
spacy > 3.0 transformers contextual vectors pipeline component
from spacy.language import Language
from spacy.tokens import Doc
@Language.factory('trf_vectors')
class TrfContextualVectors:
"""
Spacy pipeline which add transformer vectors to each token based on user hooks.
https://spacy.io/usage/processing-pipelines#custom-components-user-hooks
https://github.com/explosion/spaCy/discussions/6511
@yeus
yeus / UniqueDynamicPriorityQueue.py
Created February 23, 2022 21:18
UniqueDynamicPriorityQueue - dynamically modifiable unique priority task queue
class UniqueDynamicPriorityQueue(asyncio.Queue):
"""A subclass of Queue; retrieves entries in priority order (lowest first).
Entries are typically tuples of the form: (priority number, data).
Entries can only be unique and when the same data gets pushed twice,
it will replace the old one.
Entries can be modified hence this this a "dynamic" priority queue.
@yeus
yeus / fastapi_spa_server.py
Created February 19, 2022 17:58
Single Page Application Server based on fastapi/starlette.
from fastapi import FastAPI
from fastapi.responses import FileResponse
from fastapi.staticfiles import StaticFiles
app = FastAPI()
STATIC_FILES_DIR = "./static/"
index_file = FileResponse(
STATIC_FILES_DIR + "index.html",
@yeus
yeus / rate_limiter.py
Last active December 2, 2021 19:25
async rate limiting function using token bucket algorithm
"""
author: Thomas Meschede
license: MIT
"""
def rate_limit_burst(max_calls: float, interval: float):
"""
Rate-limits the decorated function locally, for one process. Using Token Bucket algorithm.
max_calls: maximum number of calls of function in interval
import { Component, defineAsyncComponent, defineComponent, h } from 'vue'
export function hydrateNever(componentOrFactory: Component): Component {
return makeHydrationBlocker(componentOrFactory, {
beforeCreate() {
this.never = true
},
})
}