Skip to content

Instantly share code, notes, and snippets.

View AdityaSoni19031997's full-sized avatar
🎯
Focusing

Aditya Soni AdityaSoni19031997

🎯
Focusing
View GitHub Profile
@nymous
nymous / README.md
Last active May 6, 2024 23:01
Logging setup for FastAPI, Uvicorn and Structlog (with Datadog integration)

Logging setup for FastAPI

This logging setup configures Structlog to output pretty logs in development, and JSON log lines in production.

Then, you can use Structlog loggers or standard logging loggers, and they both will be processed by the Structlog pipeline (see the hello() endpoint for reference). That way any log generated by your dependencies will also be processed and enriched, even if they know nothing about Structlog!

Requests are assigned a correlation ID with the asgi-correlation-id middleware (either captured from incoming request or generated on the fly). All logs are linked to the correlation ID, and to the Datadog trace/span if instrumented. This data "global to the request" is stored in context vars, and automatically added to all logs produced during the request thanks to Structlog. You can add to these "global local variables" at any point in an endpoint with `structlog.contextvars.bind_contextvars(custom

@skrawcz
skrawcz / my_functions.py
Last active November 13, 2021 06:45
Hamilton with a single value node
# --- my_functions.py
import pandas as pd
def avg_3wk_spend(spend: pd.Series) -> pd.Series:
"""Rolling 3 week average spend."""
return spend.rolling(3).mean()
def spend_per_signup(spend: pd.Series, signups: pd.Series) -> pd.Series:
@EricPostMaster
EricPostMaster / star_wars_ep4_network.ipynb
Last active May 5, 2021 15:49
Star Wars Episode 4 Character Network
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@cccntu
cccntu / csv.py
Created February 8, 2021 11:58
python mmap to concatenate csv files
❯ rm out.csv
❯ cat 1.py
from glob import glob
import mmap
files = glob("data/*")
files.sort(key=lambda x: int(x.split("/")[-1].split(".")[0]))
write_f = open("out.csv", "w+b")
@tykurtz
tykurtz / grokking_to_leetcode.md
Last active May 7, 2024 05:50
Grokking the coding interview equivalent leetcode problems

GROKKING NOTES

I liked the way Grokking the coding interview organized problems into learnable patterns. However, the course is expensive and the majority of the time the problems are copy-pasted from leetcode. As the explanations on leetcode are usually just as good, the course really boils down to being a glorified curated list of leetcode problems.

So below I made a list of leetcode problems that are as close to grokking problems as possible.

Pattern: Sliding Window

@pvsune
pvsune / timeout.py
Created May 31, 2020 15:03
A helper function to limit execution of a function in seconds.
import logging
from multiprocessing import Pool, TimeoutError
logging.basicConfig(level=logging.INFO)
def timeout(func, args=None, kwds=None, timeout=10):
"""Call function and wait until timeout.
@pvsune
pvsune / concurrent.kafka.consumer.py
Last active April 29, 2024 07:25
A multiprocess multithreaded Kafka consumer
#!/usr/bin/env python
import logging
import os
import threading
import time
from multiprocessing import Process
from queue import Queue
from confluent_kafka import Consumer
@AdityaSoni19031997
AdityaSoni19031997 / file_streaming_pytorch.py
Created April 21, 2020 02:19
In this gist i have tried to explain a very smart way of loading datasets by streaming them from bytes into PyTorch; It can be achieved in multiple ways, but here my focus was confined to David's idea of streaming records from a bytes file;
import torch
import io
import pandas as pd
import gc
import numpy as np
import transformers
'''
Original Code Author [@dlibenzi](https://github.com/dlibenzi)
from PIL import Image
import numpy as np
import hashlib
import os
import sys
import torch
import torch_xla.utils.tf_record_reader as tfrr
a = """
image/class/label tensor([82])