Skip to content

Instantly share code, notes, and snippets.

View kevcodez's full-sized avatar
🚀

Kevin Grüneberg kevcodez

🚀
View GitHub Profile
@kevcodez
kevcodez / httpClient.ts
Last active January 24, 2024 02:11
Native Node.js HTTP Client with retries, proxy support, timeouts
import { ProxyAgent } from 'undici';
const SAFE_HTTP_METHODS = ['GET', 'HEAD', 'OPTIONS'];
const IDEMPOTENT_HTTP_METHODS = SAFE_HTTP_METHODS.concat(['PUT', 'DELETE']);
const HTTP_STATUS_TO_RETRY = [408, 429, 500, 501, 502, 503, 504];
const DEFAULT_TIMEOUT = 20_000;
const retryDenyList = new Set([
'ENOTFOUND',
async timestampCreateSeries({
key,
retentionInSeconds,
labels,
}: {
key: string;
retentionInSeconds: number;
labels: string[][];
}) {
const args: (string | number)[] = [key, 'RETENTION', retentionInSeconds, 'DUPLICATE_POLICY', 'LAST'];
@kevcodez
kevcodez / document.json
Created October 7, 2022 15:16
medium_mongodb_performance_sample_document
{
"_id": "1232312323",
"name": "Apple",
"isin": "US123456890",
"marketCap": 1233413233.123,
"ipoDate": "2000-01-01",
"lastPrice": 123.23,
"dividends": [
{
"date": "2022-01-01",
@kevcodez
kevcodez / query.sh
Created October 7, 2022 15:15
medium_mongodb_performance_include_field
db.assets.find({
marketCap: { $gt: 10000 }
}, {
projection: { isin: 1 }
})
@kevcodez
kevcodez / client_options.ts
Created October 7, 2022 15:04
medium_mongodb_performance_client_options
const client = await MongoClient.connect('mongo-url', {
// perf optimization - https://github.com/mongodb/node-mongodb-native/releases/tag/v4.3.0
enableUtf8Validation: false,
});
@kevcodez
kevcodez / pooling.ts
Created October 7, 2022 15:03
medium_mongodb_performance_pooling
const client = await MongoClient.connect('mongodb-url', {
minPoolSize: 50,
maxPoolSize: 250,
});
@kevcodez
kevcodez / parallel.ts
Created October 7, 2022 15:02
medium_mongodb_performance_parallel_fetching
async function getQuotesForPeriod(
assetIdentifier: string,
from: Date,
to: Date
): Promise<Quote[]> {
const quotesCollection = await getQuotesCollection();
/**
* When fetching multiple years of data, queries are fast, but retrieval not so much.
* To improve query performance, we split the interval into a few chunks and query/fetch the data in parallel,
@kevcodez
kevcodez / mongo.ts
Created October 7, 2022 15:02
medium_mongodb_performance_node_snappy
const client = await MongoClient.connect(mongoConfig().mongoUrl, {
compressors: 'snappy', // make sure Snappy is installed
});
@kevcodez
kevcodez / query.sh
Created October 7, 2022 14:50
medium_mongodb_performance_read_pref
db.assets
.find({ ipoDate: { $gt: ISODate('2022-01-01') } })
.readPref('secondaryPreferred')
@kevcodez
kevcodez / query.sh
Created October 7, 2022 14:49
medium_mongodb_performance_merge
db.holdings.aggregate([
{
$group: {
_id: '$portfolio',
assetIdentifiers: {
$addToSet: '$asset.identifier',
},
},
},
{