Skip to content

Instantly share code, notes, and snippets.

Avatar
🏠
https://tradingstrategy.ai

Mikko Ohtamaa miohtama

🏠
https://tradingstrategy.ai
View GitHub Profile
@miohtama
miohtama / indexnow.py
Created Jan 11, 2022
Bing IndexNow URL submitter for Python
View indexnow.py
"""Use IndexNow API to send the latest updates to Bing.
Ensures Bing picks up the pages that should be fresh and interesting.
Do not wait them picked up by the natural crawler process.
To submit the pages you need to have a Bing known file at the root of your website.
https://www.indexnow.org/faq
IndexNow will also submit the Yandex.
@miohtama
miohtama / find-long-queries.sql
Created Dec 2, 2021
PostgreSQL find long queries
View find-long-queries.sql
SELECT
application_name,
pid,
now() - pg_stat_activity.query_start AS duration,
query,
state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';
@miohtama
miohtama / pgdash-loop.sh
Created Dec 2, 2021
Bash script for pgmetrics + pgdash reporting loop
View pgdash-loop.sh
#!/bin/bash
#
# Monitor loop for TimescaleDB using pgdash
# - TimescaleDB runs in a Docker, exposed over TCP/IP socket 127.0.0.1:5555
# - pgdash SaaS offering used https://pgdash.io
# - run the sample collection loop every 30 seconds
#
#
set -e
set -x
@miohtama
miohtama / dramatiq-task-queue.py
Last active Nov 6, 2021
Throttling background task queue using Dramatiq - wait until more workers are freed
View dramatiq-task-queue.py
import time
from typing import Dict, Optional
from dramatiq import Message, Broker
from dramatiq.results import ResultMissing
class TaskQueue:
"""A task queue using Dramatiq background task framework.
@miohtama
miohtama / bsh-notes.bash
Created Nov 4, 2021
Binance Smart Chain notes
View bsh-notes.bash
./geth_linux \
--config ./config.toml \
--datadir ./data/bsc \
--cache 8000 \
--rpc.allow-unprotected-txs \
--txlookuplimit 0 \
--http.port 9545 \
--http.addr 127.0.0.1 \
--http.vhosts=* \
View erdpy-failed-transaction.py
from erdpy.accounts import Account, Address
from erdpy.proxy import ElrondProxy
from erdpy.transactions import BunchOfTransactions
from erdpy.transactions import Transaction
from erdpy.wallet import signing
proxy = ElrondProxy("https://devnet-gateway.elrond.com")
sender = Account(pem_file="test-wallet.pem")
sender.sync_nonce(proxy)
View colormapper.sol
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.7;
pragma abicoder v2;
contract ArrayTest {
function colourToString(uint r, uint g, uint b) private pure returns(string memory) {
bytes memory alphabet = "0123456789abcdef";
@miohtama
miohtama / fast-web3-py-event-decoder.py
Created Aug 19, 2021
Fast event fetcher and decoder for web3.py
View fast-web3-py-event-decoder.py
"""Below is an example for faster JSON-RPC event fetcher.
It skips a lot of steps, like converting raw binary values to corresponding numbers (float, ints),
looking up ABI labels and building `AttributedDict` object.
The resulting Event dictitionary is generated faster, but harder to use and you need to know what you are doing.
"""
def _fetch_events_for_all_contracts(
@miohtama
miohtama / write-parquet-larger-than-ram.py
Created Jul 14, 2021
Writing Parquet files incrementally from Python
View write-parquet-larger-than-ram.py
with pq.ParquetWriter(
fname,
Candle.to_pyarrow_schema(small_candles),
compression='snappy',
allow_truncated_timestamps=True,
version='2.0', # Highest available schema
data_page_version='2.0', # Highest available schema
) as writer:
def reset_data():
@miohtama
miohtama / ethereum_address_type_for_sqlalchemy.py
Created Jun 27, 2021
Ethereum address to for SQLAlchemy
View ethereum_address_type_for_sqlalchemy.py
class EthereumAddress(types.TypeDecorator):
"""SQLAlchemy class to store Ethereum addresses as binary in SQL database.
Ethereum address is 160 bits, the last bytes of 256 bit public key of the address's signer.
Ethereum address checksum is encoded in the case of hex letters.
We skip any Ethereum address checksum checks, as they slow down large data processing too much.
Any returned address data will be in lowercase.
"""