Skip to content

Instantly share code, notes, and snippets.

@antonydevanchi
antonydevanchi / kolduvachestvo.sh
Created July 27, 2021 03:32
Как читать SMS с блядского USB-модема «Yota 4G LTE» через консоль
#!/usr/bin/env bash
# Ahalai-mahalai
FLAG="GET_RCV_SMS_LOCAL"
PAGE="1"
# Krible-krable
XMLRESPONSE=$(curl -sL -XPOST 'http://10.0.0.1/xml_action.cgi?method=set&module=duster&file=message' \
@wshayes
wshayes / python_example.py
Created December 24, 2019 18:46
[python multiprocessing example] writing to file from a queue #python #multiprocessing
# https://stackoverflow.com/a/13530258/886938
import multiprocessing as mp
import time
fn = 'c:/temp/temp.txt'
def worker(arg, q):
'''stupidly simulates long running process'''
start = time.clock()
@snakers4
snakers4 / parse_cc_index.py
Last active September 14, 2023 20:00
Plain common crawl pre-processing
import gc
import gzip
import time
import json
import shutil
import os,sys
import tldextract
import collections
import pandas as pd
from tqdm import tqdm
@wwek
wwek / httpsproxy.go
Created October 29, 2017 05:41
https proxy in golang
// https://medium.com/@mlowicki/http-s-proxy-in-golang-in-less-than-100-lines-of-code-6a51c2f2c38c
// #!/usr/bin/env bash
// case `uname -s` in
// Linux*) sslConfig=/etc/ssl/openssl.cnf;;
// Darwin*) sslConfig=/System/Library/OpenSSL/openssl.cnf;;
// esac
// openssl req \
// -newkey rsa:2048 \
// -x509 \
@sirodoht
sirodoht / migrate-django.md
Last active April 20, 2024 09:52
How to migrate Django from SQLite to PostgreSQL

How to migrate Django from SQLite to PostgreSQL

Dump existing data:

python3 manage.py dumpdata > datadump.json

Change settings.py to Postgres backend.

Make sure you can connect on PostgreSQL. Then:

@racitup
racitup / html_to_text.py
Last active July 29, 2021 09:43
Extract text from html in python using BeautifulSoup4
from bs4 import BeautifulSoup, NavigableString, Tag
def html_to_text(html):
"Creates a formatted text email message as a string from a rendered html template (page)"
soup = BeautifulSoup(html, 'html.parser')
# Ignore anything in head
body, text = soup.body, []
for element in body.descendants:
# We use type and not isinstance since comments, cdata, etc are subclasses that we don't want
if type(element) == NavigableString:
@valyala
valyala / README.md
Last active April 19, 2024 13:00
Optimizing postgresql table for more than 100K inserts per second

Optimizing postgresql table for more than 100K inserts per second

  • Create UNLOGGED table. This reduces the amount of data written to persistent storage by up to 2x.
  • Set WITH (autovacuum_enabled=false) on the table. This saves CPU time and IO bandwidth on useless vacuuming of the table (since we never DELETE or UPDATE the table).
  • Insert rows with COPY FROM STDIN. This is the fastest possible approach to insert rows into table.
  • Minimize the number of indexes in the table, since they slow down inserts. Usually an index on time timestamp with time zone is enough.
  • Add synchronous_commit = off to postgresql.conf.
  • Use table inheritance for fast removal of old data:
@neu5ron
neu5ron / valid_domain_name_regex
Last active July 2, 2023 10:40
Valid domain name regex including internationalized domain name
domain_regex = r'(([\da-zA-Z])([_\w-]{,62})\.){,127}(([\da-zA-Z])[_\w-]{,61})?([\da-zA-Z]\.((xn\-\-[a-zA-Z\d]+)|([a-zA-Z\d]{2,})))'
#Python
domain_regex = '{0}$'.format(domain_regex)
valid_domain_name_regex = re.compile(domain_regex, re.IGNORECASE)
self.domain_name = self.domain_name.lower().strip().encode('ascii')
if re.match(valid_domain_name_regex, self.domain_name ):
return True
else:
return False