Skip to content

Instantly share code, notes, and snippets.

View iAnanich's full-sized avatar

Illia Ananich iAnanich

  • Ukraine
View GitHub Profile
@iAnanich
iAnanich / check.py
Created February 1, 2018 14:22
set of FieldsStorage classes which behaves like mutable mapping with preset set of keys
import typing
TypeOrNone = typing.Union[type, None]
def has_wrong_type(obj, expected_obj_type: TypeOrNone) -> bool:
"""
Checks if given `obj` object has not given `expected_obj_type` type. If
`expected_obj_type` is `None` than it will return `True`
@iAnanich
iAnanich / counter.py
Created February 1, 2018 14:23
counter with threshold
import typing
ThresholdBasis = typing.Union[int, None]
class Threshold:
def __init__(self, val: ThresholdBasis=None):
if val is None or val is 0:
@iAnanich
iAnanich / check.py
Created February 1, 2018 14:24
strongly typed function and sequence of functions
import typing
TypeOrNone = typing.Union[type, None]
def has_wrong_type(obj, expected_obj_type: TypeOrNone) -> bool:
"""
Checks if given `obj` object has not given `expected_obj_type` type. If
`expected_obj_type` is `None` than it will return `True`
@iAnanich
iAnanich / check.py
Created February 1, 2018 14:30
IterManager - helps with complex iteration cases - counters, excludes, context, type checks etc.
import typing
TypeOrNone = typing.Union[type, None]
def has_wrong_type(obj, expected_obj_type: TypeOrNone) -> bool:
"""
Checks if given `obj` object has not given `expected_obj_type` type. If
`expected_obj_type` is `None` than it will return `True`
@iAnanich
iAnanich / README.md
Created February 1, 2018 14:39
ScrapingHub tool kit

This gist presents part of my Scrapy NTK library: scraping_hub package.

Especialy this one might help you to communicate with ScrapingHub API. It provides set of classes build for fetching items from cloud.

Note: some of the imported modules are not included, but you can find them in my library on ni list of my gists.

@iAnanich
iAnanich / common.py
Last active April 27, 2018 13:02
Scrapy httpbin.org/anything spider for benchmarking purposes
# put it with settings.py and spiders folder
from urllib.parse import urlparse, urlencode, urlunparse
def update_query(url: str, query_dict: dict) -> str:
QUERY_INDEX = 4
components = urlparse(url)
query = components[QUERY_INDEX]
@iAnanich
iAnanich / async_scraper.py
Created July 23, 2018 22:04 — forked from carlosmcevilly/async_scraper.py
asyncio scraper
import asyncio
import aiofiles
import aiohttp
import logging
import re
import sys
import os
import lxml.html
@iAnanich
iAnanich / install-postgres-10-ubuntu.md
Created August 5, 2018 19:48 — forked from alistairewj/install-postgres-10-ubuntu.md
Install PostgreSQL 10 on Ubuntu

Install PostgreSQL 10 on Ubuntu

This is a quick guide to install PostgreSQL 10 - tested on Ubuntu 16.04 but likely can be used for Ubuntu 14.04 and 17.04 as well, with one minor modification detailed below.

(Optional) Uninstall other versions of postgres

To make life simple, remove all other versions of Postgres. Obviously not required, but again, makes life simple.

dpkg -l | grep postgres

Keybase proof

I hereby claim:

  • I am iananich on github.
  • I am iananich (https://keybase.io/iananich) on keybase.
  • I have a public key ASBYPJtCfPWT52GR3ljVWuOcyiqu2KubKlaUvofCPiJ2zAo

To claim this, I am signing this object:

@iAnanich
iAnanich / pipeline.py
Created October 25, 2019 23:59
AsyncIO Pipeline
import typing
import asyncio
class Layer:
class STATES:
IDLE = 1
RUNNING = 2
GOING_TO_STOP = 3
STOPPED = 4