samukasmk

Simple queues reprocessing

Executing

python simple_queues.py

Example

samukasmk

Proof of Concept: Implementing MongoEngine lib with support for multi database connections

Reason:

Currently MongoEngine library only support a single database connection, managing collections by Document classes in global scope.

It has a support to change database connections by switch_db but it's made by a global class attribute called ._meta['db_alias']

When you define many sub ReferenceFields in a hierarchical tree, only first Document object has its ._meta['db_alias'] atttibute changed, not working to query Documents that implement other documents.

samukasmk

import asyncio

async def create_async_results(total_items: int):
    for r in range(total_items):
        await asyncio.sleep(1)
        yield r

async def example_of_async_for_loop():
    async for item in create_async_results(10):
        print(item)

samukasmk

import asyncio
from random import randint


async def execute_your_task(task_number: int):
    """ Specific task to run in parallel """
    
    # display begin of parallel task in console
    print('-> starting parallel task:', task_number)
    

samukasmk

import asyncio
from random import randint


async def execute_your_task(task_number: int):
    """ Specific task to run in parallel """
    
    # display begin of parallel task in console
    print('-> starting parallel task:', task_number)
    

samukasmk

Installing

pip install confluent-kafka

Initialization

It's important to declare producer and consumer objects in global scope for better performance

samukasmk

ssh -o "ProxyCommand=nc -X 5 -x 127.0.0.1:9050 %h %p" root@your-domain.com

samukasmk

This is a easy example of how to profiling memory usage by memory_profiler library in 5 steps:

1.) Install memory_profiler libraries

pip install -U memory_profiler matplotlib

2.) Insert @profile decorator in your script

samukasmk

def replace_dynamic_variable(element):
    # If the element is a string, replace "{variable_dynamic}" with "hacked"
    if isinstance(element, str):
        element = element.replace("{variable_dynamic}", "hacked")
    
    # If the element is a dictionary, iterate through each key-value pair
    elif isinstance(element, dict):
        for key, value in element.items():
            element[key] = replace_dynamic_variable(value)

samukasmk

import traceback

# define some sample function with a nonexisting var
def my_function():
    some_var = nonexisting_var

# create dict to store all raised exceptions
raised_exceptions = {}

# for 3 times execute a broken function (that generates execptions)