wapiflapi

#!/bin/sh

source ./venv/bin/activate
"$@"

wapiflapi

# name: wapiflapi

function _git_branch_name
  command git symbolic-ref HEAD ^/dev/null | sed -e 's|^refs/heads/||'
end

function _git_repo_name
  basename (git rev-parse --show-toplevel ^/dev/null) ^/dev/null
end

wapiflapi

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import collections

import zmq

import ipc

wapiflapi

The purpose of this document is to explain the design decisions being taken for the multiprocessing and IPC architecture of binglide v2.

Synopsis

The main idea that this document ends up advocating is to have N workers for each type of service. In this scenario each worker knows how to do only one task. Those workers receive tasks from the broker who is responsable for splitting jobs into smaller tasks accoring to the cache. This endsup being hard

wapiflapi

Remarks about distributed computing.

• Right now we have the need to compute histogram data on large files with cache support. This can work great by having a reducer process agregating results from mappers. this works because we have one type of reducer and one type of mapper with a minimum map/reduce ratio among the workers.

• What happens if in the future we need something like A <- B <- C. This will require at least three processes with two of them waiting. We need to make

wapiflapi

OUTDATED

This document is now outdated, thanks for the feedback everyone. This is basically what I will be trying out next:

The rest of this document is available for reference.

Goals