hanx11

# maximum capability of system
user@ubuntu:~$ cat /proc/sys/fs/file-max
708444

# available limit
user@ubuntu:~$ ulimit -n
1024

# To increase the available limit to say 200000
user@ubuntu:~$ sudo vim /etc/sysctl.conf

hanx11

#!/usr/bin/python
import datetime
import logging
import threading
import weakref

import tornado
import tornado.gen
import pika

hanx11

HTTP Streaming (or Chunked vs Store & Forward)

The standard way of understanding the HTTP protocol is via the request reply pattern. Each HTTP transaction consists of a finitely bounded HTTP request and a finitely bounded HTTP response.

However it's also possible for both parts of an HTTP 1.1 transaction to stream their possibly infinitely bounded data. The advantages is that the sender can send data that is beyond the sender's memory limit, and the receiver can act on

hanx11

#!/usr/bin/env python

"""A Tornado example of RPC.

Designed to work with rpc_server.py as found in RabbitMQ Tutorial #6:
http://www.rabbitmq.com/tutorials/tutorial-six-python.html

Some code is borrowed from pika's tornado example.
"""

hanx11

Asynchronous programming with Tornado

Asynchronous programming can be tricky for beginners, therefore I think it's useful to iron some basic concepts to avoid common pitfalls.

For an explanation about generic asynchronous programming, I recommend you one of the [many][2] [resources][3] [online][4].

I will focus on solely on asynchronous programming in [Tornado][1]. From Tornado's homepage:

hanx11

from time import sleep
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.web import Application, asynchronous, RequestHandler
from multiprocessing.pool import ThreadPool

_workers = ThreadPool(10)

def run_background(func, callback, args=(), kwds={}):
    def _callback(result):

hanx11

import numpy as np
from bokeh.plotting import figure, show, output_file
from bokeh.models import ColumnDataSource, LabelSet

num_vars = 9

theta = np.linspace(0, 2*np.pi, num_vars, endpoint=False)
# rotate theta such that the first axis is at the top
theta += np.pi/2

hanx11

# -*- coding:utf-8 -*-

from datetime import datetime


def read_many_lines(fp, buffer_size=10000):
    """
    :param fp:
    :param buffer_size: default 10000
    :return:

hanx11

# -*- coding:utf-8 -*-


def fib(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

hanx11

# -*- coding:utf-8 -*-

import codecs
from datetime import datetime

source_file = 'big_gbk_file.csv'
target_file = 'big_utf8_file.csv'
block_size = 10000 * 1024  # or some other, desired size in bytes