databill86

# Requirement: pip install tweepy
import tweepy

# Credentials go here (generate at: https://apps.twitter.com)
auth = tweepy.OAuthHandler('consumer_key', 'consumer_secret')
auth.set_access_token('access_token', 'access_token_secret')

# Connect to Twitter
api = tweepy.API(auth)

databill86

"""
Minimal character-level Vanilla RNN model. Written by Andrej Karpathy (@karpathy)
BSD License
"""
import numpy as np

# data I/O
data = open('input.txt', 'r').read() # should be simple plain text file
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)

databill86

import pickle
import threading
from Queue import Queue

import time
from bson import InvalidDocument

from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

databill86

# Task primitives, allows pipeline execution using celery
@app.task
def dmap(it, callback, final=None):
    # Map a callback over an iterator and return as a group
    callback = subtask(callback)

    # Hack for mapping a chain to values, due to a bug where args are not copied in group creation
    if isinstance(callback, chain):
        if final:
            raise ValueError('task_processor: Cannot run reducer for dmap excecuted with a chain.')

databill86

from flask import Flask, request, jsonify, json, abort
from flask_cors import CORS, cross_origin

import pandas as pd

app = Flask(__name__)

cors = CORS(app)
app.config['CORS_HEADERS'] = 'Content-Type'

databill86

Requirements

• no upfront installation/agents on remote/slave machines - ssh should be enough
• application components should use third-party software, e.g. HDFS, Spark's cluster, deployed separately
• configuration templating
• environment requires/asserts, i.e. we need a JVM in a given version before doing deployment
• deployment process run from Jenkins

Solution

databill86

# Copyright (c) 2019-present, Thomas Wolf.
# All rights reserved. This source code is licensed under the MIT-style license.
""" A very small and self-contained gist to train a GPT-2 transformer model on wikitext-103 """
import os
from collections import namedtuple
from tqdm import tqdm
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from ignite.engine import Engine, Events

databill86

import matplotlib
import matplotlib.pyplot as plt
import pandas as pd
import squarify

# qualtities plotted
# squarre area is the town surface area (superf)
# color scale is the town population in 2011 (p11_pop)

# read data from csv file

databill86

Moved to git repository: https://github.com/denji/nginx-tuning

NGINX Tuning For Best Performance

For this configuration you can use web server you like, i decided, because i work mostly with it to use nginx.

Generally, properly configured nginx can handle up to 400K to 500K requests per second (clustered), most what i saw is 50K to 80K (non-clustered) requests per second and 30% CPU load, course, this was 2 x Intel Xeon with HyperThreading enabled, but it can work without problem on slower machines.

You must understand that this config is used in testing environment and not in production so you will need to find a way to implement most of those features best possible for your servers.

databill86

Table Of Content

Skip to the relevant sections if needed.

• 2-min tutorial to do it the quick-and-dirty-way
• Concepts for resolving Git conflicts
• Setting up different editors / tool for using git mergetool
  • Finding out what mergetool editors are supported
• mergetool simple code example for vimdiff
  • Resolving conflict from a Git pull
• Other great references and tutorials