jmilagroso

from airflow.hooks.postgres_hook import PostgresHook
from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults
from datetime import datetime, timedelta
from os import environ
import csv


class DataSourceToCsvOperator(BaseOperator):
    """

jmilagroso

Status for the jail: nginx-4xx
|- Filter
|  |- Currently failed:	2
|  |- Total failed:	8
|  `- File list:	/var/log/nginx/access.log
`- Actions
   |- Currently banned:	166
   |- Total banned:	168
   `- Banned IP list:	103.141.136.70 103.145.13.24 104.192.2.74 104.197.39.24 106.14.214.61 106.52.4.195 108.53.119.33 111.7.100.21 113.20.17.112 118.123.1.34 125.35.118.6 128.14.133.58 128.14.134.134 128.14.134.170 128.199.139.130 128.199.193.3 128.199.25.131 129.204.116.107 13.124.222.242 13.209.28.104 13.233.73.212 13.77.214.37 13.79.160.159 134.122.34.26 134.175.144.96 138.197.89.132 138.68.137.21 139.59.25.246 142.11.242.11 142.93.207.236 142.93.230.99 143.198.58.47 145.40.80.163 147.139.32.106 149.202.8.122 154.16.51.85 157.230.247.213 157.55.39.116 158.101.42.3 159.65.144.176 159.89.55.1 165.227.61.152 165.232.129.207 167.99.151.56 168.62.189.211 170.130.126.96 172.105.89.161 178.128.194.144 179.43.169.181 18.191.163.164 18.194.196.202 18.197.125.181 18.216.92.75 18.224.151.181 18.237.205.164 18.237.5.42 185.14

jmilagroso

Status for the jail: sshd
|- Filter
|  |- Currently failed:	1510
|  |- Total failed:	2072
|  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned:	614
   |- Total banned:	614
   `- Banned IP list:	101.32.94.47 103.134.177.163 103.136.42.230 103.136.42.241 103.147.12.210 103.208.218.150 103.225.99.36 103.37.151.84 103.6.5.101 103.86.160.91 103.94.183.164 104.129.63.59 104.131.91.148 104.192.102.10 104.244.74.23 104.244.74.29 104.248.1.92 104.248.130.10 104.248.181.156 104.248.201.251 106.12.172.184 106.13.19.6 106.51.78.18 106.51.79.96 106.52.202.82 106.52.222.11 106.53.92.85 106.70.130.73 106.75.10.105 106.75.110.204 106.75.153.166 106.75.8.16 107.189.11.96 109.104.151.109 111.229.157.126 111.229.169.182 111.229.237.32 111.229.3.206 111.231.79.227 111.67.202.234 112.145.72.45 114.4.227.194 115.72.129.166 115.72.134.134 115.78.4.219 116.106.16.177 116.110.126.254 116.110.218.191 116.110.3.205 116.110.88.233 116.110.88.95 116.125.141.56 116.98.161.19 116.98.167.174 116.9

jmilagroso

# A dictionary is a collection which is unordered, changeable and indexed. 
# In Python dictionaries are written with curly brackets, and they have keys and values.
thisdict =	{
  "brand": "Ford",
  "model": "Mustang",
  "year": 1964
}
print(thisdict)

# You can access the items of a dictionary by referring to its key name, inside square brackets:

jmilagroso

Start Kafka service

The following commands will start a container with Kafka and Zookeeper running on mapped ports 2181 (Zookeeper) and 9092 (Kafka).

docker pull spotify/kafka
docker run -d -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 --name kafka spotify/kafka

Why Spotify?

ADVERTISTED_HOST was set to kafka, which will allow other containers to be able to run Producers and Consumers.

jmilagroso

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h):

• -E: will describe the underlaying queries of the \ commands (cool for learning!)
• -l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)

jmilagroso

Moved to git repository: https://github.com/denji/nginx-tuning

NGINX Tuning For Best Performance

For this configuration you can use web server you like, i decided, because i work mostly with it to use nginx.

Generally, properly configured nginx can handle up to 400K to 500K requests per second (clustered), most what i saw is 50K to 80K (non-clustered) requests per second and 30% CPU load, course, this was 2 x Intel Xeon with HyperThreading enabled, but it can work without problem on slower machines.

You must understand that this config is used in testing environment and not in production so you will need to find a way to implement most of those features best possible for your servers.

jmilagroso

public class RedisClient {

    private JedisPool pool;

    @Inject
    public RedisClient(Settings settings) {
        try {
            pool = new JedisPool(new JedisPoolConfig(), settings.get("redis.host"), settings.getAsInt("redis.port", 6379));
        } catch (SettingsException e) {
            // ignore

jmilagroso

package com.cloud4wi.finatra.controller

import org.cloud4wi.finagle.{WebServerDriverManager, WebServerQueryBuilder, WebServerQueryBuilderService, WebServerLevelHandler, WebServerConfig}
import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat

import scala.collection.immutable.Map
import scala.collection.mutable.ListBuffer
import com.typesafe.config.ConfigFactory

jmilagroso

#!/bin/bash
echo "Generating an SSL private key to sign your certificate..."
openssl genrsa -des3 -out myssl.key 1024

echo "Generating a Certificate Signing Request..."
openssl req -new -key myssl.key -out myssl.csr

echo "Removing passphrase from key (for nginx)..."
cp myssl.key myssl.key.org
openssl rsa -in myssl.key.org -out myssl.key