elprup

import os
from Crypto.Cipher import AES
import pyaes
import base64

def _create_secret_key(size):
    randlist = map(lambda xx: (hex(ord(xx))[2:]), os.urandom(size))
    return (''.join(randlist))[0:16]

def _aes_encrypt2(text, sec_key):

elprup

"""
Simple example for streamcallback usage in Tornado.
"""
import tornado.httpserver
import tornado.ioloop
import tornado.options
import tornado.web
import tornado.httpclient

from tornado.options import define, options

elprup

#coding=utf8

import urllib
import urllib2
import cookielib
import base64
import re, sys, json
import binascii
import rsa

elprup

替换上你的Email，密码，域名ID，记录ID等参数，就可以运行了。 会在后台一直运行，每隔30秒检查一遍IP，如果修改了就更新IP。

获得domain_id可以用curl curl -k https://dnsapi.cn/Domain.List -d "login_email=xxx&login_password=xxx"

获得record_id类似 curl -k https://dnsapi.cn/Record.List -d "login_email=xxx&login_password=xxx&domain_id=xxx"

elprup

Migrate from Facebook scribe to Apache Flume (Part II)

In last article we talked about how to setup flume and write files HDFS. This article, we begin to change flume to write file in scribe like style category. Multiplexing Way?

The first thought is using source multiplex to distribute log to different destination. Flume distribute log events by event header. So we google to find out which field in header is referring to scribe header.

https://apache.googlesource.com/flume/+/d66bf94b1dd059bc7e4b1ff332be59a280498077/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java

category in header will refer to scribe category. So we try to use multiplexing source:

elprup

Migrate from Facebook scribe to Apache Flume (Part I)

Reason

We use scribe as our logging server for a long time. At first, everything works fine. Easy to config, easy to manage. As data grows everyday, single scribe server can’t handle that. We have to migrate some category to second log server and attach a big disk. As data is keep growing, we want a big data storage for that instead of local disk. So we decide to use Scribe with HDFS plugin. It is as tough as we first compile scribe from source. Finally we complied the scribed with hdfs support. But after a short period usage, we find a bug that haven’t solved by facebook. (the project is deprecated several years ago). The bug cause scribe can’t write to hdfs if it accidently killed by SIG 9. So we start to test flume and find out ways to migrate.

Configure Flume

Flume is easy to deploy because it is written in Java. Install java and download jar package, we’ve done all the jobs

elprup

class BaseHandler(tornado.web.RequestHandler):
    def get_current_user(self):
        user_json = self.get_secure_cookie("authdemo_user")
        if not user_json: return None
        return tornado.escape.json_decode(user_json)

class GoogleOAuth2LoginHandler(tornado.web.RequestHandler,
                               tornado.auth.GoogleOAuth2Mixin):
    @tornado.gen.coroutine
    def get(self):

elprup

#!/usr/bin/env python
'''
Scribe checker with heartbeat

desc: check scribe alive status, alert if fail
'''

import os
import sys
import time

elprup

#!/usr/bin/python
import subprocess as spb
import os.path
import sys

msgfmt_path = '/tmp/msgfmt.py'

def run(cmd):
    print 'func[cmd]:' + cmd
    sock = spb.Popen(cmd, shell=True, stdout=spb.PIPE)

elprup

蓄水池抽样 Reservoir sampling