dqtweb

Migrate from Facebook scribe to Apache Flume (Part II)

In last article we talked about how to setup flume and write files HDFS. This article, we begin to change flume to write file in scribe like style category. Multiplexing Way?

The first thought is using source multiplex to distribute log to different destination. Flume distribute log events by event header. So we google to find out which field in header is referring to scribe header.

https://apache.googlesource.com/flume/+/d66bf94b1dd059bc7e4b1ff332be59a280498077/flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java

category in header will refer to scribe category. So we try to use multiplexing source:

dqtweb

Migrate from Facebook scribe to Apache Flume (Part I)

Reason

We use scribe as our logging server for a long time. At first, everything works fine. Easy to config, easy to manage. As data grows everyday, single scribe server can’t handle that. We have to migrate some category to second log server and attach a big disk. As data is keep growing, we want a big data storage for that instead of local disk. So we decide to use Scribe with HDFS plugin. It is as tough as we first compile scribe from source. Finally we complied the scribed with hdfs support. But after a short period usage, we find a bug that haven’t solved by facebook. (the project is deprecated several years ago). The bug cause scribe can’t write to hdfs if it accidently killed by SIG 9. So we start to test flume and find out ways to migrate.

Configure Flume

Flume is easy to deploy because it is written in Java. Install java and download jar package, we’ve done all the jobs

dqtweb

# Redis Cheatsheet
# All the commands you need to know


redis-server /path/redis.conf  # start redis with the related configuration file
redis-cli                      # opens a redis prompt


# Strings.

dqtweb

window.open($(".zoom-modal img:nth-child(2)").attributes.src.value, "_blank");

dqtweb

def valid_date_type(arg_date_str):
    """custom argparse *date* type for user dates values given from the command line"""
    try:
        return datetime.datetime.strptime(arg_date_str, "%Y-%m-%d")
    except ValueError:
        msg = "Given Date ({0}) not valid! Expected format, YYYY-MM-DD!".format(arg_date_str)
        raise argparse.ArgumentTypeError(msg)
        
def valid_datetime_type(arg_datetime_str):
    """custom argparse type for user datetime values given from the command line"""

dqtweb

#!/usr/bin/python
import subprocess
import os
import sys

#res = subprocess.Popen(['ls','-al','/ahome'],stdout=subprocess.PIPE,stderr=subprocess.PIPE);
#output,error = res.communicate()

#if res.returncode:
#   #raise Exception(error)

dqtweb

from datetime import datetime
import time

#-------------------------------------------------
# conversions to strings
#-------------------------------------------------
# datetime object to string
dt_obj = datetime(2008, 11, 10, 17, 53, 59)
date_str = dt_obj.strftime("%Y-%m-%d %H:%M:%S")
print date_str

dqtweb

import numpy  
import pandas as pd  
import math as m

#@author: Bruno Franca
#@author: Peter Bakker

#Moving Average  
def MA(df, n):  
    MA = pd.Series(pd.rolling_mean(df['Close'], n), name = 'MA_' + str(n))  

dqtweb

var http = require('http');
var Static = require('node-static');
var WebSocketServer = new require('ws');

// list of users
var CLIENTS=[];
var id;

// web server is using 8081 port
var webSocketServer = new WebSocketServer.Server({ port: 8081 });

dqtweb

var mongoose = require('mongoose');

var hitSchema = mongoose.Schema({
  text: String,
  music: String
});
hitSchema.statics.bulkInsert = function(models, fn) {
  if (!models || !models.length)
    return fn(null);