prodeezy

    protected Object bitmapToBase64(BitmapAgg myagg) throws HiveException {
        ByteArrayOutputStream bos;
        ObjectOutputStream oo;

        try {

            bos = new ByteArrayOutputStream();
            oo = new ObjectOutputStream(bos);
            myagg.bitmap.writeExternal(oo);
            oo.close();

prodeezy

Ad Network data format:

Account | Ad Network | App (Ad Network provided) | Date | metric1 | metric2 | metric3 | metric4 | metric5 | metric6 |


Tempest Message Format 


Message
{

prodeezy

api1-north  2014-06-20  7548827
api10-north  2014-06-20  7546419
api11-north  2014-06-20  7552706
api12-north  2014-06-20  7547739
api13-north  2014-06-20  7553455
api14-north  2014-06-20  7550577
api15-north  2014-06-20  7549240
api16-north  2014-06-20  7550762
api17-north  2014-06-20  7552714
api18-north  2014-06-20  7549276

prodeezy

Lag - Last 24 hours - open

        avg_lag  hours    max_lag  min_lag
0      0.000000  27/23      0.000    0.000
1      0.000000  27/22      0.000    0.000
2      0.000000  27/21      0.000    0.000
3      0.000000  27/20      0.000    0.000
4   5677.550671  27/19  15575.327    5.141
5     68.555253  27/18   7786.404    3.864
6      8.769147  27/17     21.812    3.879

prodeezy

Tesseract

                     open_redshift  tesseract_open     delta
ts                                               
2014-06-19 00:00:00      385908  373417  0.016450
2014-06-19 01:00:00      400922  388141  0.016198
2014-06-19 02:00:00      392543  379782  0.016523
2014-06-19 03:00:00      362953  351028  0.016702
2014-06-19 04:00:00      325817  315403  0.016241
2014-06-19 05:00:00      290542  281247  0.016256

prodeezy

node511: 

Region Server:

2014-06-05 07:31:54,308 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Close and delete failed
2014-06-05 07:32:08,418 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 3 retries
2014-06-05 07:32:08,419 ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine: Region server exiting
java.io.IOException: Bad response ERROR for block BP-1912823949-10.40.191.16-1393114060686:blk_7002517951301512390_68018352 from datanode 10.81.138.62:50010
java.io.IOException: Bad response ERROR for block BP-1912823949-10.40.191.16-1393114060686:blk_-7342833490402110940_68018364 from datanode 10.81.138.62:50010

prodeezy

Current List of Repos : 

valhalla-core
valhalla-message-loader
valhalla-meta
metric-meta
valkyrie
tesseract-loader
tesseract-server

prodeezy

    @Override
    public TupleEntryIterator openForRead(FlowProcess<JobConf> flowProcess,
                                          RecordReader input)

            throws IOException  {

        System.out.println("[CustomTap] => openForRead()");
        this.table = new HTable(flowProcess.getConfigCopy(), tableName) ;

prodeezy

// *****  Construct a custom Scan object based on custom filters *****
        Filter filterList = MessageOperations.getFilter("app1",
                                                        "foo",
                                                        "123",
                                                        2014, 2, 12,
                                                        -1, -1, -1, -1);
        Scan tableScan = new Scan();
        tableScan.setFilter(filterList);
        tableScan.addFamily("c".getBytes());

prodeezy

package org.apache.hadoop.hbase.util;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.util.ToolRunner;
import org.apache.hadoop.conf.Configuration;

import java.io.IOException;
import java.util.ArrayList;