commandlineconsole

package com.mapr.kafka.serializer.json;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.connect.json.JsonSerializer;

import java.util.Properties;

commandlineconsole

hadoop fs -cat /Work/lon_text/lon_order_data_t/cdw320_lon_order_data_t.1.txt | head -100 | gzip > test.csv.gz

cat cdw320_lon_order_data_t.1.txt | head -100 | gzip > ../../tsnyder/cdw320_lon_order_data_t.1.txt.gz

hadoop fs -cat /Work/tsnyder/cdw320_lon_order_data_t.1.txt.gz | gunzip

-------------------
./test.sh  // Execute following script, keep following script in the test.sh
for appending the file >>, replacing  the file >
---------

commandlineconsole

Using Kafka MirrorMaker on DCOS to migrate from DCOS_Kafka version=1.1.19-3.2.0e  (OLD)  to DCOS_Kafka version=2.0.3-3.3.1e. (NEW)

This was done in a lab with sample data

Your results may differ than mine.

This is recently upgraded DCOS cluster from 1.8.8 to 1.1.0.3 

Assumptions and environment:

commandlineconsole

import java.util

import org.apache.kafka.clients.consumer.KafkaConsumer

import scala.collection.JavaConverters._

object ConsumerExample extends App {

  import java.util.Properties

commandlineconsole

import kafka.producer._
import kafka.utils._
import java.util._
import java.text._
import java.util.concurrent.atomic._

class KafkaPartitioner(props: VerifiableProperties = null) extends Partitioner {
  
  val counter = new AtomicInteger(0)
  val batch = new AtomicInteger(0)

commandlineconsole

# Run every time you login or update .bash_profile
export KAFKA_HOME=/usr/hdp/2.5.0.0-1245/kafka
export PATH=$PATH:$KAFKA_HOME/bin

# Create topics
kafka-topics.sh --create \
  --zookeeper nn01.itversity.com:2181,nn02.itversity.com:2181,rm01.itversity.com:2181 \
  --replication-factor 1 \
  --partitions 1 \
  --topic kafkadg

commandlineconsole

#!/usr/bin/env ruby

require 'poseidon'
require 'pg'

conn = PG.connect(host: 'localhost', port: 5432, dbname: 'postgres', user: 'postgres')

begin
  conn.exec("CREATE DATABASE metrics")
rescue PG::DuplicateDatabase

commandlineconsole

---------- Forwarded message ----------
From: Beth Rodgers <b.rodgers@nixor.net>
Date: 28 March 2018 at 17:41
Subject: 12 Month React Contact- Home Office-Police- £500pd-£600pd 
To: Janaidu01@gmail.com


Hi Ja, 
I hope you're well, you may have seen an email from me about 4 weeks ago about 12 months React project for the Home Office, I've finally been given the green light for this and interview slots to book people in for who are interested. 

commandlineconsole

show dbs
// LIst Data Base

show collections
// List Collections

use test
// Select Db

db

commandlineconsole

mail