HDF workshop steps

HDF workshop steps.md

HDF workshop

Setup already done

• Download and import latest HDP 2.3 sandbox VM from http://hortonworks.com/sandbox and import into VMware Fusion
• Deployed and install VNC ambari service (to be able to 'remote desktop' in and use eclipse)

  sudo git clone https://github.com/hortonworks-gallery/ambari-vnc-service.git /var/lib/ambari-server/resources/stacks/HDP/2.3/services/VNCSERVER
  
  

• Deployed Nifi ambari service (but not yet installed)

sudo git clone https://github.com/abajwa-hw/ambari-nifi-service.git   /var/lib/ambari-server/resources/stacks/HDP/2.3/services/NIFI
service ambari restart

• Banana setup for Twitter demo

  #download prebuilt banana dashboard
  cd /opt/lucidworks-hdpsearch/solr/server/solr-webapp/webapp/banana/app/dashboards/
  mv default.json default.json.orig
  wget https://raw.githubusercontent.com/abajwa-hw/ambari-nifi-service/master/demofiles/default.json
  
  

• Solr setup for Twitter demo

  #modify solrconfig to handle twitter time format
  vi /opt/lucidworks-hdpsearch/solr/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml
  
  #start Solr in cloud mode
  /opt/lucidworks-hdpsearch/solr/bin/solr start -c -z localhost:2181
  
  #create collection called tweets
  /opt/lucidworks-hdpsearch/solr/bin/solr create -c tweets \
   -d data_driven_schema_configs \
   -s 1 \
   -rf 1 
  

• Download custom network processor

  cd
  sudo git clone https://github.com/abajwa-hw/nifi-network-processor.git  
  
  

• Eclipse setup: Created run configuration for /root/nifi-network-processor using /usr/share/apache-maven

Steps

Getting started

Start with steps from: https://github.com/abajwa-hw/ambari-nifi-service

1. Install Nifi via Ambari
2. Basic overview of OOTB processors
3. Build simple flow to Read log over UDF and write to HDFS
4. Import template for more complete example of log ingest flow
5. Import template for Twitter -> HDFS/Solr to show Nifi interacting with HDP
6. Nifi features:

• provenance: show how replay/download affects lineage

• priority

• backpressure

• control throughput vs latency

• statistics

• Production example

  • Acquisition Hub: https://distro-1.demo.onyara.com:8443/nifi/
  • Collects data from:
    • Syslog data
    • Network traffic
    • Twitter
    • HL7
    • Photos
  • Processing Cluster: https://processing.demo.onyara.com:8443/nifi/
  • Unwraps:
    • HL7 data
    • Photos data
    • translates/finds interesting tweets

Custom processor

Steps available at: https://github.com/abajwa-hw/nifi-network-processor

• Build network snooping processor using expression language (https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html)
• Show how to do same using custom processors:
  • Import code into eclipse
  • Code walk through
  • Maven build from eclipse
  • Deploy to Nifi lib dir and restart Nifi
  • Open Nifi and use processor in flow