pgaref

Spark HWC integration - HDP 3 Secure cluster

Prerequisites :

• Kerberized Cluster

• Enable hive interactive server in hive

• Get following details from hive for spark or try this HWC Quick Test Script

pgaref

###############################
### ENVIRONMENTAL VARIABLES ###
###############################

# HomeBrew PATH
export PATH=/usr/local/bin:/usr/local/sbin:$PATH

# Jenv
export PATH="$HOME/.jenv/bin:$PATH"
eval "$(jenv init -)"

pgaref

##################
###   ALIASES  ###
##################

# git command autocompletion script
source ~/.git-completion.bash
source ~/.git-prompt.sh
source ~/.profile

# git commamands simplified

pgaref

#!/bin/bash

DIRNAME="/backups/production"
DOW=`eval date +%A`
REPLICANAME="production_replica"

declare -a DBNAMES=("example_v1_0" "example_v2_0" "example_v3_0" example_v4_0)

for DBN in "${DBNAMES[@]}"
do

pgaref

# Path is: /etc/logrotate.d/my_job

/var/log/supervisor/my_job.log{
        daily
        missingok
        rotate 1
        compress
        delaycompress
        notifempty
        create 0755 datateam adm

pgaref

**/etc/apt/preferences.d/trusty-preferences**

Package: *
Pin: release a=trusty
Pin-Priority: 1010

Package: *
Pin: release a=trusty-updates
Pin-Priority: 1011

pgaref

getent passwd | cut -d : -f 1 | xargs groups 

| grep sudo
| grep root

You also have to check: /etc/sudoers 

pgaref

 1) Setting "managed=false" to "managed=true" in the file: /etc/NetworkManager/NetworkManager.conf
 
 2) /etc/network/interfaces (Can find defaults using nm-tool)
  auto lo
  iface lo inet loopback
  auto eth0
  iface eth0 inet static
  address 192.168.1.108
  netmask 255.255.255.0
  network 192.168.1.0

pgaref

#To find any expired repository keys and their IDs, use apt-key as follows:

apt-key list | grep expired
#You will get a result similar to the following:

pub   4096R/BE1DB1F1 2011-03-29 [expired: 2014-03-28]
#The key ID is the bit after the / i.e. BE1DB1F1 in this case.

#To update the key, run

pgaref

awk '{print $7}' access.log | # Split by whitespace, 7th field is request path
    sort                    | # Make occurrences of the same URL appear consecutively in file
    uniq -c                 | # Replace consecutive occurrences of the same URL with a count
    sort -rn                | # Sort by number of occurrences, descending
    head -n 5                 # Output top 5 URLs