seanorama

helmfile jsonPatches does not work with packaged (e.g. tgz) charts

It fails for local file-system or remote packaged charts, but they return different errors (see below).

The errors appear to be from chartify.

It works if the package is extracted and chart used from the directory: chart: ./datadog/.

Steps to reproduce

seanorama

#!/usr/bin/env bash -e

# How to use this script:
# 1. Follow these instructions to configure a single AWS account to do initial login with SSO
#    https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html
# 2. Export AWS_PROFILE=... and then run "aws sso login" to get an SSO token
# 3. Once signed in with AWS SSO, run this script to automatically list out all the other accounts and roles and add them to your config file

# If you want to filter roles / accounts in the process, or validate config before committing it, you can customise the script to do this.

seanorama

.PHONY: help

help:
	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

init: ## Initializes the terraform remote state backend and pulls the correct environments state.
	@if [ -z ${env} ]; then echo "env was not set" ; exit 10 ; fi
	@rm -rf .terraform/*.tf*
	@terraform init \
        -backend-config=environments/${env}/backend.conf

seanorama

Compress files which are already on HDFS

This hacky method processes 1 file at a time:

1. copy to a local disk
2. compress
3. put back onto HDFS
4. delete original file from HDFS and compressed file from local disk.

BE CAREFUL: Before executing, inspect the size of each file!

• The risk is: a single large file could fill the local disk or you could leave the server compressing a single large file for hours.

seanorama

group="hadoop-admins"

dirs="
  /var/log/ambari-agent
  /var/log/ambari-server
  /var/log/ambari-infra-solr
  /var/log/ambari-infra-solr-client
  /var/log/ambari-logsearch-logfeeder
  /var/log/ambari-logsearch-portal
  /var/log/ambari-metrics-agent

seanorama

PySpark on YARN in self-contained environments

Author: https://github.com/seanorama

Note: This was tested on HDP 3.1. It may not work with other Spark/YARN distributions.

Reference:

• https://community.cloudera.com/t5/Community-Articles/Using-VirtualEnv-with-PySpark/ta-p/245905
• https://community.cloudera.com/t5/Community-Articles/Running-PySpark-with-Conda-Env/ta-p/247551

seanorama

How to work with nssdb and certutil

remove database

rm -r ~/.pki/nssdb

create database directory

seanorama

From 6fae0efce307b0ba8d5520189af0de4be21edccb Mon Sep 17 00:00:00 2001
From: Andrew Onishuk <aonishuk@hortonworks.com>
Date: Fri, 1 Feb 2019 11:20:49 +0200
Subject: [PATCH] AMBARI-25123. /var/lib/ambari-agent/cache not updating
 (Ambari 2.7) (aonishuk)

---
 .../src/main/python/ambari_agent/CustomServiceOrchestrator.py    | 1 +
 1 file changed, 1 insertion(+)

seanorama

Spark Hive Warehouse Connector (HWC) and Spark HBase Connector (SHC)

For Zeppelin %livy2 and Spark

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/using-zeppelin/content/using_spark_hwc_and_shc_client_jar_files_with_livy.html

---

Install dependencies:

seanorama

Ambari Alert to check if NiFi's /nifi-api/flow/process-groups/root is working

Installation:

• Save alert_check_nifi_processgroup.py to /var/lib/ambari-server/resources/common-services/NIFI/1.0.0/packages/scripts/
• sudo chmod a+rx alert_check_nifi_processgroup.py
• Save alerts.json.patch to /tmp/
• Apply patch: sudo patch -b /var/lib/ambari-server/resources/common-services/NIFI/1.0.0/alerts.json /tmp/alerts.json.patch
• On Ambari Server: sudo ambari-server refresh-stack-hash
• On all hosts: sudo ambari-agent restart