David G. Cueva Tello davidcueva
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.110.mysqlPwd.sh
Created
April 29, 2020 15:31
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| echo "mysql-root-password-99" | \ | |
| gcloud kms encrypt \ | |
| --location=global \ | |
| --keyring=my-keyring \ | |
| --key=my-key \ | |
| --plaintext-file=- \ | |
| --ciphertext-file=mysql-root-password.encrypted |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.120.createRangerBucket.sh
Created
April 29, 2020 15:32
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gsutil mb -l ${REGION} gs://${PROJECT_ID}-ranger |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.130.cpEncrypted.sh
Created
April 29, 2020 15:33
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gsutil -m cp *.encrypted gs://${PROJECT_ID}-ranger |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.140.createSolrBucket.sh
Created
April 29, 2020 15:34
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gsutil mb -l ${REGION} gs://${PROJECT_ID}-solr |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.150.exportsRanger.sh
Created
April 29, 2020 15:36
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| export BACKEND_CLUSTER=backend-cluster | |
| export PROJECT_ID=$(gcloud info --format='value(config.project)') | |
| export REGION=us-central1 | |
| export ZONE=us-central1-b | |
| export CLOUD_SQL_NAME=cloudsql-mysql | |
| export RANGER_KMS_KEY_URI=\ | |
| projects/${PROJECT_ID}/locations/global/keyRings/my-keyring/cryptoKeys/my-key |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.160.createBackendCluster.sh
Created
April 29, 2020 15:37
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gcloud beta dataproc clusters create ${BACKEND_CLUSTER} \ | |
| --optional-components=SOLR,RANGER \ | |
| --region ${REGION} \ | |
| --zone ${ZONE} \ | |
| --enable-component-gateway \ | |
| --scopes=default,sql-admin \ | |
| --service-account=visualization-security-sa@${PROJECT_ID}.iam.gserviceaccount.com \ | |
| --properties="\ | |
| dataproc:ranger.kms.key.uri=${RANGER_KMS_KEY_URI},\ | |
| dataproc:ranger.admin.password.uri=${RANGER_ADMIN_PWD_URI},\ |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.170.createHiveBucket.sh
Created
April 29, 2020 15:41
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gsutil mb -l ${REGION} gs://${PROJECT_ID}-hive |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.180.cpParquet.sh
Created
April 29, 2020 15:42
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gsutil cp gs://hive-solution/part-00000.parquet \ | |
| gs://${PROJECT_ID}-hive/dataset/transactions/part-00000.parquet |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.190.sshBackend.sh
Created
April 29, 2020 15:43
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| gcloud compute ssh --zone ${ZONE} ${BACKEND_CLUSTER}-m |
davidcueva
/ com.google.cloud.hadoop2GCP.visualizationSec.200.beelineBackend.sh
Created
April 29, 2020 15:46
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| beeline -u "jdbc:hive2://localhost:10000/;transportMode=http;httpPath=cliservice admin admin-password"\ | |
| --hivevar PROJECT_ID=$(gcloud info --format='value(config.project)') |