onefoursix

In the Groovy Stage's init script, look for this line of code (around line 138):

            // Execute the request      
            HttpResponse response = httpClient.execute(request)

Comment out that line and replace that line with this section of code:

            // mbrooks 06/01/2024 -- comment out this line
            // HttpResponse response = httpClient.execute(request)

onefoursix

import os
from datetime import datetime
from time import time
import sys
from streamsets.sdk import ControlHub
import json

# Set to true to echo the metrics to stdout
print_metrics = True

onefoursix

 #!/usr/bin/env python

'''
This script writes a continuous stream of CPU and Memory metrics for a given SDC
as well as counts of all running pipelines' input records, output records, and error records.

The script uses StreamSets Platform's REST API to pull metrics directly from SDCs; it does not connect to Control Hub

On each refresh interval, the script will record CPU and memory metrics and then the record counts for each running pipeline.

onefoursix

export CRED_ID="..."

export CRED_TOKEN="..."

curl -X GET  "https://rancher.onefoursix.com:31910/rest/v1/system/jmx" \
 -H "X-SS-App-Component-Id: $CRED_ID" \
 -H "X-SS-App-Auth-Token: $CRED_TOKEN"

onefoursix

#!/usr/bin/env python

'''
This script writes a rolling log file that contains running Pipeline names and record counts,
along with SDC CPU usage and JVM heap memory metrics for all Data Collectors registered 
with StreamSets Platform that match the specified set of Labels.

The script writes a sdc-resource-metrics.log as a rolling log file with the pipeline and SDC metrics
as well as a sdc-resource-metrics-messages-and-errors.log file that shows the SDCs that are discovered
and whether connections to them are successful.

onefoursix

#!/usr/bin/env bash

# Script that calls the /provisioning/rest/v1/csp/deployment/${DEP_ID}/unsafeForceStop 
# endpoint to delete StreamSets Kubernetes Deployments stuck in an Activating state.
#
# Use this REST API endpoint only if Kubernetes Deployments are stuck 
# in an Activating state.  
#
# After the script completes, the selected Deployments will be in a 
# "Deactivation Error" state and can be deleted using the Control Hub UI

onefoursix

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: streamsets-deployment-576a6314-076a-4eba-9916-1482c89cae72
  name: streamsets-deployment-576a6314-076a-4eba-9916-1482c89cae72
  namespace: ns100
spec:
  replicas: 1
  selector:

onefoursix

#!/usr/bin/env python

'''
This script writes a rolling log file that contains CPU usage and JVM heap memory metrics
for a given Data Collector registered with Control Hub 3.x, with a user definable refresh
interval, along with the number and names of Jobs running on the Data Collector at the time
of metrics collection.

Prerequisites:

onefoursix

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: streamsets-deployment-cfa81f1d-baf1-4d7d-9136-e0114a083bc9
  name: streamsets-deployment-cfa81f1d-baf1-4d7d-9136-e0114a083bc9
  namespace: ns1
spec:
  replicas: 1
  selector:

onefoursix

#!/usr/bin/env python

'''
This script exports Fragments, Pipelines, Jobs, and Job Templates from StreamSets DataOps Platform
 
The current version of this script does not export Connections, Tasks, nor Topologies
 
 
Prerequisites:
 - Python 3.6+; Python 3.9+ preferred