Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

View anand086's full-sized avatar

Anand Prakash anand086

6 followers · 22 following
View GitHub Profile
@anand086
anand086 / comprehend-detect-pii-entities.ipynb
Created September 6, 2022 05:37
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@anand086
anand086 / glue-pii-find-sensitive-data-in-each-row-and-redact-detected-text.py
Last active September 4, 2022 22:48
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import EntityDetector
from pyspark.sql.types import StringType
from awsglue.dynamicframe import DynamicFrame
from pyspark.sql.functions import *
@anand086
anand086 / glue-pii-find-columns-that-contain-sensitive-data-output-redacted.py
Last active September 4, 2022 22:22
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import EntityDetector
from awsglue.dynamicframe import DynamicFrame
from pyspark.sql.functions import *
@anand086
anand086 / glue-pii-find-columns-that-contain-sensitive-data-output-detection-results.py
Created September 4, 2022 22:04
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import EntityDetector
from pyspark.sql.types import MapType, StringType, StructType, StructField
from awsglue.dynamicframe import DynamicFrame
@anand086
anand086 / glue-pii-find-sensitive-data-in-each-row-and-encrich-data-with-detection-results.py
Created September 4, 2022 20:12
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import EntityDetector
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
@anand086
anand086 / msk_connector_cloudwatch_eu-central-1
Created September 4, 2022 08:23
[Worker-064f2e13cca81bb69] [2022-09-04 08:12:54,563] INFO [lensesio-msk-transaction-events-src-s3-1|task-0] WorkerSinkTask{id=lensesio-msk-transaction-events-src-s3-1-0} Committing offsets asynchronously using sequence number 11: {transaction_events-0=OffsetAndMetadata{offset=8, leaderEpoch=null, metadata=''}, transaction_events-1=OffsetAndMetadata{offset=3, leaderEpoch=null, metadata=''}, transaction_events-2=OffsetAndMetadata{offset=9, leaderEpoch=null, metadata=''}, transaction_events-3=OffsetAndMetadata{offset=107, leaderEpoch=null, metadata=''}, transaction_events-4=OffsetAndMetadata{offset=8, leaderEpoch=null, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:352)
[Worker-064f2e13cca81bb69] [2022-09-04 08:12:54,733] INFO [lensesio-msk-transaction-events-src-s3-1|task-1] WorkerSinkTask{id=lensesio-msk-transaction-events-src-s3-1-1} Committing offsets asynchronously using sequence number 11: {transaction_events-8=OffsetAndMetadata{offset=105, leaderEpoch=null, metadata=''}, transaction_events
@anand086
anand086 / msk_connector_configuration_eu-central-1
Created September 4, 2022 08:22
connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector
tasks.max=2
topics=transaction_events
connect.s3.vhost.bucket=true
schema.enable=false
key.converter.schemas.enable=false
connect.s3.kcql=INSERT INTO msk-lensesio-plugin:MSKCluster SELECT * FROM transaction_events STOREAS `JSON` WITH_FLUSH_INTERVAL = 300
aws.region=eu-central-1
aws.custom.endpoint=https://s3.eu-central-1.amazonaws.com/
value.converter.schemas.enable=false
@anand086
anand086 / traffic_violations_p table
Created March 29, 2022 01:26
\d+ traffic_violations_p
Partitioned table "admin.traffic_violations_p"
+-------------------------+------------------------+-----------+----------+---------+----------+--------------+-------------+
| Column | Type | Collation | Nullable | Default | Storage | Stats target | Description |
+-------------------------+------------------------+-----------+----------+---------+----------+--------------+-------------+
| seqid | text | | | | extended | | |
| date_of_stop | date | | not null | | plain | | |
| time_of_stop | time without time zone | | | | plain | | |
| agency | text | | | | extended | | |
| subagency | tex
@anand086
anand086 / athena_transaction_details.sql
Created February 17, 2022 18:25
SELECT additionaldata.cardholdername,
card_detail.cardtype,
transaction.currency[1] as currency,
transaction.amount,
from_unixtime(transaction.created_at/1000.0) as timestamp
FROM msk_transaction_events WHERE success=true and transaction.amount > 7000 order by 5 desc
@anand086
anand086 / transaction_events_per_partition_message_count
Created February 17, 2022 08:40
[ec2-user@ip-10-0-1-43 ~]$ kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $BOOTSTRAP --topic transaction_events | awk -F ":" '{print $3}'
22369
22060
22547
22231
22340