Anand Prakash anand086
anand086
/ fake_transaction_data_kafka.py
Created
February 17, 2022 08:27
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import os | |
| import sys | |
| import json | |
| import datetime | |
| import time | |
| import random | |
| import uuid | |
| import logging | |
| from faker import Faker | |
| from kafka import KafkaProducer |
anand086
/ pyspark_pg_view_to_s3_create_dynamic_frame_from_catalog.py
Created
January 18, 2022 02:37
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import sys | |
| import os | |
| import logging | |
| from awsglue.transforms import * | |
| from awsglue.utils import getResolvedOptions | |
| from pyspark.context import SparkContext | |
| from awsglue.context import GlueContext | |
| from awsglue.dynamicframe import DynamicFrame | |
| from awsglue.job import Job |
anand086
/ hr_female_all_data_view.sql
Created
January 18, 2022 02:33
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| CREATE OR REPLACE VIEW admin.hr_female_all_data as SELECT hr.empid, | |
| hr.firstname, | |
| hr.lastname, | |
| hr.email, | |
| hr.doj, | |
| hr.salary, | |
| hr.phone, | |
| hr.country, | |
| hr.zip, | |
| hr.username |
anand086
/ pyspark_pg_to_s3_create_dynamic_frame_from_options.py
Created
January 18, 2022 02:26
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import sys | |
| import os | |
| import logging | |
| from awsglue.transforms import * | |
| from awsglue.utils import getResolvedOptions | |
| from pyspark.context import SparkContext | |
| from awsglue.context import GlueContext | |
| from awsglue.dynamicframe import DynamicFrame | |
| from awsglue.job import Job |
anand086
/ glue_job_run_9.log
Created
January 18, 2022 02:23
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [GlueFleetdbPGS3:36 - <module>() ] 2022-01-02:22:33:58 Create dynamicframe by reading data from admin.hr table in fleetdb | |
| [GlueFleetdbPGS3:45 - <module>() ] 2022-01-02:22:34:15 datasource03 rowcount - 1 | |
| [GlueFleetdbPGS3:50 - <module>() ] 2022-01-02:22:34:16 Number of partitions for datasource03 - 20 | |
| [GlueFleetdbPGS3:54 - <module>() ] 2022-01-02:22:34:16 Number of partitions for datasource03 after coalesce - 1 | |
| [GlueFleetdbPGS3:70 - <module>() ] 2022-01-02:22:34:17 Write dynamicframe to S3 | |
| [GlueFleetdbPGS3:74 - <module>() ] 2022-01-02:22:34:19 Completed |
anand086
/ pg_insert_5.sql
Created
January 18, 2022 02:22
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| insert into hr values (987,'Ophelia','Pena','F','ophelia.pena@gmail.com','1/11/86','6/23/16',89068,'144-23-2416','229-494-7152','Camden','Kingsland','GA',31548,'South','ozpena'); | |
| INSERT 0 1 | |
| Time: 3.358 ms | |
| insert into hr values (1000,'Andy','Gillespie','M','andy.gillespie@yahoo.com','7/21/89','10/21/12',143979,'558-99-4685','231-470-6122','Oakland','Franklin','MI',48025,'Midwest','aygillespie'); | |
| INSERT 0 1 | |
| Time: 3.437 ms |
anand086
/ glue_job_run_8.log
Created
January 18, 2022 02:22
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [GlueFleetdbPGS3:36 - <module>() ] 2022-01-02:22:28:56 Create dynamicframe by reading data from admin.hr table in fleetdb | |
| [GlueFleetdbPGS3:45 - <module>() ] 2022-01-02:22:29:12 datasource03 rowcount - 1 | |
| [GlueFleetdbPGS3:50 - <module>() ] 2022-01-02:22:29:14 Number of partitions for datasource03 - 20 | |
| [GlueFleetdbPGS3:54 - <module>() ] 2022-01-02:22:29:14 Number of partitions for datasource03 after coalesce - 1 | |
| [GlueFleetdbPGS3:70 - <module>() ] 2022-01-02:22:29:14 Write dynamicframe to S3 | |
| [GlueFleetdbPGS3:74 - <module>() ] 2022-01-02:22:29:17 Completed |
anand086
/ pg_insert_4.sql
Created
January 18, 2022 02:21
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| insert into hr values (999,'Maritza','Christian','F','maritza.christian@microsoft.com','5/19/87','3/15/12',66854,'336-11-5030','203-224-9056','Fairfield','Stamford','CT',6910,'Northeast','mhchristian') |
anand086
/ glue_job_run_7.log
Created
January 18, 2022 02:21
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [GlueFleetdbPGS3:36 - <module>() ] 2022-01-02:22:21:11 Create dynamicframe by reading data from admin.hr table in fleetdb | |
| [GlueFleetdbPGS3:36 - <module>() ] 2022-01-02:22:21:11 Create dynamicframe by reading data from admin.hr table in fleetdb | |
| [GlueFleetdbPGS3:45 - <module>() ] 2022-01-02:22:21:28 datasource03 rowcount - 0 | |
| [GlueFleetdbPGS3:74 - <module>() ] 2022-01-02:22:21:30 Completed |
anand086
/ glue_job_run_6.log
Created
January 18, 2022 02:20
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [GlueFleetdbPGS3:36 - <module>() ] 2022-01-02:21:55:25 Create dynamicframe by reading data from admin.hr table in fleetdb | |
| [GlueFleetdbPGS3:45 - <module>() ] 2022-01-02:21:55:41 datasource03 rowcount - 500 | |
| [GlueFleetdbPGS3:50 - <module>() ] 2022-01-02:21:55:42 Number of partitions for datasource03 - 20 | |
| [GlueFleetdbPGS3:54 - <module>() ] 2022-01-02:21:55:43 Number of partitions for datasource03 after coalesce - 1 | |
| [GlueFleetdbPGS3:70 - <module>() ] 2022-01-02:21:55:43 Write dynamicframe to S3 | |
| [GlueFleetdbPGS3:74 - <module>() ] 2022-01-02:21:55:45 Completed |