This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There are two modes to compare the datasets. | |
a. Compare two way : Compares source and destination records. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import xlrd | |
import csv | |
import glob | |
import os | |
from os import sys | |
def csv_from_excel(excel_file): | |
workbook = xlrd.open_workbook(excel_file) | |
all_worksheets = workbook.sheet_names() | |
for worksheet_name in all_worksheets: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for file in *; do mv "$file" "${file//[ ()@$]/_}" ; done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CREATE TABLE dataq_demo.large_test (num1 bigint, num2 double precision, num3 double precision); | |
INSERT INTO dataq_demo.large_test (num1, num2, num3) | |
SELECT round(random()*10), random(), random()*142 | |
FROM generate_series(1, 20000000) s(i); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CREATE TABLE dataq_demo.tbl_ItemTransactions1 | |
( | |
TranID SERIAL | |
,TransactionDate TIMESTAMPTZ | |
,TransactionName TEXT | |
); | |
drop table dataq_demo.tbl_ItemTransactions1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ | |
{ | |
"classification":"spark", | |
"properties":{ | |
"maximizeResourceAllocation":"true", | |
"spark.yarn.executor.memoryOverhead":"1024M" | |
}, | |
"configurations":[ | |
] |