Skip to content

Instantly share code, notes, and snippets.

@johnfelipe

johnfelipe/"datahub docker ingest-sample-data".sh Secret

Created June 20, 2022 21:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save johnfelipe/c42939ad27793fbabbd745f636b770b2 to your computer and use it in GitHub Desktop.
Save johnfelipe/c42939ad27793fbabbd745f636b770b2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
"datahub docker ingest-sample-data".sh
Downloading sample data...
Downloaded to /tmp/tmp7h06u5np.json
Starting ingestion...
[2022-06-20 16:44:01,149] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:0
[2022-06-20 16:44:01,169] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:1
[2022-06-20 16:44:01,231] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:2
[2022-06-20 16:44:01,277] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:3
[2022-06-20 16:44:01,376] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:4
[2022-06-20 16:44:01,447] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:5
[2022-06-20 16:44:01,537] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:6
[2022-06-20 16:44:01,605] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:7
[2022-06-20 16:44:01,665] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:8
[2022-06-20 16:44:01,724] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:9
[2022-06-20 16:44:01,811] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:10
[2022-06-20 16:44:01,867] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:11
[2022-06-20 16:44:01,909] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:12
[2022-06-20 16:44:02,037] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:13
[2022-06-20 16:44:02,069] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:14
[2022-06-20 16:44:02,140] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:15
[2022-06-20 16:44:02,322] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:16
[2022-06-20 16:44:02,385] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:17
[2022-06-20 16:44:02,408] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:18
[2022-06-20 16:44:02,432] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:19
[2022-06-20 16:44:02,482] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:20
[2022-06-20 16:44:02,497] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:21
[2022-06-20 16:44:02,553] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:22
[2022-06-20 16:44:02,575] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:23
[2022-06-20 16:44:02,599] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:24
[2022-06-20 16:44:02,621] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:25
[2022-06-20 16:44:02,659] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:26
[2022-06-20 16:44:02,709] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:27
[2022-06-20 16:44:02,739] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:28
[2022-06-20 16:44:02,792] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:29
[2022-06-20 16:44:02,819] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:30
[2022-06-20 16:44:02,840] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:31
[2022-06-20 16:44:02,869] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:32
[2022-06-20 16:44:02,893] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:33
[2022-06-20 16:44:02,913] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:34
[2022-06-20 16:44:02,957] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:35
[2022-06-20 16:44:03,017] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:36
[2022-06-20 16:44:03,069] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:37
[2022-06-20 16:44:03,105] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:38
[2022-06-20 16:44:03,124] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:39
[2022-06-20 16:44:03,164] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:40
[2022-06-20 16:44:03,195] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:41
[2022-06-20 16:44:03,224] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:42
[2022-06-20 16:44:03,244] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:43
[2022-06-20 16:44:03,259] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:44
[2022-06-20 16:44:03,291] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:45
[2022-06-20 16:44:03,346] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:46
[2022-06-20 16:44:03,359] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:47
[2022-06-20 16:44:03,418] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:48
[2022-06-20 16:44:03,441] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:49
[2022-06-20 16:44:03,480] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:50
[2022-06-20 16:44:03,520] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:51
[2022-06-20 16:44:03,542] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:52
[2022-06-20 16:44:03,565] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:53
[2022-06-20 16:44:03,597] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:54
[2022-06-20 16:44:03,626] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:55
[2022-06-20 16:44:03,681] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:56
[2022-06-20 16:44:03,716] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:57
[2022-06-20 16:44:03,734] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:58
[2022-06-20 16:44:03,775] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:59
[2022-06-20 16:44:03,809] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:60
[2022-06-20 16:44:03,843] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:61
[2022-06-20 16:44:03,884] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:62
[2022-06-20 16:44:03,906] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:63
[2022-06-20 16:44:03,933] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:64
[2022-06-20 16:44:03,964] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:65
[2022-06-20 16:44:04,020] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:66
[2022-06-20 16:44:04,068] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:67
[2022-06-20 16:44:04,112] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:68
[2022-06-20 16:44:04,131] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:69
[2022-06-20 16:44:04,174] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:70
[2022-06-20 16:44:04,219] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:71
[2022-06-20 16:44:04,290] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:72
[2022-06-20 16:44:04,326] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:73
[2022-06-20 16:44:04,379] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:74
[2022-06-20 16:44:04,407] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:75
[2022-06-20 16:44:04,447] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:76
[2022-06-20 16:44:04,485] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:77
[2022-06-20 16:44:04,530] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:78
[2022-06-20 16:44:04,579] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:79
[2022-06-20 16:44:04,608] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:80
[2022-06-20 16:44:04,685] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:81
[2022-06-20 16:44:04,741] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:82
[2022-06-20 16:44:04,827] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:83
[2022-06-20 16:44:04,842] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:84
[2022-06-20 16:44:04,876] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:85
[2022-06-20 16:44:04,915] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:86
[2022-06-20 16:44:04,962] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:87
[2022-06-20 16:44:05,002] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:88
[2022-06-20 16:44:05,098] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:89
[2022-06-20 16:44:05,132] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:90
[2022-06-20 16:44:05,156] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:91
[2022-06-20 16:44:05,199] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:92
[2022-06-20 16:44:05,221] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:93
[2022-06-20 16:44:05,253] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:94
[2022-06-20 16:44:05,271] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:95
[2022-06-20 16:44:05,299] INFO {datahub.ingestion.run.pipeline:104} - sink wrote workunit file:///tmp/tmp7h06u5np.json:96
Source (file) report:
{'workunits_produced': 97,
'workunit_ids': ['file:///tmp/tmp7h06u5np.json:0',
'file:///tmp/tmp7h06u5np.json:1',
'file:///tmp/tmp7h06u5np.json:2',
'file:///tmp/tmp7h06u5np.json:3',
'file:///tmp/tmp7h06u5np.json:4',
'file:///tmp/tmp7h06u5np.json:5',
'file:///tmp/tmp7h06u5np.json:6',
'file:///tmp/tmp7h06u5np.json:7',
'file:///tmp/tmp7h06u5np.json:8',
'file:///tmp/tmp7h06u5np.json:9',
'file:///tmp/tmp7h06u5np.json:10',
'file:///tmp/tmp7h06u5np.json:11',
'file:///tmp/tmp7h06u5np.json:12',
'file:///tmp/tmp7h06u5np.json:13',
'file:///tmp/tmp7h06u5np.json:14',
'file:///tmp/tmp7h06u5np.json:15',
'file:///tmp/tmp7h06u5np.json:16',
'file:///tmp/tmp7h06u5np.json:17',
'file:///tmp/tmp7h06u5np.json:18',
'file:///tmp/tmp7h06u5np.json:19',
'file:///tmp/tmp7h06u5np.json:20',
'file:///tmp/tmp7h06u5np.json:21',
'file:///tmp/tmp7h06u5np.json:22',
'file:///tmp/tmp7h06u5np.json:23',
'file:///tmp/tmp7h06u5np.json:24',
'file:///tmp/tmp7h06u5np.json:25',
'file:///tmp/tmp7h06u5np.json:26',
'file:///tmp/tmp7h06u5np.json:27',
'file:///tmp/tmp7h06u5np.json:28',
'file:///tmp/tmp7h06u5np.json:29',
'file:///tmp/tmp7h06u5np.json:30',
'file:///tmp/tmp7h06u5np.json:31',
'file:///tmp/tmp7h06u5np.json:32',
'file:///tmp/tmp7h06u5np.json:33',
'file:///tmp/tmp7h06u5np.json:34',
'file:///tmp/tmp7h06u5np.json:35',
'file:///tmp/tmp7h06u5np.json:36',
'file:///tmp/tmp7h06u5np.json:37',
'file:///tmp/tmp7h06u5np.json:38',
'file:///tmp/tmp7h06u5np.json:39',
'file:///tmp/tmp7h06u5np.json:40',
'file:///tmp/tmp7h06u5np.json:41',
'file:///tmp/tmp7h06u5np.json:42',
'file:///tmp/tmp7h06u5np.json:43',
'file:///tmp/tmp7h06u5np.json:44',
'file:///tmp/tmp7h06u5np.json:45',
'file:///tmp/tmp7h06u5np.json:46',
'file:///tmp/tmp7h06u5np.json:47',
'file:///tmp/tmp7h06u5np.json:48',
'file:///tmp/tmp7h06u5np.json:49',
'file:///tmp/tmp7h06u5np.json:50',
'file:///tmp/tmp7h06u5np.json:51',
'file:///tmp/tmp7h06u5np.json:52',
'file:///tmp/tmp7h06u5np.json:53',
'file:///tmp/tmp7h06u5np.json:54',
'file:///tmp/tmp7h06u5np.json:55',
'file:///tmp/tmp7h06u5np.json:56',
'file:///tmp/tmp7h06u5np.json:57',
'file:///tmp/tmp7h06u5np.json:58',
'file:///tmp/tmp7h06u5np.json:59',
'file:///tmp/tmp7h06u5np.json:60',
'file:///tmp/tmp7h06u5np.json:61',
'file:///tmp/tmp7h06u5np.json:62',
'file:///tmp/tmp7h06u5np.json:63',
'file:///tmp/tmp7h06u5np.json:64',
'file:///tmp/tmp7h06u5np.json:65',
'file:///tmp/tmp7h06u5np.json:66',
'file:///tmp/tmp7h06u5np.json:67',
'file:///tmp/tmp7h06u5np.json:68',
'file:///tmp/tmp7h06u5np.json:69',
'file:///tmp/tmp7h06u5np.json:70',
'file:///tmp/tmp7h06u5np.json:71',
'file:///tmp/tmp7h06u5np.json:72',
'file:///tmp/tmp7h06u5np.json:73',
'file:///tmp/tmp7h06u5np.json:74',
'file:///tmp/tmp7h06u5np.json:75',
'file:///tmp/tmp7h06u5np.json:76',
'file:///tmp/tmp7h06u5np.json:77',
'file:///tmp/tmp7h06u5np.json:78',
'file:///tmp/tmp7h06u5np.json:79',
'file:///tmp/tmp7h06u5np.json:80',
'file:///tmp/tmp7h06u5np.json:81',
'file:///tmp/tmp7h06u5np.json:82',
'file:///tmp/tmp7h06u5np.json:83',
'file:///tmp/tmp7h06u5np.json:84',
'file:///tmp/tmp7h06u5np.json:85',
'file:///tmp/tmp7h06u5np.json:86',
'file:///tmp/tmp7h06u5np.json:87',
'file:///tmp/tmp7h06u5np.json:88',
'file:///tmp/tmp7h06u5np.json:89',
'file:///tmp/tmp7h06u5np.json:90',
'file:///tmp/tmp7h06u5np.json:91',
'file:///tmp/tmp7h06u5np.json:92',
'file:///tmp/tmp7h06u5np.json:93',
'file:///tmp/tmp7h06u5np.json:94',
'file:///tmp/tmp7h06u5np.json:95',
'file:///tmp/tmp7h06u5np.json:96'],
'warnings': {},
'failures': {},
'cli_version': '0.8.38.2',
'cli_entry_location': '/usr/local/lib/python3.8/dist-packages/datahub/__init__.py',
'py_version': '3.8.10 (default, Mar 15 2022, 12:22:08) \n[GCC 9.4.0]',
'py_exec_path': '/usr/bin/python3',
'os_details': 'Linux-5.4.0-120-generic-x86_64-with-glibc2.29'}
Sink (datahub-rest) report:
{'records_written': 97,
'warnings': [],
'failures': [],
'downstream_start_time': datetime.datetime(2022, 6, 20, 16, 44, 1, 91061),
'downstream_end_time': datetime.datetime(2022, 6, 20, 16, 44, 5, 299475),
'downstream_total_latency_in_seconds': 4.208414,
'gms_version': 'v0.8.38'}
Pipeline finished successfully producing 97 workunits
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment