Skip to content

Instantly share code, notes, and snippets.

@farhangithub27

farhangithub27/mp-etl data pipeline issues and resolutions.

Last active March 4, 2020 04:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save farhangithub27/e23a8b440f78c0f2bf4b2b4ad7f09b8f to your computer and use it in GitHub Desktop.
Save farhangithub27/e23a8b440f78c0f2bf4b2b4ad7f09b8f to your computer and use it in GitHub Desktop.
Download ZIP
Issues encountered with mp-etl data pipeline and resolution.
Raw
mp-etl data pipeline issues and resolutions.
1. Vertical value in lake verticals table.
If even one vertical has this value missing it will affect all verticals.
Example:
Bhanu reported that myer2 is not getting any data in datamart tables after its mp-etl pipeline deployment.
She also pointed that other verticals have some missing data too like from 1st of march. e.g. Analytic table etc
She pointed out following tables
BE_AU
Analytic – Feb 1st
Search – Feb 15th
BE_UK
Purchasing – Feb 8th
Analytic – Feb 1st
Search – Feb 15th
Fishbrain
Analytic Feb 1st
BobJane
Search Feb 15th
CartridgeWorld
Analytic Feb 1st
1:44
Jayco
1:44
Analytic Feb 1st
1:44
Search Feb 15th
Issue was that myer2 doesn't have value 'myer2' in its lake.verticals table.
After fixing that myer2 all tables related with datapipeline started getting values.
and also other verticals analytic tables got values from 1st amrch also.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment