Skip to content

Instantly share code, notes, and snippets.

View KevinYang21's full-sized avatar

Kevin Yang KevinYang21

46 followers · 1 following
View GitHub Profile
@KevinYang21
KevinYang21 / gist:daaece6d8ea6ca2714e891caf4d1ca4d
Created March 9, 2019 05:26
template
#### 2018-03-05:
#### 2018-03-06:
#### 2018-03-07:
#### 2018-03-08:
#### 2018-03-09:
@KevinYang21
KevinYang21 / owner_run_as_user_dag_id.task_id
Created August 24, 2018 23:02
amit_pahwa svc_di_data_eng minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_universal
amit_pahwa svc_di_data_eng minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_search
amit_pahwa svc_di_data_eng minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_performance
pai_liu svc_di_trip_platform trip_platform_metrics.wf_jitney_daily_all_logging_itinerary
pai_liu svc_di_trip_platform trip_platform_metrics.wf_jitney_daily_all_logging_systems
pai_liu svc_di_trip_platform trip_platform_metrics.wf_jitney_daily_all_logging_performance
xirui_liu svc_di_china_and_india china_story.jitneyevents__logging_saved
teng_wang svc_di_growth pg_remarketing_events.wf_jitney_events_daily_all
aaron_keys svc_di_data_infra airolap_batch_general.wf_jitney_events_daily_all
shawn_chen svc_di_growth pg_remarketing_dashboard.wf_jitney_events_daily_all
@KevinYang21
KevinYang21 / list_of_corrupted_success
Last active August 24, 2018 21:37
core_payments ps__gold_deposits_mobile_daily_all 2018-08-24 00:00:23 2018-08-24 05:54:56
core_payments ps__gold_payments_web_daily_all 2018-08-24 00:00:23 2018-08-24 05:55:04
core_payments ps__gold_grouptravel_daily_all 2018-08-24 00:00:23 2018-08-24 05:55:07
select wf_jitney_events_daily_all_namespace_logging_selectsupply 2018-08-24 00:01:01 2018-08-24 05:55:20
host_paid_growth_data_integrity wf_jitney_logging_hostgrowthconversionevent 2018-08-24 00:01:13 2018-08-24 05:55:30
china_tns jitney_events_daily_all 2018-08-24 00:01:26 2018-08-24 05:55:43
china_story jitneyevents__logging_saved 2018-08-24 00:01:36 2018-08-24 05:55:53
core_context wf_jitney_daily_all_webpayments 2018-08-24 00:01:29 2018-08-24 05:56:03
core_context wf_jitney_daily_all_payments 2018-08-24 00:01:29 2018-08-24 05:56:18
visitors_tool wf_jitney_ns_rookery 2018-08-24 00:01:59 2018-08-24 05:56:46
@KevinYang21
KevinYang21 / dag_ids
Last active August 24, 2018 19:03
svc_di_data_eng minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_universal
svc_di_data_eng minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_search
svc_di_data_eng minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_performance
svc_di_trip_platform trip_platform_metrics.wf_jitney_daily_all_logging_itinerary
svc_di_trip_platform trip_platform_metrics.wf_jitney_daily_all_logging_systems
svc_di_trip_platform trip_platform_metrics.wf_jitney_daily_all_logging_performance
svc_di_china_and_india china_story.jitneyevents__logging_saved
svc_di_growth pg_remarketing_events.wf_jitney_events_daily_all
svc_di_data_infra airolap_batch_general.wf_jitney_events_daily_all
svc_di_growth pg_remarketing_dashboard.wf_jitney_events_daily_all
@KevinYang21
KevinYang21 / clear
Last active August 24, 2018 08:35
airflow clear -t wf__gold__jitney_events__daily_all__namespace__logging_universal -s 2018-08-23 -e 2018-08-23 -sd /home/kevin_yang/data/airflow/minerva_pipelines/minerva_pipelines.py -d -c minerva_compute;airflow clear -t wf__gold__jitney_events__daily_all__namespace__logging_search -s 2018-08-23 -e 2018-08-23 -sd /home/kevin_yang/data/airflow/minerva_pipelines/minerva_pipelines.py -d -c minerva_compute;airflow clear -t wf__gold__jitney_events__daily_all__namespace__logging_performance -s 2018-08-23 -e 2018-08-23 -sd /home/kevin_yang/data/airflow/minerva_pipelines/minerva_pipelines.py -d -c minerva_compute;airflow clear -t wf_jitney_daily_all_logging_itinerary -s 2018-08-23 -e 2018-08-23 -sd /home/kevin_yang/data/airflow/trip_platform/trip_platform_metrics.py -d -c trip_platform_metrics;airflow clear -t wf_jitney_daily_all_logging_systems -s 2018-08-23 -e 2018-08-23 -sd /home/kevin_yang/data/airflow/trip_platform/trip_platform_metrics.py -d -c trip_platform_metrics;airflow clear -t wf_jitney_daily_all_logging
@KevinYang21
KevinYang21 / list
Created August 24, 2018 07:44
minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_universal partition_name: jitney_events.daily_all/ds={{ ds }}/namespace=logging_universal
minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_search partition_name: jitney_events.daily_all/namespace=logging_search/ds={{ ds }}
minerva_compute.wf__gold__jitney_events__daily_all__namespace__logging_performance partition_name: jitney_events.daily_all/ds={{ ds }}/namespace=logging_performance
trip_platform_metrics.wf_jitney_daily_all_logging_itinerary partition_name: jitney_events.daily_all/namespace=logging_itinerary/ds={{ ds }}
trip_platform_metrics.wf_jitney_daily_all_logging_systems partition_name: jitney_events.daily_all/namespace=logging_systems/ds={{ ds }}
trip_platform_metrics.wf_jitney_daily_all_logging_performance partition_name: jitney_events.daily_all/namespace=logging_performance/ds={{ ds }}
china_story.jitneyevents__logging_saved partition_name: jitney_events.daily_all/ds={{ds}}/namespace=logging_saved
pg_remark
@KevinYang21
KevinYang21 / print_path.py
Created August 16, 2018 00:31
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
@KevinYang21
KevinYang21 / template.sh
Last active July 15, 2018 07:43
Template command to sync old devbox and new devbox
# To use the template, replace <instance_id> with the real devbox instance ids and run them on your local machine.
USERNAME=`whoami` # your SSH username
NEW_DEVBOX=<instance_id>.inst.aws.us-east-1.prod.musta.ch # hostname of the new devbox
OLD_DEVBOX=<instance_id>.inst.aws.us-east-1.prod.musta.ch # hostname of the old devbox
ssh -A $NEW_DEVBOX "sudo SSH_AUTH_SOCK="\$"SSH_AUTH_SOCK rsync -avzhe 'ssh -o StrictHostKeyChecking=no' —rsync-path='sudo rsync' $USERNAME@$OLD_DEVBOX:/home/$USERNAME/ /home/$USERNAME/"
@KevinYang21
KevinYang21 / airflow_2463_test_dag.py
Created May 24, 2018 09:19
test DAG
# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#