Skip to content

Instantly share code, notes, and snippets.

@KellenJohn
KellenJohn / bq_tools.py
Last active September 18, 2023 05:44
架構自己的 BigQuery 易用性工具
import pandas as pd
from google.api_core.exceptions import GoogleAPICallError, BadRequest, NotFound
from google.cloud import bigquery, bigquery_storage, storage
class BQDataFetcher:
def __init__(self, project, dataset, table):
self.project = project
self.dataset = dataset
self.table = table
self.bqclient = bigquery.Client(project=project)
@KellenJohn
KellenJohn / classification_model_eval_metrics.py
Created September 15, 2023 15:26
Vertex Kubeflow pipeline - model eval component
@component(
base_image="gcr.io/deeplearning-platform-release/tf2-cpu.2-3:latest",
output_component_file="tabular_eval_component.yaml",
packages_to_install=["google-cloud-aiplatform"],
)
def classification_model_eval_metrics(
project: str,
location: str, # "us-central1",
api_endpoint: str, # "us-central1-aiplatform.googleapis.com",
thresholds_dict_str: str,
@KellenJohn
KellenJohn / demo-kfp-pipeline.py
Created September 15, 2023 15:24
Vertex Kubeflow pipeline demo
@pipeline(name="automl-tab-beans-training-v2",
pipeline_root=PIPELINE_ROOT)
def pipeline(
bq_source: str = "bq://aju-dev-demos.beans.beans1",
display_name: str = DISPLAY_NAME,
project: str = PROJECT_ID,
gcp_region: str = "us-central1",
api_endpoint: str = "us-central1-aiplatform.googleapis.com",
thresholds_dict_str: str = '{"auRoc": 0.95}',
):
@KellenJohn
KellenJohn / kfp_mlflow_integration.py
Last active September 11, 2023 09:06
外掛 KFP 至 MLflow
import kfp
from kfp import dsl
import mlflow
# 準備要登記實驗結果的 function 讓 DA 可以照 DA 主管想要的模型管理資訊放入
def register_experiment_results_in_mlflow(experiment_name, metric_name, metric_value):
# Initialize MLflow
mlflow.start_run()
# Set the experiment name
@KellenJohn
KellenJohn / main.py
Last active August 30, 2023 08:39
cloud funtion main.py
def hello_world(request):
"""Responds to any HTTP request.
Args:
request (flask.Request): HTTP request object.
Returns:
The response text or any set of values that can be turned into a
Response object using
`make_response <http://flask.pocoo.org/docs/1.0/api/#flask.Flask.make_response>`.
"""
request_json = request.get_json()
@KellenJohn
KellenJohn / deploy.sh
Created August 30, 2023 07:58
gcloud build script
#!/bin/bash
# method 1
gcloud functions deploy demo --entry-point hello_world --runtime python39 --trigger-http --allow-unauthenticated
# method 2
gcloud functions deploy demo2 \
--entry-point hello_world \
--runtime python39 \
--trigger-http \
--allow-unauthenticated \
@KellenJohn
KellenJohn / cloudbuild.yaml
Last active August 30, 2023 09:24
demo cloudbuild 執行檔
steps:
# 檢查代碼,安裝必要的工具
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- |
gcloud components install beta
# 部署 Cloud Function
@KellenJohn
KellenJohn / inheritance and arguments.md
Last active April 17, 2020 05:49
Clean Code - Ch3

Composition and Inheritance

重點說明

  • 繼承很有用,但是還有一種能實現相同效果的方法,就是使用其他類和模組,而不是依賴隱式繼承。
  • 因為物件的三大特性都涉及編寫新的代碼來替換或更改函數的功能
    • 但這可以很容易通過調用 class 模組中的函數來接著使用
    • a.k.a 不一定要父子 is-a 關係,而是使用 has-a 關係,其中 Child(has-a) Other 來完成子類的工作
  • 何時使用組合與繼承
@KellenJohn
KellenJohn / speedup.py
Created March 28, 2020 08:58 — forked from wzpan/speedup.py
A python script to speed up the rendering process of Hexo 3.
#!/usr/bin/python2
'''
SYNOPSIS:
$ python speedup.py -f FILE
or
$ python speedup.py -d DIR
'''
@KellenJohn
KellenJohn / timestamp.py
Created March 28, 2020 08:57 — forked from wzpan/timestamp.py
Python - timestamp
import datetime
# 2012-12-15 10:14:51.898000
print datetime.datetime.utcnow()
# The now differs from utcnow as expected
# otherwise they work the same way:
# 2012-12-15 11:15:09.205000
print datetime.datetime.now()