Skip to content

Instantly share code, notes, and snippets.

@AseiSugiyama

AseiSugiyama/vertex-ai-handson-gcpug-joshikai.ipynb Secret

Last active April 9, 2022 00:21
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AseiSugiyama/8d00ce5eb2784f3516990cda5cabf074 to your computer and use it in GitHub Desktop.
Save AseiSugiyama/8d00ce5eb2784f3516990cda5cabf074 to your computer and use it in GitHub Desktop.
Download ZIP
vertex-ai-handson-gcpug-joshikai.ipynb
Display the source blob
Display the rendered blob
Raw
vertex-ai-handson-gcpug-joshikai.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "vertex-ai-handson-gcpug-joshikai.ipynb",
"provenance": [],
"collapsed_sections": [],
"authorship_tag": "ABX9TyNmg/3yjsgy7Hpmjiae/b5q",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/AseiSugiyama/8d00ce5eb2784f3516990cda5cabf074/vertex-ai-handson-gcpug-joshikai.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# 第15回 GCPUG 女子会 Vertex AI ではじめる MLOps ハンズオン用補足資料\n",
"\n",
"当日使用する予定の [QwikLab Vertex Pipelines: Qwik Start](https://www.cloudskillsboost.google/focuses/21234?locale=ja&parent=catalog) のうち、ハマリそうな箇所についての補足資料です。\n",
"\n",
"## 最初のパイプラインを作成する\n",
"\n",
"### compiler.Compiler().compile でエラーメッセージが出る\n",
"\n",
"次のコードを実行するとエラーメッセージが表示されます。\n",
"\n",
"```python\n",
"compiler.Compiler().compile(\n",
" pipeline_func=intro_pipeline, package_path=\"intro_pipeline_job.json\"\n",
")\n",
"```\n",
"\n",
"```console\n",
"/home/jupyter/.local/lib/python3.7/site-packages/kfp/v2/compiler/compiler.py:1281: FutureWarning: APIs imported from the v1 namespace (e.g. kfp.dsl, kfp.components, etc) will not be supported by the v2 compiler since v2.0.0\n",
" category=FutureWarning,\n",
"```\n",
"\n",
"今回のイベントにおいて、このメッセージは無視して構いません。このメッセージは KFP v2 である `kfp.v2.compiler` `Compiler` クラスの `compile` メソッドを用いた際に表示されます。このメッセージは将来への注意喚起用で、 将来的には `kfp.v2.compiler` 配下の `Compiler` を使った場合には、パイプラインに KFP v1 用の名前空間 `kfp.dsl` や `kfp.components`が使えなくなることを示しています。\n",
"\n",
"たとえば、QwikLab の資料のうち、`@dsl.pipeline` は v1 を前提としたもので、ここで用いている `compiler` は v2 を前提としたもの (`kfp.v2.compiler`) です。 (KFP SDK v1.8.12 時点)\n",
"\n",
"KFP SDK v2 を用いる場合には次のようにします。"
],
"metadata": {
"id": "azdW3dobF6qI"
}
},
{
"cell_type": "code",
"source": [
"@kfp.v2.dsl.pipeline(\n",
" name=\"hello-world\",\n",
" description=\"An intro pipeline\",\n",
" pipeline_root=PIPELINE_ROOT,\n",
")\n",
"# ここでパラメータ `text` と `emoji_str` を変更することでパイプラインの出力を変更可能\n",
"def intro_pipeline(text: str = \"Vertex Pipelines\", emoji_str: str = \"sparkles\"):\n",
" product_task = product_name(text)\n",
" emoji_task = emoji(emoji_str)\n",
" consumer_task = build_sentence(\n",
" product_task.output,\n",
" emoji_task.outputs[\"emoji\"],\n",
" emoji_task.outputs[\"emoji_text\"],\n",
" )\n",
"\n",
"compiler.Compiler().compile(\n",
" pipeline_func=intro_pipeline, package_path=\"intro_pipeline_job.json\"\n",
")"
],
"metadata": {
"id": "22ByInHlF6Tt"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"### AIPlatformClient の初期化でエラーメッセージが出る\n",
"\n",
"次のコードを実行するとエラーメッセージが出力されます。\n",
"\n",
"```python\n",
"api_client = AIPlatformClient(\n",
" project_id=PROJECT_ID,\n",
" region=REGION,\n",
")\n",
"```\n",
"\n",
"```\n",
"/home/jupyter/.local/lib/python3.7/site-packages/kfp/v2/google/client/client.py:173: FutureWarning: AIPlatformClient will be deprecated in v2.0.0. Please use PipelineJob https://googleapis.dev/python/aiplatform/latest/_modules/google/cloud/aiplatform/pipeline_jobs.html in Vertex SDK. Install the SDK using \"pip install google-cloud-aiplatform\"\n",
" category=FutureWarning,\n",
"```\n",
"\n",
"これはここで用いている `kfp.v2.google.client.AIPlatformClient` (正確には `kfp.v2.google` 配下すべて) が将来削除予定のため表示されているエラーメッセージです。これを解消するためには次を実行します。\n",
"\n",
"用いている [Vertex SDK](https://github.com/googleapis/python-aiplatform) が古いので、念のため最新のものにします。"
],
"metadata": {
"id": "s3s7L0a_IQ_P"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "EZXOsL7wF1yr"
},
"outputs": [],
"source": [
"!pip install --user google-cloud-aiplatform --upgrade"
]
},
{
"cell_type": "markdown",
"source": [
"最新のものを利用するためには Jupyter Lab のカーネルの再起動が必要です。"
],
"metadata": {
"id": "5kNsOBeSJdAN"
}
},
{
"cell_type": "code",
"source": [
"import os\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" # インストール後カーネルを自動で再起動\n",
" import IPython\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
],
"metadata": {
"id": "FPrg2NxsI9TT"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"カーネルを再起動したため、そこまでのすべての操作をもう一度行います。"
],
"metadata": {
"id": "fPIIbJChJqVO"
}
},
{
"cell_type": "code",
"source": [
"import os\n",
"PROJECT_ID = \"\"\n",
"# gcloud から Google Cloud プロジェクト ID を取得\n",
"if not os.getenv(\"IS_TESTING\"):\n",
" shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
" PROJECT_ID = shell_output[0]\n",
" print(\"Project ID: \", PROJECT_ID)"
],
"metadata": {
"id": "UdjbyMXEJpaz"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"from typing import NamedTuple\n",
"import kfp\n",
"# from kfp import dsl <- ここだけ v1 なので将来的には不要です\n",
"from kfp.v2 import compiler\n",
"from kfp.v2.dsl import (Artifact, Dataset, Input, InputPath, Model, Output,\n",
" OutputPath, ClassificationMetrics, Metrics, component)\n",
"from kfp.v2.google.client import AIPlatformClient\n",
"from google.cloud import aiplatform\n",
"from google_cloud_pipeline_components import aiplatform as gcc_aip"
],
"metadata": {
"id": "KrlNFq4lJ6Mw"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"BUCKET_NAME=\"gs://\" + PROJECT_ID + \"-bucket\""
],
"metadata": {
"id": "q-2uhpl9KD5K"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"PATH=%env PATH\n",
"%env PATH={PATH}:/home/jupyter/.local/bin\n",
"REGION=\"us-central1\"\n",
"PIPELINE_ROOT = f\"{BUCKET_NAME}/pipeline_root/\"\n",
"PIPELINE_ROOT"
],
"metadata": {
"id": "MYYM5cDeKHN8"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"@component(base_image=\"python:3.9\", output_component_file=\"first-component.yaml\")\n",
"def product_name(text: str) -> str:\n",
" return text"
],
"metadata": {
"id": "5c-hxHY6KKDs"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# この方法は現状 v2 では提供されていません\n",
"# product_name_component = kfp.components.load_component_from_file('./first-component.yaml')"
],
"metadata": {
"id": "9FIZJRHIKMUF"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"@component(packages_to_install=[\"emoji\"])\n",
"def emoji(\n",
" text: str,\n",
") -> NamedTuple(\n",
" \"Outputs\",\n",
" [\n",
" (\"emoji_text\", str), # 戻り値のパラメータ\n",
" (\"emoji\", str),\n",
" ],\n",
"):\n",
" import emoji\n",
" emoji_text = text\n",
" emoji_str = emoji.emojize(':' + emoji_text + ':', use_aliases=True)\n",
" print(\"output one: {}; output_two: {}\".format(emoji_text, emoji_str))\n",
" return (emoji_text, emoji_str)"
],
"metadata": {
"id": "Q_MBKaheKRP3"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"@component\n",
"def build_sentence(\n",
" product: str,\n",
" emoji: str,\n",
" emojitext: str\n",
") -> str:\n",
" print(\"We completed the pipeline, hooray!\")\n",
" end_str = product + \" is \"\n",
" if len(emoji) > 0:\n",
" end_str += emoji\n",
" else:\n",
" end_str += emojitext\n",
" return(end_str)"
],
"metadata": {
"id": "z6TAuh6tKSm_"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"@kfp.v2.dsl.pipeline(\n",
" name=\"hello-world\",\n",
" description=\"An intro pipeline\",\n",
" pipeline_root=PIPELINE_ROOT,\n",
")\n",
"# ここでパラメータ `text` と `emoji_str` を変更することでパイプラインの出力を変更可能\n",
"def intro_pipeline(text: str = \"Vertex Pipelines\", emoji_str: str = \"sparkles\"):\n",
" product_task = product_name(text)\n",
" emoji_task = emoji(emoji_str)\n",
" consumer_task = build_sentence(\n",
" product_task.output,\n",
" emoji_task.outputs[\"emoji\"],\n",
" emoji_task.outputs[\"emoji_text\"],\n",
" )\n",
"\n",
"compiler.Compiler().compile(\n",
" pipeline_func=intro_pipeline, package_path=\"intro_pipeline_job.json\"\n",
")\n",
"\n",
"# 現状提供されている実装の問題で、FutureWarnning が発生します"
],
"metadata": {
"id": "jM143mWTKZOP"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"最後に、作成したパイプラインを Vertex Pipelines にデプロイします。\n"
],
"metadata": {
"id": "K2-QnA5bKfT7"
}
},
{
"cell_type": "code",
"source": [
"from google.cloud.aiplatform.pipeline_jobs import PipelineJob\n",
"\n",
"pipeline_job = PipelineJob(\n",
" display_name=\"hello-world-pipeline\",\n",
" template_path=\"intro_pipeline_job.json\",\n",
" project=PROJECT_ID,\n",
" location=REGION,\n",
")"
],
"metadata": {
"id": "cV_Oxb89JBDR"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"pipeline_job.submit()"
],
"metadata": {
"id": "FnaJgWDmJIxX"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"パイプラインをデプロイする方法については [Google Cloud の Run a pipeline](https://cloud.google.com/vertex-ai/docs/pipelines/run-pipeline#vertex-ai-sdk-for-python) を参照してください。今回用いた `PipelineJob` の詳細については [Vertex SDK のドキュメント](https://googleapis.dev/python/aiplatform/latest/_modules/google/cloud/aiplatform/pipeline_jobs.html)を参照してください。\n",
"\n",
"また、今回用いた `kfp.v2` という名前空間は将来的に `kfp` のみに変更される予定です。変更の詳細は[[SDK] KFP SDK V2 Namespace Plan · Issue #7238 · kubeflow/pipelines](https://github.com/kubeflow/pipelines/issues/7238) を参照してください。"
],
"metadata": {
"id": "WHdwO6iMKy20"
}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment