Skip to content

Instantly share code, notes, and snippets.

@Sillians
Created September 4, 2020 12:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Sillians/ca26e8cffb9a9b2715d079a275dfc296 to your computer and use it in GitHub Desktop.
Save Sillians/ca26e8cffb9a9b2715d079a275dfc296 to your computer and use it in GitHub Desktop.
Error Encountered whilst Deploying Kubeflow on GCP
These were the steps I took :
# make a bin folder to contain the kfctl
mkdir ~/bin
# download kfctl from the releases page
wget -P /tmp https://github.com/kubeflow/kfctl/releases/download/v1.0/kfctl_v1.0-0-g94c35cf_linux.tar.gz
# unzip to the bin folder
tar -xvf /tmp/kfctl_v1.0-0-g94c35cf_linux.tar.gz -C ~/bin
# add the kfctl binary to the PATH
export PATH=$PATH:~/bin
## Configure gcloud default values for zone and project:
# Set your GCP project ID and the zone where you want to create
# the Kubeflow deployment:
export PROJECT=<your GCP project ID>
export ZONE=<your GCP zone>
gcloud config set project ${PROJECT}
gcloud config set compute/zone ${ZONE}
Step 6: Select the KFDef spec to use as the basis for your deployment
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml"
Step 7: Create environment variables containing the OAuth client ID and secret that you created earlier
export CLIENT_ID=<CLIENT_ID from OAuth page>
export CLIENT_SECRET=<CLIENT_SECRET from OAuth page>
Step 8: Pick names for your Kubeflow deployment and directory for your configuration:
export KF_NAME=<your choice of name for the Kubeflow deployment>
export BASE_DIR=<path to a base directory>
export KF_DIR=${BASE_DIR}/${KF_NAME}
Deploy Kubeflow with Customization
Step 1: Download the KFDef file to your local directory to allow modification:
export CONFIG_FILE="kfdef.yaml"
mkdir -p ${KF_DIR}
cd ${KF_DIR}
curl -L -o ${CONFIG_FILE} https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml
Step 2: Edit the KFDef spec in the yaml file. The following commands shows you how to set values in the configuration file using yq:
yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.project' ${PROJECT}
yq w -i ${CONFIG_FILE} 'spec.plugins[0].spec.zone' ${ZONE}
yq w -i ${CONFIG_FILE} 'metadata.name' ${KF_NAME}
Step 3: Run the kfctl build command to generate kustomize and GCP deployment manager configuration files for your deployment:
cd ${KF_DIR}
kfctl build -V -f ${CONFIG_FILE}
# run the kfctl apply command to deploy Kubeflow:
cd ${KF_DIR}
kfctl apply -V -f ${CONFIG_FILE}
# So, when I run the last command, I get the following errors below:
ERRO[0034] Updating new-deployment error: &{Code:RESOURCE_ERROR Location:/deployments/new-deployment/resources/new-deployment Message:{"ResourceType":"gcp-types/container-v1
beta1:projects.locations.clusters","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"message":"Insufficient project quota to satisfy request: resource \"CPUS_ALL
_REGIONS\": request requires '16.0' and is short '4.0'. project has a quota of '12.0' with '12.0' available. View and manage quotas at https://console.cloud.google.com/iam-a
dmin/quotas?usage=USED&project=my-kubeflow-deployment.","status":"PERMISSION_DENIED","statusMessage":"Forbidden","requestPath":"https://container.googleapis.com/v1beta1/proj
ects/my-kubeflow-deployment/locations/us-central1-a/clusters","httpMethod":"POST"}} ForceSendFields:[] NullFields:[]} filename="gcp/gcp.go:386"
Error: failed to apply: (kubeflow.error): Code 500 with message: coordinator Apply failed for gcp: (kubeflow.error): Code 400 with message: gcp apply could not update depl
oyment manager Error could not update deployment manager entries; Updating new-deployment error(400): BAD REQUEST
Usage:
kfctl apply -f ${CONFIG} [flags]
Flags:
-f, --file string Static config file to use. Can be either a local path:
export CONFIG=./kfctl_gcp_iap.yaml
or a URL:
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_istio_dex.v1.0.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws.v1.0.0.yaml
export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.0.yaml
kfctl apply -V --file=${CONFIG}
-h, --help help for apply
-V, --verbose verbose output default is false
failed to apply: (kubeflow.error): Code 500 with message: coordinator Apply failed for gcp: (kubeflow.error): Code 400 with message: gcp apply could not update deployment
manager Error could not update deployment manager entries; Updating new-deployment error(400): BAD REQUEST
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment