Skip to content

Instantly share code, notes, and snippets.

@troyharvey
Last active January 8, 2024 00:38
Show Gist options
  • Save troyharvey/bae82c86c27a3aa539dea83857ee9ecd to your computer and use it in GitHub Desktop.
Save troyharvey/bae82c86c27a3aa539dea83857ee9ecd to your computer and use it in GitHub Desktop.
Deploy Google Cloud Functions: GitLab CI/CD Pipeline Config File
# Update Jan 2024
# Deploying Cloud Functions is much simpler than it was 6 years ago.
# I'm leaving the gist in it's original 2018 state for now,
# but skip the the recent comments below for a simpler solution.
variables:
GCP_ZONE: us-central1-a
stages:
- npm-install
- push
npm-install:
image: node:8-alpine
stage: npm-install
only:
- master
script:
- cd replaceWithYourCloudFunction
- npm install
artifacts:
paths:
- replaceWithYourCloudFunction/node_modules/
push:
stage: push
image: docker:latest
only:
- master
dependencies:
- npm-install
script:
# Install CA certs, openssl to https downloads, python for gcloud sdk
- apk add --update make ca-certificates openssl python
- update-ca-certificates
# Download and install Google Cloud SDK
- wget https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz
- tar zxvf google-cloud-sdk.tar.gz && ./google-cloud-sdk/install.sh --quiet --usage-reporting=false --path-update=true
- PATH="google-cloud-sdk/bin:${PATH}"
- gcloud --quiet components update
- gcloud components install beta
# Authenticate using the service account stored here: https://gitlab.com/groups/{YOUR_GITLAB_PROJECT}/-/settings/ci_cd
- echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
- gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json
# Deploy
- gcloud beta functions deploy replaceWithYourCloudFunction --source=./replaceWithYourCloudFunction --trigger-http
@hugosantos-io
Copy link

That helped me a lot. Tks.

@andrzj
Copy link

andrzj commented Sep 3, 2018

Would you mind to explain why you need docker:dind as service and why you used docker as image for deploy the Function? Couldn't you just use a Debian image for that and remove the service?

@GIBZ-GIPE
Copy link

Hi @troyharvey
Thanks a lot for this gist. It's exactly what I was looking for.
Unfortunately I don't have a lot of experience with GitLab CI/CD and therefore couldn't manage to get this working.

Since my project only consists of a simple main.py file along with a requirements.txt and some additional files (README.md, .gitignore, ...) I suppose I won't need all the npm-install stuff in your receipt? I deleted it, pushed changes to my repository and got the Job running. It failed although :-(

My jobs always end with a error message like this:

$ echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
$ gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json
Activated service account credentials for: [gibz-one50@appspot.gserviceaccount.com]
$ gcloud config set project $GCP_PROJECT_ID
Updated property [core/project].
$ gcloud functions deploy send_2fa_sms --trigger-http --runtime=python37 --region=${GCP_ZONE}
ERROR: (gcloud.functions.deploy) Failed to upload the function source code to signed url: https://storage.googleapis.com/gcf-upload-europe-west1-f84788ab-087c-4c71-b28c-1d04b7e3fd74/202eed5b-3a52-4dd6-bff4-fed4dbecc8e7.zip?GoogleAccessId=service-576747503219@gcf-admin-robot.iam.gserviceaccount.com&Expires=1541631358&Signature=pWvPNbexuu7wpvJo%2F9c4Ppqp%2BRU%2B4L3JbGmtWljzT7has%2BuTIrTsLYuCmmQMh3fucL3BcW%2FQzLuOPrBjSWRbBk5I7gXMk9Jj89icBkLOqg%2Fh2WrjyY0Uger0PXv9BJwOKsp4EIlOE%2BFPPcc5q0O8CmXFhguyuNDCYxS%2Bk7a14BcVWkpdS3P9Y0EzD%2FTfNpmqS4iXFvYVHfx%2F%2BowB9H4TCMzfDRUeBI53mMbmHNkvcJmhcBg03nnmn1BOku1ibgXI7sl%2Bhiv%2B9qL1nJNWaRmAHhCBTC7%2BJowTsDdNtnO6Q0BVF%2Fv4tOODEewJnSp06R9ihgMh9BJrJRQs4vxYpVeovQ%3D%3D. Status: [400:<?xml version='1.0' encoding='UTF-8'?><Error><Code>EntityTooLarge</Code><Message>Your proposed upload is larger than the maximum object size specified in your Policy Document.</Message><Details>Content-length exceeds upper bound on range</Details></Error>]
ERROR: Job failed: exit code 1

Do you have any hints for me what I could do to get it running? Where do you suppose lies the problem?

Kind regards,
Peter

@viggy28
Copy link

viggy28 commented Jan 3, 2019

Thanks.

You don't need Dind for this. Also, you don't have to set project (when you authenticate using Service Account it automatically knows the project).

@jannakha
Copy link

jannakha commented Jun 3, 2019

bingo - there's a container with SDK from Google - https://hub.docker.com/r/google/cloud-sdk

@JannikZed
Copy link

You need to add a --quiet also here:
- tar zxvf google-cloud-sdk.tar.gz && ./google-cloud-sdk/install.sh --usage-reporting=false --path-update=true
(at least I needed to do so)

@troyharvey
Copy link
Author

troyharvey commented Jun 20, 2019

Thanks @JannikZed and @viggy28. Does it look consistent with what you are using in production now?

  1. Removed the dind service
  2. Added --quiet on the install.sh step
  3. Removed the gcloud config set project $GCP_PROJECT_ID

@JannikZed
Copy link

looks like our setup now, thank you :) we use gsutil afterwards to load our machine-learning models from a storage bucket. Thank you for this gist. Reduced my effort a lot!!

@samuraitruong
Copy link

hi @troyharvey,

Do you know why I got this error

constants.js
constants.js.map
db
dialogFlowApp.js
dialogFlowApp.js.map
enums.js
enums.js.map
index.js
index.js.map
intents
models
templates
utils

$ gcloud beta functions deploy fulfillment --source=./voice/google-assistant/webhook/functions --trigger-http --runtime nodejs8
Created .gcloudignore file. See `gcloud topic gcloudignore` for details.
Deploying function (may take a while - up to 2 minutes)...
..............failed.
ERROR: (gcloud.beta.functions.deploy) OperationError: code=3, message=Function failed on loading user code. Error message: File lib/index.js that is expected to define function doesn't exist.

inside source folder, I have index.js file but when gcloud deploy I got above error message

@muratcorlu
Copy link

This should be enough for deployment:

deploy to gcloud:
  image: google/cloud-sdk:latest
  script:
    - echo $GCLOUD_SERVICE_KEY > /tmp/$CI_PIPELINE_ID.json
    - gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
    - gcloud functions deploy function-name --runtime nodejs8 --entry-point handler --trigger-http
    - rm /tmp/$CI_PIPELINE_ID.json

@jligeza
Copy link

jligeza commented Oct 17, 2019

@muratcorlu
If you set the service key as a file variable instead of a normal variable, then you don't need to create the tmp file, removing first and last step.

@muratcorlu
Copy link

@jligeza Good information, thanks. So it'll be that much simple:

deploy to gcloud:
  image: google/cloud-sdk:latest
  script:
    - gcloud auth activate-service-account --key-file $GCLOUD_SERVICE_KEY_FILE
    - gcloud functions deploy function-name --runtime nodejs8 --entry-point handler --trigger-http

@linceaerian
Copy link

Hello @muratcorlu,

First thanks for sharing the CI/CD :)
I'll share my tests at this moment to help if other needs them.

For me, the project ID is still mandatory, else I get a:

ERROR: (gcloud.app.deploy) The required property [project] is not currently set.
You may set it for your current workspace by running:
  $ gcloud config set project VALUE
or it can be set temporarily by the environment variable [CLOUDSDK_CORE_PROJECT]
ERROR: Job failed: exit code 1

(Which is still strange, cause It's clearly indicated in the JSON as project_id).

Also not sure of what @jiligeza said because i'm not sure of what a "file variable" is (maybe a setting in giltab CI I did not found) and passing it without create a file still cause an error as:

 ERROR: (gcloud.auth.activate-service-account) argument --key-file: expected one argument
Usage: gcloud auth activate-service-account [ACCOUNT] --key-file=KEY_FILE [optional flags]

For the moment i'm fighting to find why the CI/CD successfully deploy on staging, but not on prod. Same key, same code and I get a:

ERROR: (gcloud.auth.activate-service-account) Could not read json file /tmp/124294711.json: Expecting value: line 2 column 1 (char 1)
ERROR: Job failed: exit code 1

While on dev branch

$ gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
Activated service account credentials for: [gitlab@[MASKED].iam.gserviceaccount.com]

@linceaerian
Copy link

Finally got it, problem of variable protection, https://gitlab.com/gitlab-com/support-forum/issues/4660 gave me the lead I was missing.

Got working App Engine CI/CD with:

deploy:
  stage: deploy
  image: google/cloud-sdk:alpine
  only:
    - dev
  before_script:
    - echo $FIREBASE_KEY > ${CI_PROJECT_DIR}/key.json
    - echo $GCLOUD_SERVICE_KEY > /tmp/$CI_PIPELINE_ID.json
    - gcloud auth activate-service-account --key-file /tmp/$CI_PIPELINE_ID.json
    - gcloud config set project $CLOUDSDK_CORE_PROJECT
  script:
    - cd ${CI_PROJECT_DIR} 
    - gcloud app deploy
  allow_failure: false

And Gitlab CI/CD conf as:

image

@muratcorlu
Copy link

@linceaerian File variable is the type of Pipeline Variable as shown below:

image

Content will be same. In that case your variable will hold file path of the temporary file that is autmatically created by Gitlab CI. So you will not need to manually create a file and put the content in it.

@linceaerian
Copy link

@muratcorlu, Thanks a lot, I found it later in the day, i've already switched it for the credential part :)

@cjoshmartin
Copy link

This was a helpful page if you are not familiar with gcloud deploy https://cloud.google.com/sdk/gcloud/reference/functions/deploy

@EchedelleLR
Copy link

This + the comments helped me as a template. Thank you so much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment