Google App Engine Service Accounts that work in local development: A guide for the lost and weary

appengine-service-accounts-on-devserver.md

It is easy to get service accounts working with App Engine's app_devserver.py - once you know how.

On the way there, you might have pulled out all your hair following one documentation dead end after another, trying to piece together the right information.

Here are the steps you need to take, in exact order, to get this working. Once you follow these steps, you'll be able to use service accounts in local development, so that you can interact with Google APIs (e.g.: Spreadsheet, Calendar) in a way that is consistent with the deployment environment on App Engine.

In order to follow the instructions, you'll be better off using the latest UI for Google Cloud projects. Older interfaces (such as the dedicated App Engine dashboard) have things in different places, under different names, etc. It is a world of pain there.

Also note that I've tested this on several 1.9.x releases of App Engine; I can't confirm the behaviour of earlier releases.

Do it

1. Ensure you have Google's API client installed and available.

   pip install google-api-python-client

2. Add pycrypto to libraries in your app.yaml

   app.yaml

   libraries:

   • name: pycrypto version: latest

3. Go to cloud.google.com, and:

   • go to the main dashboard for your particular project
   • go to APIS & AUTH > Credentials > Create new Client ID > Service Account > Create Client ID

4. After completing 3, a new Service Account client will appear with your other credentials, and a json file will be downloaded automatically. You do not need this JSON file. Instead:

   • click "Generate new P12 key" under your new Service Account.

This will download another key, not surprisingly, with a .p12 suffix.

5. Now the .p12 key needs to be converted to .pem format - the format that App Engine's devserver requires. The code snippet below assumes the working directory is the location of the .p12 key. The .p12 key is called "secret.p12", the password for the key is "notasecret", and the output file is "secret.pem".

   cat secret.p12 | openssl pkcs12 -nodes -nocerts -passin pass:notasecret | openssl rsa > secret.pem

6. Now, take note of the email adddress of the service account you just created, and the path to the .pem file you just created. Run your devserver with these, like so:

   dev_appserver.py --appidentity_email_address {EMAIL_ADDRESS} --appidentity_private_key_path {PATH_TO_KEY} {PATH_TO_CODE}

7. When the development server is run with these flags, and the values are valid, service accounts can be used in development. Here is a short example, using AppAssertionCredentials to authorize gspread: a pleasant-to-use client for the Google Spreadsheet API

   from somewhere import config from oauth2client import appengine import gspread

   scope = config['SERVICE_ACCOUNT']['scopes']['spreadsheets']

   credentials = appengine.AppAssertionCredentials(scope=scope)

   client = gspread.authorize(credentials)

This example assumes configuration values are available via the config dict.

See any other examples in Google's API documentation for different implementations.

The point here being, that however you implement your code, if it works in production, it will also now work in development if you run dev_appserver.py with the correct appidentity flags.

I greatly appreciate this explanation. I spent the past few hours poring over documentation and experimenting and was baffled as to why I was receiving an 'Invalid Credentials' response from the Google calendar API in my dev environment. I followed the steps you outlined here and am happy that the API requests now function as expected with my service account in dev. Thanks!

Good to hear @binarynate. I'm glad it helped you, and I sure as hell wish I had a similar guide when I started trying to solve this frustrating problem :).

Fantastic. Thanks for this. Likely saved me hours of head banging :)

Could someone explain what {PATH_TO_CODE} means?

I am doing this as extra flags with the GUI GoogleAppEngineLauncher. This is what I get:

*** Running dev_appserver with the following flags:
    --skip_sdk_update_check=yes --port=12080 --admin_port=8004 --appidentity_email_address {EMAIL} --appidentity_private_key_path {PEM_FILE}
Python command: /usr/bin/python2.7
Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/dev_appserver.py", line 83, in <module>
    _run_file(__file__, globals())

Could anyone help me figure this out?

@pwalsh You are a life saver!! Thanks a lot!

@pwalsh No matter what I seem to do, I constantly get no module named "_winreg" Any ideas?

ERROR    2017-05-12 16:45:08,460 wsgi.py:263]
Traceback (most recent call last):
  File "C:\Users\Lance\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "C:\Users\Lance\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "C:\Users\Lance\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "C:\Users\Lance\Desktop\untitled2\main.py", line 18, in <module>
    import cal
  File "C:\Users\Lance\Desktop\untitled2\cal.py", line 1, in <module>
    import gspread
  File "C:\Users\Lance\Desktop\untitled2\lib\gspread\__init__.py", line 21, in <module>
    from .client import Client, authorize
  File "C:\Users\Lance\Desktop\untitled2\lib\gspread\client.py", line 20, in <module>
    from .httpsession import HTTPSession
  File "C:\Users\Lance\Desktop\untitled2\lib\gspread\httpsession.py", line 11, in <module>
    import requests
  File "C:\Users\Lance\Desktop\untitled2\lib\requests\__init__.py", line 63, in <module>
    from . import utils
  File "C:\Users\Lance\Desktop\untitled2\lib\requests\utils.py", line 42, in <module>
    if platform.system() == 'Windows':
  File "C:\Python27\lib\platform.py", line 1265, in system
    return uname()[0]
  File "C:\Python27\lib\platform.py", line 1161, in uname
    release,version,csd,ptype = win32_ver()
  File "C:\Python27\lib\platform.py", line 634, in win32_ver
    from _winreg import OpenKeyEx, QueryValueEx, CloseKey, HKEY_LOCAL_MACHINE
  File "C:\Users\Lance\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\python\sandbox.py", line 964, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named _winreg

Can confirm this is still working, with appengine 1.9.67 and v3 of the Google Drive API.

Thanks!

Thank you, so much. I confirmed that it works. I have been struggling with it for couple of days.