How to debug an unresponsive App Engine Flex app

README.md

How to debug an unresponsive App Engine Flex app

You can debug an unhealthy app using Google Cloud Logging and other GCP tools even if the app has become unresponsive. A test app is provided that simulates the problem in an experiment for the purpose of explaining the debugging process with App Engine Flex.

Prerequisites:

1. GCP Project with billing enabled
2. Google Cloud SDK installed

The app randomly stalls on incoming requests by going to sleep.

Follow these steps to deploy and run the app.

Set your project as the default in gcloud

GOOGLE_CLOUD_PROJECT=[Your project]
gcloud config set project $GOOGLE_CLOUD_PROJECT

Enable App Engine

gcloud app create

Deploy the app

gcloud app deploy

Browse to the app

gcloud app browse

Set up Apache Bench In the Cloud Shell or Linux command line

sudo apt-get install apache2-utils

Send some requests to your app

ab -n 1000 -c 5 https://${GOOGLE_CLOUD_PROJECT}.appspot.com/

an_unhealthy_app_demo.md

How to debug an unresponsive App Engine Flex app

See the README.md for details on how to run this

app.yaml

runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
  python_version: 3
manual_scaling:
  instances: 1
resources:
  cpu: 1
  memory_gb: 0.5
  disk_size_gb: 10
readiness_check:
  path: "/readiness_check"
  check_interval_sec: 5
  timeout_sec: 4
  failure_threshold: 2
  success_threshold: 2
  app_start_timeout_sec: 300

main.py

# Copyright 2020 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Simulate an unhealthy app for demonstration of troubleshooting"""

import logging
import random
import sys
import time
from flask import Flask, g


app = Flask(__name__)


def get_app_healthy() -> bool:
  """Get the application health"""
  if 'app_healthy' not in g:
    g.app_healthy = True
  return g.app_healthy


def set_app_healthy(value: bool):
  """Set the application health"""
  if 'app_healthy' not in g:
    g.app_healthy = value
  g.app_healthy = value
    

@app.route('/')
def home():
  """1 out of 20 requests change the app to an unhealthy state"""
  msg = 'Welcome to the home page!\n'
  if not get_app_healthy():
    msg = 'Changing to an unhealthy state'
    logging.error(msg)
    #sys.exit(1) # exit the process
    time.sleep(90) # seconds
    return msg
  r = random.uniform(0.0, 1.0)
  if r > 0.95:
    msg = 'App is unhealthy'
    logging.error(msg)
    set_app_healthy(False)
    time.sleep(90) # seconds
  return msg


@app.route('/readiness_check')
def readiness_check():
  #Respond to the load balancer's readiness check
  msg = 'OK\n'
  r = random.uniform(0.0, 1.0)
  if r > 0.9:
    msg = 'App is refuses to start'
    logging.error(msg)
    sys.exit(1)
  return msg


if __name__ == '__main__':
  logging.basicConfig(level=logging.INFO)
  logging.info('Starting up')
  app.run(host='127.0.0.1', port=8080)

requirements.txt

Flask==1.1.1
gunicorn==20.0.4