Skip to content

Instantly share code, notes, and snippets.

@tomswartz07

tomswartz07/pgc_update_troubleshooting.md

Last active June 4, 2018 19:36
Show Gist options
  • Save tomswartz07/a75a5d336f20fabaf8a8f8733c76da1d to your computer and use it in GitHub Desktop.
Save tomswartz07/a75a5d336f20fabaf8a8f8733c76da1d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
pgc_update_troubleshooting.md

PGC Parameter Troubleshooting

The commands listed in this document use BOSH CLI v2+ and CF CLI version 6.37.0+a40009753.2018-05-25+

Background

In addition to standard PCF service parameters, Our recent releases have included the ability for customers to modify the PostgreSQL configuration parameters while creating or updating service instances.

In Crunchy PCF release version 03.090512.109, users could create a service instance with the pgc_ parameters.

An example of creating a service instance with modifying the max_wal_senders and log_rotation_size PostgreSQL options is as follows:

$ cf create-service postgresql-9.5-odb small myService -c '
{
    "db_name": "exampledb",
    "db_username": "exampleuser",
    "owner_name": "Example User",
    "owner_email": "example.user@company.com"
    "pgc_max_wal_senders": 20,
    "pgc_log_rotation_size": 200000,
}

Starting in 04.x.001, users were able to pass those parameters to the update-service call to update the instance as they saw fit without having to recreate it.

Any PCF Service Parameter that is prefixed with pgc_ refers to a parameter that would modify a PostgreSQL configuration option.

Known Issue

(As of 04 June 2018)

Due to a bug in the crunchy-postgresql-service-adapter, if a customer uses a pgc_ parameter at any time, no further updates may be performed on the service instance.

This means that if a pgc_ is used either during the creation of a service instance or used by calling the cf update-service command, that service instance will no longer accept any further cf update-service commands and will report a 502 error.

NOTE:

  • The service instance continues to run after the update-service call fails.
  • Other parameters can be used with no affect to the service instance.
  • This issue is only related to parameters prefixed with pgc_.

As an example, if a customer required an update to a pgc_ parameter for max_wal_senders, using the service creation example from above, they would issue the command

$ cf update-service myService -c '{"pgc_max_wal_senders": 30}'

and the result code back to them would look like

Updating service instance myService as $USERNAME...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker error: Service cannot be updated at this time, please try again later or contact your operator for more information

If the customer looks at the service status, it will return a status message of update failed

$ cf service myService
Showing info of service myService in org pcfdev-org / space pcfdev-space as admin...

name:            myService
service:         postgresql-9.5-odb
tags:
plan:            small
description:     Crunchy PostgreSQL 9.5 On-Demand Managed Service
documentation:
dashboard:

There are no bound apps for this service.

Showing status of last operation from service myService...

status:    update failed
message:   Service broker error: Service cannot be updated at this time, please try again later or contact your operator for more information
started:   2018-05-31T17:04:04Z
updated:   2018-05-31T17:04:04Z

NOTE: This will happen for any parameters updates including non-pgc_ parameters and regardless of the inclusion in the original setup. A service cannot be fixed by re-running cf update-service with a non-pgc parameter.

Resolution

The service instance, even if showing update failed, will continue to run.

  • If the parameter is not required, the customer should be advised that that no further action is required.
  • If the customer must have the updated PostgreSQL configuration parameter(s) applied to the service instance, there are three possible resolutions to the situation.
    • After any of these resolutions are applied, the pgc_ parameters should not be used until the next Crunchy PCF update is released and installed.

NOTE: Pivotal advises against performing the following manual intervention resolutions, unless absolutely necessary. The recommended advice to a customer is to create a new service instance with the desired parameters, transfer the data, and rebind any applications.

Automatically Fix All Instances

The first possible remedy is to have the customer's platform administration team run the upgrade-all-service-instances errand against the broker. This will remove the pgc_ references and reset the instances to defaults.

The functionality of the Pivotal On Demand Broker is such that when the upgrade-all-service-instances errand is called, all of the manifests will have the pgc_ parameters removed and set back to platform defaults.

To run the errand, the platform administration team (not the end-user client) needs to execute:

bosh -e $ENV -d crunchy-postgresql-on-demand-broker run-errand upgrade-all-service-instances

NOTE: This will iterate over all service instances and will cause interruption to any existing service instances.

This methodology will also be used to apply the PCF Team's official Code Fix when it is released.

Manually Redeploying An Instance With Updated Parameters

These steps will allow a customer to update the service instance with the desired parameters. However, it will still be in a state in which the update-service call will fail and any alterations must be made by these steps and with the assistance of the customer's platform administration team.

  1. The customer must obtain the GUID associated with the service instance using the Cloud Foundry Command Line.
    • cf service $SERVICE_NAME --guid
    • This will be referenced as $GUID for the following steps.
  2. Platform Administrator Required: Download the manifest in a temporary YAML file.
    • bosh -e $ENV -d $service-instance_$GUID manifest > /tmp/service-instance_$GUID.yml
    • Example: bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 manifest > /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml
  3. Using an editor that saves in Unix format, open the manifest file, modify the parameters needed, and save. 
    - name: postgresql
    properties:
      postgresql:
        config:
    • An example for changing the example provided in the background section would be to change the max_wal_senders property under the config from 20 to 30.
    - name: postgresql
    properties:
      postgresql:
        config:
        -   max_wal_senders: 20
        +   max_wal_senders: 30
  4. After the manifest has been updated, the Platform Administration team can update the instance using the deploy command
  • $ bosh -e vbox -d $service-instance_$GUID /tmp/service-instance_$GUID.yml

Here is an example update that modifies the max_wal_senders and adds in a value for wal_keep_segments.

$ bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 deploy /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml

Using environment '192.168.50.6' as client 'admin'

Using deployment 'service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825'

  instance_groups:
  - name: postgresql
    properties:
      postgresql:
        config:
-         max_wal_senders: "<redacted>"
+         max_wal_senders: "<redacted>"
+         wal_keep_segments: "<redacted>"

Continue? [yN]: y

Task 20

Task 20 | 21:25:39 | Preparing deployment: Preparing deployment (00:00:01)
Task 20 | 21:25:42 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 20 | 21:25:42 | Updating instance postgresql: postgresql/e9f2e1ba-886e-4238-af1f-bc87a7769efc  (00:01:30))
Task 20 | 21:27:12 | Updating instance postgresql: postgresql/7b3e1007-fd6d-4287-9695-0c5fa4b27d35 (1) (00:03:27)

Task 20 Started  Thu May 31 21:25:39 UTC 2018
Task 20 Finished Thu May 31 21:30:39 UTC 2018
Task 20 Duration 00:05:00
Task 20 done

Succeeded

Manually Resetting An Instance Back To Defaults

These steps will allow a customer to reset the instance back to the default settings and all the use of the update-service call one time.

  1. The customer must obtain the GUID associated with the service instance using the Cloud Foundry Command Line.
    • cf service $SERVICE_NAME --guid
    • This will be referenced as $GUID for the following steps.
  2. Platform Administrator Required: Download the manifest in a temporary YAML file.
    • bosh -e $ENV -d $service-instance_$GUID manifest > /tmp/service-instance_$GUID.yml
    • Example: bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 manifest > /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml
  3. Using an editor that saves in Unix format, open the manifest file, and remove any pgc_ parameters that been added to the instance. 
    - name: postgresql
    properties:
      postgresql:
        config:
  4. After the manifest has been updated, the Platform Administration team can update the instance using the deploy command
  • $ bosh -e vbox -d $service-instance_$GUID /tmp/service-instance_$GUID.yml

Here is an example update that removes the max_wal_senders.

$ bosh -e vbox -d service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825 deploy /tmp/service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825.yml

Using environment '192.168.50.6' as client 'admin'

Using deployment 'service-instance_73d8c03e-3285-45d1-9c6e-ab9c9750e825'

  instance_groups:
  - name: postgresql
    properties:
      postgresql:
        config:
-         max_wal_senders: "<redacted>"

Continue? [yN]: y

Task 107

Task 107 | 21:25:39 | Preparing deployment: Preparing deployment (00:00:01)
Task 107 | 21:25:42 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 107 | 21:25:42 | Updating instance postgresql: postgresql/e9f2e1ba-886e-4238-af1f-bc87a7769efc  (00:01:30))
Task 107 | 21:27:12 | Updating instance postgresql: postgresql/7b3e1007-fd6d-4287-9695-0c5fa4b27d35 (1) (00:03:27)

Code Fix

In our next release, 04.090513.002, the service adapter will accommodate the adjusted paths and will, after running the upgrade-all-service-instances process in PCF to update all of the services instances to the new release, the update-service call will work properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment