Rolling back from capi-release 1.47.0

rollback.md

capi-release 1.47.0 is marked as having downtime issues, so we rolled back to capi-release 1.46.0 in cf-deployment. Teams that have been using the release-candidate branch of cf-deployment may notice problems in their pipelines, caused by the fact that capi-release 1.47.0 included a database migration that got rolled back as well. Steps provided here should help teams resolve this issue.

To un-bork your environment based on release-candidate, you can take the following steps:

1. SSH to the API VM so that you can target the database node:

bosh ssh api/0

2. Install a mysql client

sudo -i
apt-get install mysql-client

3. Connect to the database

mysql -h <host> -u <username> -p

4. Drop the encyrption_key_label column from the table:

> alter table env_groups drop column encryption_key_label;
> alter table service_brokers drop column encryption_key_label;
> alter table service_bindings drop column encryption_key_label;
> alter table service_instances drop column encryption_key_label;
> alter table service_keys drop column encryption_key_label;
> alter table apps drop column encryption_key_label;
> alter table buildpack_lifecycle_buildpacks drop column encryption_key_label;
> alter table buildpack_lifecycle_data drop column encryption_key_label;
> alter table droplets drop column encryption_key_label;
> alter table packages drop column encryption_key_label;
> alter table tasks drop column encryption_key_label;

5. Make sure to remove the record of having run the migration, so that future capi-release versions will make sure to run it again:

> delete from schema_migrations where filename like '201712%';
> exit

6. Clean up after yourself:

apt-get remove mysql-client
exit

apt-get remove mysql-client

doesn't actually remove the files from the file system
follow up with cleaning the file system

apt-get -y remove mysql-client
apt-get -y autoremove

Downgrade Caution

If you had any apps that were created/updated while on capi-release 1.47.0 you may run into UnknownErrors from the API while interacting with them after downgrading CC. This would affect other tables that have encrypted columns as well (e.g. service bindings).

Looking at the logs you'll see something like this:

{"timestamp":1515711729.1879618,"message":"Request failed: 500: {\"error_code\"=>\"UnknownError\", \"description\"=>\"An unknown error occurred.\", \"code\"=>10001, \"test_mode_info\"=>{\"description\"=>\"salt must be an 8-octet string\", \"error_code\"=>\"CF-CipherError\", \"backtrace\"=>[\"/var/vcap/data/packages/cloud_controller_ng/bbccb38817cab47dd33cd3250ea8e5f2726b35dd/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `pkcs5_keyivgen'\", \"/var/vcap/data/packages/cloud_controller_ng/bbccb38817cab47dd33cd3250ea8e5f2726b35dd/cloud_controller_ng/lib/cloud_controller/encryptor.rb:32:in `run_cipher'\
...

CAPI Release 1.47.0 had some changes to how CC encrypts data in its database and one of these changes was that we switched to a 16-byte salt while 1.46.0 used an 8-byte salt. To get around this either delete the affected rows from the db or upgrade CC back to 1.47.0 where the updated code can handle the longer salts.