Deis upgrade issue

gistfile1.txt

INFO [firmware]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware]: adding 30s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe
INFO [firmware]: This deployments overall timeout is 150s - batch timout is 150s and there are 1 batches to deploy with a total of 3 pods
INFO [firmware]: waited 10s and 3 pods are in service
INFO [firmware]: waited 20s and 3 pods are in service
INFO [firmware]: waited 30s and 3 pods are in service
INFO [firmware-api]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api]: adding 50s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe
INFO [firmware-api]: This deployments overall timeout is 170s - batch timout is 170s and there are 1 batches to deploy with a total of 1 pods
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
ERROR [firmware-api-staging]: There was a problem deploying v20. Rolling back process types to release v19.
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: buildpack type detected. Defaulting to $PORT 5000
INFO [firmware-api-staging]: adding 50s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe
INFO [firmware-api-staging]: This deployments overall timeout is 170s - batch timout is 170s and there are 1 batches to deploy with a total of 1 pods
ERROR [firmware-api-staging]: (app::deploy): There was a problem deploying v20. Rolling back process types to release v19.
('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')
Traceback (most recent call last):
  File "/app/scheduler/__init__.py", line 253, in deploy
    deployment = self.deployment.get(namespace, name).json()
  File "/app/scheduler/resources/deployment.py", line 29, in get
    raise KubeHTTPException(response, message, *args)
scheduler.exceptions.KubeHTTPException: ('failed to get Deployment "firmware-api-staging-rake" in Namespace "firmware-api-staging": 404 Not Found deployments.extensions "firmware-api-staging-rake" not found', 'firmware-api-staging-rake', 'firmware-api-staging')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/scheduler/resources/pod.py", line 138, in manifest
    self.secret.get(namespace, 'objectstorage-keyfile')
  File "/app/scheduler/resources/secret.py", line 25, in get
    raise KubeHTTPException(response, message, *args)
scheduler.exceptions.KubeHTTPException: ('failed to get Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 404 Not Found secrets "objectstorage-keyfile" not found', 'objectstorage-keyfile', 'firmware-api-staging')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/models/app.py", line 487, in deploy
    async_run(tasks)
  File "/app/api/utils.py", line 169, in async_run
    raise error
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/app/api/utils.py", line 182, in async_task
    yield from loop.run_in_executor(None, params)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/app/scheduler/__init__.py", line 260, in deploy
    namespace, name, image, entrypoint, command, **kwargs
  File "/app/scheduler/resources/deployment.py", line 111, in create
    entrypoint, command, **kwargs)
  File "/app/scheduler/resources/deployment.py", line 105, in manifest
    manifest['spec']['template'] = self.pod.manifest(namespace, name, image, **kwargs)
  File "/app/scheduler/resources/pod.py", line 141, in manifest
    self.secret.create(namespace, 'objectstorage-keyfile', secret['data'])
  File "/app/scheduler/resources/secret.py", line 88, in create
    'failed to create Secret "{}" in Namespace "{}"', name, namespace
scheduler.exceptions.KubeHTTPException: ('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/api/models/app.py", line 496, in deploy
    raise DeisException('{}\n{}'.format(err, str(e))) from e
api.exceptions.DeisException: There was a problem deploying v20. Rolling back process types to release v19.
('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/app/api/management/commands/load_db_state_to_k8s.py", line 34, in handle
    entrypoint, command, **kwargs)
  File "/app/scheduler/resources/deployment.py", line 105, in manifest
    manifest['spec']['template'] = self.pod.manifest(namespace, name, image, **kwargs)
  File "/app/scheduler/resources/pod.py", line 141, in manifest
    self.secret.create(namespace, 'objectstorage-keyfile', secret['data'])
  File "/app/scheduler/resources/secret.py", line 88, in create
    'failed to create Secret "{}" in Namespace "{}"', name, namespace
scheduler.exceptions.KubeHTTPException: ('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')
Task exception was never retrieved
future: <Task finished coro=<async_task() done, defined at /app/api/utils.py:174> exception=KubeHTTPException('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')>
Traceback (most recent call last):
  File "/app/scheduler/__init__.py", line 253, in deploy
    deployment = self.deployment.get(namespace, name).json()
  File "/app/scheduler/resources/deployment.py", line 29, in get
    raise KubeHTTPException(response, message, *args)
scheduler.exceptions.KubeHTTPException: ('failed to get Deployment "firmware-api-staging-worker" in Namespace "firmware-api-staging": 404 Not Found deployments.extensions "firmware-api-staging-worker" not found', 'firmware-api-staging-worker', 'firmware-api-staging')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/scheduler/resources/pod.py", line 138, in manifest
    self.secret.get(namespace, 'objectstorage-keyfile')
  File "/app/scheduler/resources/secret.py", line 25, in get
    raise KubeHTTPException(response, message, *args)
scheduler.exceptions.KubeHTTPException: ('failed to get Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 404 Not Found secrets "objectstorage-keyfile" not found', 'objectstorage-keyfile', 'firmware-api-staging')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/app/api/utils.py", line 182, in async_task
    yield from loop.run_in_executor(None, params)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/app/scheduler/__init__.py", line 260, in deploy
    namespace, name, image, entrypoint, command, **kwargs
  File "/app/scheduler/resources/deployment.py", line 111, in create
    entrypoint, command, **kwargs)
  File "/app/scheduler/resources/deployment.py", line 105, in manifest
    manifest['spec']['template'] = self.pod.manifest(namespace, name, image, **kwargs)
  File "/app/scheduler/resources/pod.py", line 141, in manifest
    self.secret.create(namespace, 'objectstorage-keyfile', secret['data'])
  File "/app/scheduler/resources/secret.py", line 88, in create
    'failed to create Secret "{}" in Namespace "{}"', name, namespace
scheduler.exceptions.KubeHTTPException: ('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')
    entrypoint, command, **kwargs)
  File "/app/scheduler/resources/deployment.py", line 105, in manifest
    manifest['spec']['template'] = self.pod.manifest(namespace, name, image, **kwargs)
  File "/app/scheduler/resources/pod.py", line 141, in manifest
    self.secret.create(namespace, 'objectstorage-keyfile', secret['data'])
  File "/app/scheduler/resources/secret.py", line 88, in create
    'failed to create Secret "{}" in Namespace "{}"', name, namespace
scheduler.exceptions.KubeHTTPException: ('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')
Task exception was never retrieved
future: <Task finished coro=<async_task() done, defined at /app/api/utils.py:174> exception=KubeHTTPException('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')>
Traceback (most recent call last):
  File "/app/scheduler/__init__.py", line 253, in deploy
    deployment = self.deployment.get(namespace, name).json()
  File "/app/scheduler/resources/deployment.py", line 29, in get
    raise KubeHTTPException(response, message, *args)
scheduler.exceptions.KubeHTTPException: ('failed to get Deployment "firmware-api-staging-worker" in Namespace "firmware-api-staging": 404 Not Found deployments.extensions "firmware-api-staging-worker" not found', 'firmware-api-staging-worker', 'firmware-api-staging')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/scheduler/resources/pod.py", line 138, in manifest
    self.secret.get(namespace, 'objectstorage-keyfile')
  File "/app/scheduler/resources/secret.py", line 25, in get
    entrypoint, command, **kwargs)
  File "/app/scheduler/resources/deployment.py", line 105, in manifest
    manifest['spec']['template'] = self.pod.manifest(namespace, name, image, **kwargs)
  File "/app/scheduler/resources/pod.py", line 141, in manifest
    self.secret.create(namespace, 'objectstorage-keyfile', secret['data'])
  File "/app/scheduler/resources/secret.py", line 88, in create
    'failed to create Secret "{}" in Namespace "{}"', name, namespace
scheduler.exceptions.KubeHTTPException: ('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')
Task exception was never retrieved
future: <Task finished coro=<async_task() done, defined at /app/api/utils.py:174> exception=KubeHTTPException('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')>
Traceback (most recent call last):
  File "/app/scheduler/__init__.py", line 253, in deploy
    deployment = self.deployment.get(namespace, name).json()
  File "/app/scheduler/resources/deployment.py", line 29, in get
    raise KubeHTTPException(response, message, *args)
scheduler.exceptions.KubeHTTPException: ('failed to get Deployment "firmware-api-staging-worker" in Namespace "firmware-api-staging": 404 Not Found deployments.extensions "firmware-api-staging-worker" not found', 'firmware-api-staging-worker', 'firmware-api-staging')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/scheduler/resources/pod.py", line 138, in manifest
    self.secret.get(namespace, 'objectstorage-keyfile')
  File "/app/scheduler/resources/secret.py", line 25, in get
    raise KubeHTTPException(response, message, *args)
scheduler.exceptions.KubeHTTPException: ('failed to get Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 404 Not Found secrets "objectstorage-keyfile" not found', 'objectstorage-keyfile', 'firmware-api-staging')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "/app/api/utils.py", line 182, in async_task
    yield from loop.run_in_executor(None, params)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/app/scheduler/__init__.py", line 260, in deploy
    namespace, name, image, entrypoint, command, **kwargs
  File "/app/scheduler/resources/deployment.py", line 111, in create
    entrypoint, command, **kwargs)
  File "/app/scheduler/resources/deployment.py", line 105, in manifest
    manifest['spec']['template'] = self.pod.manifest(namespace, name, image, **kwargs)
  File "/app/scheduler/resources/pod.py", line 141, in manifest
    self.secret.create(namespace, 'objectstorage-keyfile', secret['data'])
  File "/app/scheduler/resources/secret.py", line 88, in create
    'failed to create Secret "{}" in Namespace "{}"', name, namespace
scheduler.exceptions.KubeHTTPException: ('failed to failed to create Secret "objectstorage-keyfile" in Namespace "firmware-api-staging": 409 Conflict secrets "objectstorage-keyfile" already exists', 'objectstorage-keyfile', 'firmware-api-staging')