annos2

Hi Steve! I guess my fix will not resolve pulling issue. 
Today I receive a page with pulling instance and spend some time on manually debugging this on the host.
I thought about fixing it without instance reschedule, and I was able to do so.
PD - https://opendns.pagerduty.com/alerts/Q35WOBHR7QPFCS
So there is a two places where the process stuck.
1. Before creating net config for instance problem with lock on `/var/run-host/host-api-instance/{$instance_name}`
https://github.com/cisco-sbg/cloudsec_quadra_quadra-api/blob/b90bc90453c6955e9bd943b1ca30a56ab0fcc164/qhost/api/host.py#L322-L323
```
qhost-api stderr | 2025-08-05 09:16:03,080 level=DEBU client=None request=834991e3-a2af-4b59-bd1b-6c16f3be5b0d user_agent=unknown_python_requests app=qhost.modules.locks: ipc lock /var/run-host/host-api-instance/dlp-scanner-inline-prod-c9c5c4f7a067.delphi.dub1 failed after 20001.278076171875ms
qhost-api stderr |

annos2

1. We have logging:
```
$ qq logging show talosintsplunklogs.ipedia
Using Streamline Auth as annos@cisco.com
Log Slug                   Project  Type    API Token                         Index                          Endpoint
-------------------------  -------  ------  --------------------------------  -----------------------------  --------------------------------------------
talosintsplunklogs.ipedia  ipedia   splunk  8bf3############################  talos-quadra-ipedia-int-event  http-inputs-cisco-talos-use1.splunkcloud.com

Deployments using talosintsplunklogs.ipedia logging configuration:
Pool                        Deployment

annos2

# Instance scheduled and then moved to pulling state
qhost-api stderr | 2025-07-23 08:46:07,268 app=qhost.modules.pooltools: instance state before: status: scheduled, hash: mps-e9ca8e78bf1a.sigenv1.mel1
qhost-api stderr | 2025-07-23 08:46:07,268 app=qhost.modules.pooltools: instance state after: status: pulling, hash: mps-e9ca8e78bf1a.sigenv1.mel1

# DDOS of `container_v2` endpoint and finally createing net config

qhost-api stderr | 2025-07-23 08:48:57,586 app=qhost.api: endpoint=containerv2.head method=HEAD response_status=200 user=quadraqhost_b auth=ClassicAuth uri=http://127.0.0.1:5050/quadra/containers_v2/mps-e9ca8e78bf1a.sigenv1.mel1 ip=146.112.186.93 time=390 size=4 user_agent=no_agent
qhost-api stderr | 2025-07-23 08:48:59,736 app=qhost.modules.pooltools: Creating netconfig for container mps-e9ca8e78bf1a.sigenv1.mel1

# Creating docker container and between still ddos of `container_v2` endpoint

annos2

  # module.dashboards.signalfx_dashboard.etcd_metrics will be destroyed
  # (because signalfx_dashboard.etcd_metrics is not in configuration)
  - resource "signalfx_dashboard" "etcd_metrics" {
      - charts_resolution = "default" -> null
      - dashboard_group   = "Gv92PS4AcAA" -> null
      - id                = "Gv96g5NAcAA" -> null
      - name              = "Etcd Metrics" -> null
      - time_range        = "-6h" -> null
      - url               = "https://cisco-opendnsbu-sse.signalfx.com/#/dashboard/Gv96g5NAcAA" -> null

annos2

+ terraform init -input=false

Initializing the backend...



Successfully configured the backend "s3"! Terraform will automatically

use this backend unless the backend configuration changes.

annos2

  p1w │ ~/cisco/github-cloud/cloudsec_quadra_oauth_server │ QQ-10188_pin…ons_and_hash ......................................................................................................... cloudsec_quadra_quadra-api  │ 15:16:56 
$ make lock-dependencies
pip3 install virtualenv && \
		python3 -m venv venv
Requirement already satisfied: virtualenv in /Users/annos/.pyenv/versions/3.8.10/lib/python3.8/site-packages (20.31.2)
Requirement already satisfied: distlib<1,>=0.3.7 in /Users/annos/.pyenv/versions/3.8.10/lib/python3.8/site-packages (from virtualenv) (0.3.9)
Requirement already satisfied: filelock<4,>=3.12.2 in /Users/annos/.pyenv/versions/3.8.10/lib/python3.8/site-packages (from virtualenv) (3.16.1)
Requirement already satisfied: platformdirs<5,>=3.9.1 in /Users/annos/.pyenv/versions/3.8.10/lib/python3.8/site-packages (from virtualenv) (4.3.6)
. ./venv/bin/activate && \
		pip3 install .[lock] && \

annos2

```
$ QUADRA_APISERVER=http://qhost-0437f7b3eb93cf443.quadra.s1.usw1.opendns.com:5050 qq logging create --project annos --logging-type splunk --logging-api-key bad-token-123 --logging-index quadra-quadra-platform-prod-index annosexamplecustomendpoint --custom-endpoint annos-test-custom.splunk.endpoint.com
Using Streamline Auth as annos@cisco.com
*WARNING* Your client is a developer version
There's a new version available!  Please run 'sudo qq upgrade' to install it
Log Slug                          Project  Type
--------------------------------  -------  ------
annosexamplecustomendpoint.annos  annos    splunk

$ qq pool show -vvv splunk-test-datadoc-annos.quadra