Skip to content

Instantly share code, notes, and snippets.

@PeterS-gd

PeterS-gd/management-server-groovy.log Secret

Created October 7, 2021 13:10
Show Gist options
  • Save PeterS-gd/75c0f507c96803a053df37caf3b25181 to your computer and use it in GitHub Desktop.
Save PeterS-gd/75c0f507c96803a053df37caf3b25181 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
management-server-groovy.log
2021-10-07 08:55:04,050 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Network id=201 is already implemented
2021-10-07 08:55:04,135 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Releasing ip address for instance=5890
2021-10-07 08:55:04,194 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Allocated a nic NicProfile[5890-1536-ba6ec20b-2962-412e-8d82-3bac7791dc1a-192.168.204.170-null] for VM[ConsoleProxy|v-1536-VM]
2021-10-07 08:55:04,273 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-eb7850fb) (logid:fdc941f8) Execute sync-queue item: SyncQueueItemVO {id:4696, queueId: 4694, contentType: AsyncJob, contentId: 4732, lastProcessMsid: 90520741394843, lastprocessNumber: 2, lastProcessTime: Thu Oct 07 08:55:04 EDT 2021, created: Thu Oct 07 08:55:02 EDT 2021}
2021-10-07 08:55:04,278 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-eb7850fb) (logid:fdc941f8) Schedule queued job-4732
2021-10-07 08:55:04,293 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) template 3 is already in store:5, type:Image
2021-10-07 08:55:04,311 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Not found (templateId:3poolId:2conf:null) in template_spool_ref, persisting it
2021-10-07 08:55:04,352 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732) (logid:461dafb4) Add job-4732 into job monitoring
2021-10-07 08:55:04,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732) (logid:e381c0f1) Executing AsyncJobVO {id:4732, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:55:02 EDT 2021, removed: null}
2021-10-07 08:55:04,366 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732) (logid:e381c0f1) Run VM work job: com.cloud.vm.VmWorkStart for VM 1537, job origin: 3440
2021-10-07 08:55:04,370 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Execute VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1537,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:55:04,402 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2021-10-07 08:55:04,404 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Successfully transitioned to start state for VM[SecondaryStorageVm|s-1537-VM] reservation id = b2622726-a201-431c-8887-4794da95439b
2021-10-07 08:55:04,410 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) template 3 is already in store:2, type:Primary
2021-10-07 08:55:04,415 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 513
2021-10-07 08:55:04,420 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Acquire lock on VMTemplateStoragePool 513 with timeout 3600 seconds
2021-10-07 08:55:04,459 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) lock is acquired for VMTemplateStoragePool 513
2021-10-07 08:55:04,467 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:55:04,479 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) DeploymentPlanner allocation algorithm: null
2021-10-07 08:55:04,480 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: (512.00 MB) 536870912
2021-10-07 08:55:04,480 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 08:55:04,485 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:55:04,496 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Searching resources only under specified Pod: 1
2021-10-07 08:55:04,505 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Pod: 1
2021-10-07 08:55:04,523 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2021-10-07 08:55:04,540 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 08:55:04,544 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 08:55:04,545 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 08:55:04,550 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) FirstFitAllocator has 2 hosts to check for allocation: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:55:04,560 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand
2021-10-07 08:55:04,568 DEBUG [c.c.a.t.Request] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Seq 1-3253569255798472727: Sending { Cmd , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.204.62/cloudstack","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6db2511f-6992-3120-b49e-fafdfa8ec257","id":"2","poolType":"RBD","host":"192.168.206.63","path":"cloudstack","port":"0","url":"RBD://192.168.206.63/cloudstack/?ROLE=Primary&STOREUUID=6db2511f-6992-3120-b49e-fafdfa8ec257","isManaged":"false"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"10800","bypassHostMaintenance":"false"}}] }
2021-10-07 08:55:04,571 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Found 2 hosts for allocation after prioritization: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:55:04,571 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Looking for speed=500Mhz, Ram=512 MB
2021-10-07 08:55:04,601 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:55:04,602 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:55:04,610 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:55:04,610 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73100 , Requested CPU: 500
2021-10-07 08:55:04,610 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (60.89 GB) 65378861056 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:55:04,610 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:55:04,610 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 1, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:55:04,610 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 1, used: (1.00 GB) 1073741824, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:55:04,610 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 1
2021-10-07 08:55:04,637 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:55:04,637 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:55:04,646 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:55:04,647 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:55:04,647 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (61.90 GB) 66469449728 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:55:04,647 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:55:04,647 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:55:04,647 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 2, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:55:04,647 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 2
2021-10-07 08:55:04,647 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320 FirstFitRoutingAllocator) (logid:e381c0f1) Host Allocator returning 2 suitable hosts
2021-10-07 08:55:04,653 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking suitable pools for volume (Id, Type): (1537,ROOT)
2021-10-07 08:55:04,653 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) We need to allocate new storagepool for this volume
2021-10-07 08:55:04,657 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Calling StoragePoolAllocators to find suitable pools
2021-10-07 08:55:04,674 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) System VMs will use shared storage for zone id=1
2021-10-07 08:55:04,674 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 08:55:04,674 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 08:55:04,674 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 08:55:04,677 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 08:55:04,685 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 08:55:04,690 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Storage pool null (2) does not supply IOPS capacity, assuming enough capacity
2021-10-07 08:55:04,696 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking pool 2 for storage, totalSize: 30005954347008, usedBytes: 11020345267, usedPct: 3.6727194674609234E-4, disable threshold: 0.85
2021-10-07 08:55:04,696 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Destination pool id: 2
2021-10-07 08:55:04,772 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Pool ID for the volume with ID 1537 is null
2021-10-07 08:55:04,805 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Found storage pool CS_Ceph of type RBD with over-provisioning factor 2
2021-10-07 08:55:04,805 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Total over-provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 08:55:04,805 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Total capacity of the pool CS_Ceph with ID 2 is (54.5805 TB) 60011908694016
2021-10-07 08:55:04,811 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking pool: 2 for storage allocation , maxSize : (54.5805 TB) 60011908694016, totalAllocatedSize : (0 bytes) 0, askingSize : (0 bytes) 0, allocated disable threshold: 0.85
2021-10-07 08:55:04,811 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
2021-10-07 08:55:04,811 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
2021-10-07 08:55:04,815 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking if host: 1 can access any suitable storage pool for volume: ROOT
2021-10-07 08:55:04,819 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Host: 1 can access pool: 2
2021-10-07 08:55:04,823 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Found a potential host id: 1 name: gd-cs01-clst01-compute01 and associated storage pools for this VM
2021-10-07 08:55:04,827 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1537|ROOT-->Pool(2))]
2021-10-07 08:55:04,827 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Deployment found - P0=VM[SecondaryStorageVm|s-1537-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1537|ROOT-->Pool(2))]
2021-10-07 08:55:04,938 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 1 host id before state transition: null
2021-10-07 08:55:04,978 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:55:04,978 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) We are allocating VM, increasing the used capacity of this host:1
2021-10-07 08:55:04,978 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Current Used CPU: 500 , Free CPU:73100 ,Requested CPU: 500
2021-10-07 08:55:04,978 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Current Used RAM: (1.00 GB) 1073741824 , Free RAM:(60.89 GB) 65378861056 ,Requested RAM: (512.00 MB) 536870912
2021-10-07 08:55:04,978 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) CPU STATS after allocation: for host: 1, old used: 500, old reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used:1000, reserved:0; requested cpu:500,alloc_from_last:false
2021-10-07 08:55:04,978 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) RAM STATS after allocation: for host: 1, old used: (1.00 GB) 1073741824, old reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; new used: (1.50 GB) 1610612736, reserved: (0 bytes) 0; requested mem: (512.00 MB) 536870912,alloc_from_last:false
2021-10-07 08:55:04,984 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:55:04,984 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:55:04,990 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:55:04,990 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) We need to allocate to the last host again, so checking if there is enough reserved capacity
2021-10-07 08:55:04,990 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Reserved CPU: 0 , Requested CPU: 500
2021-10-07 08:55:04,990 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Reserved RAM: (0 bytes) 0 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:55:04,990 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) STATS: Failed to alloc resource from host: 1 reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (512.00 MB) 536870912
2021-10-07 08:55:04,990 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Host does not have enough reserved CPU available, cannot allocate to this host.
2021-10-07 08:55:04,991 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:55:04,998 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:55:04,998 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Free CPU: 73100 , Requested CPU: 500
2021-10-07 08:55:04,998 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Free RAM: (60.89 GB) 65378861056 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:55:04,998 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:55:04,998 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) STATS: Can alloc CPU from host: 1, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:55:04,998 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) STATS: Can alloc MEM from host: 1, used: (1.00 GB) 1073741824, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:55:05,072 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Network id=200 is already implemented
2021-10-07 08:55:05,175 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Network id=202 is already implemented
2021-10-07 08:55:05,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-63b9bae3) (logid:cbb71bcb) Found 0 routers to update status.
2021-10-07 08:55:05,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-63b9bae3) (logid:cbb71bcb) Found 0 VPC's to update Redundant State.
2021-10-07 08:55:05,358 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-63b9bae3) (logid:cbb71bcb) Found 0 networks to update RvR status.
2021-10-07 08:55:06,600 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Reserved NIC for s-1537-VM [ipv4:169.254.101.45 netmask:255.255.0.0 gateway:169.254.0.1]
2021-10-07 08:55:06,808 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Network id=201 is already implemented
2021-10-07 08:55:07,315 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Releasing ip address for instance=5893
2021-10-07 08:55:07,406 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Allocated a nic NicProfile[5893-1537-b2622726-a201-431c-8887-4794da95439b-192.168.204.154-null] for VM[SecondaryStorageVm|s-1537-VM]
2021-10-07 08:55:07,486 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Network id=203 is already implemented
2021-10-07 08:55:07,716 DEBUG [c.c.n.g.StorageNetworkGuru] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Allocated a storage nic NicProfile[5894-1537-b2622726-a201-431c-8887-4794da95439b-192.168.204.187-null] for VM[SecondaryStorageVm|s-1537-VM]
2021-10-07 08:55:07,807 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) template 3 is already in store:5, type:Image
2021-10-07 08:55:07,852 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) template 3 is already in store:2, type:Primary
2021-10-07 08:55:07,856 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 513
2021-10-07 08:55:07,859 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Acquire lock on VMTemplateStoragePool 513 with timeout 3600 seconds
2021-10-07 08:55:14,176 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 08:55:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 08:55:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 08:55:14,186 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Run missing VM report. current time: 1633611314186
2021-10-07 08:55:14,192 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Detected missing VM. host: 1, vm id: 1536(04319276-972c-424d-926d-86c5dd6af1b8), power state: PowerReportMissing, last state update: 2021/10/07T08:54:58.000Z
2021-10-07 08:55:14,192 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) vm id: 1536 - time since last state update(16186ms) has not passed graceful period yet
2021-10-07 08:55:14,199 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1537
2021-10-07 08:55:14,199 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Detected missing VM. host: 1, vm id: 1537(d008b3ee-dd01-4c46-a10f-05593bb59bb5), power state: PowerReportMissing, last state update: 2021/10/07T08:55:04.000Z
2021-10-07 08:55:14,199 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) vm id: 1537 - time since last state update(10186ms) has not passed graceful period yet
2021-10-07 08:55:14,199 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 08:55:15,419 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-aaa0b557) (logid:c3457856) StorageCollector is running...
2021-10-07 08:55:15,434 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-aaa0b557) (logid:c3457856) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 08:55:15,437 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:ctx-aaa0b557) (logid:c3457856) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 08:55:15,437 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:ctx-aaa0b557) (logid:c3457856) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 08:55:15,658 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-aaa0b557) (logid:c3457856) Seq 2-4126986108531640363: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 08:55:25,131 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-bd5ca538) (logid:1e20a4b5) AutoScaling Monitor is running...
2021-10-07 08:55:35,352 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-856ad67d) (logid:3c29bdc9) Found 0 routers to update status.
2021-10-07 08:55:35,354 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-856ad67d) (logid:3c29bdc9) Found 0 VPC's to update Redundant State.
2021-10-07 08:55:35,357 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-856ad67d) (logid:3c29bdc9) Found 0 networks to update RvR status.
2021-10-07 08:55:38,578 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-019a63f8) (logid:2d6d092e) HostStatsCollector is running...
2021-10-07 08:55:38,659 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-019a63f8) (logid:2d6d092e) Seq 1-3253569255798472728: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:55:38,698 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-019a63f8) (logid:2d6d092e) Seq 2-4126986108531640364: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:55:39,730 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-a6ec004a) (logid:5fd36680) Resetting hosts suitable for reconnect
2021-10-07 08:55:39,735 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-a6ec004a) (logid:5fd36680) Completed resetting hosts suitable for reconnect
2021-10-07 08:55:39,735 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-a6ec004a) (logid:5fd36680) Acquiring hosts for clusters already owned by this management server
2021-10-07 08:55:39,736 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-a6ec004a) (logid:5fd36680) Completed acquiring hosts for clusters already owned by this management server
2021-10-07 08:55:39,736 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-a6ec004a) (logid:5fd36680) Acquiring hosts for clusters not owned by any management server
2021-10-07 08:55:39,738 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-a6ec004a) (logid:5fd36680) Completed acquiring hosts for clusters not owned by any management server
2021-10-07 08:55:57,946 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 08:55:57,948 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 08:55:57,948 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 08:55:57,954 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 08:56:05,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a6356a39) (logid:3e6ab9fe) Found 0 routers to update status.
2021-10-07 08:56:05,355 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a6356a39) (logid:3e6ab9fe) Found 0 VPC's to update Redundant State.
2021-10-07 08:56:05,357 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a6356a39) (logid:3e6ab9fe) Found 0 networks to update RvR status.
2021-10-07 08:56:14,175 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 08:56:14,176 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 08:56:14,176 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 08:56:14,184 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Run missing VM report. current time: 1633611374184
2021-10-07 08:56:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Detected missing VM. host: 1, vm id: 1536(04319276-972c-424d-926d-86c5dd6af1b8), power state: PowerReportMissing, last state update: 2021/10/07T08:54:58.000Z
2021-10-07 08:56:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) vm id: 1536 - time since last state update(76184ms) has not passed graceful period yet
2021-10-07 08:56:14,196 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1537
2021-10-07 08:56:14,196 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Detected missing VM. host: 1, vm id: 1537(d008b3ee-dd01-4c46-a10f-05593bb59bb5), power state: PowerReportMissing, last state update: 2021/10/07T08:55:04.000Z
2021-10-07 08:56:14,196 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) vm id: 1537 - time since last state update(70184ms) has not passed graceful period yet
2021-10-07 08:56:14,196 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 08:56:15,659 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-7bb6a794) (logid:76645c44) StorageCollector is running...
2021-10-07 08:56:15,677 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-7bb6a794) (logid:76645c44) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 08:56:15,682 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:ctx-7bb6a794) (logid:76645c44) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 08:56:15,682 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:ctx-7bb6a794) (logid:76645c44) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 08:56:15,968 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-7bb6a794) (logid:76645c44) Seq 1-3253569255798472729: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 08:56:25,136 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-63c22871) (logid:acab744a) AutoScaling Monitor is running...
2021-10-07 08:56:35,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-bddba458) (logid:2c37f2a2) Found 0 routers to update status.
2021-10-07 08:56:35,362 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-bddba458) (logid:2c37f2a2) Found 0 VPC's to update Redundant State.
2021-10-07 08:56:35,368 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-bddba458) (logid:2c37f2a2) Found 0 networks to update RvR status.
2021-10-07 08:56:38,702 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-8cd6cae8) (logid:c858a032) HostStatsCollector is running...
2021-10-07 08:56:38,742 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-8cd6cae8) (logid:c858a032) Seq 1-3253569255798472730: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:56:38,768 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-8cd6cae8) (logid:c858a032) Seq 2-4126986108531640365: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:56:48,553 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:ctx-e5255c55) (logid:226f4489) Task (job-4731) has been pending for 106 seconds
2021-10-07 08:56:48,554 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:ctx-e5255c55) (logid:226f4489) Task (job-4732) has been pending for 104 seconds
2021-10-07 08:56:57,942 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 08:56:57,945 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 08:56:57,945 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 08:56:57,948 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 08:57:05,351 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-6e168125) (logid:589e1fa9) Found 0 routers to update status.
2021-10-07 08:57:05,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-6e168125) (logid:589e1fa9) Found 0 VPC's to update Redundant State.
2021-10-07 08:57:05,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-6e168125) (logid:589e1fa9) Found 0 networks to update RvR status.
2021-10-07 08:57:09,727 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ab77452c) (logid:05331fda) Resetting hosts suitable for reconnect
2021-10-07 08:57:09,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ab77452c) (logid:05331fda) Completed resetting hosts suitable for reconnect
2021-10-07 08:57:09,732 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ab77452c) (logid:05331fda) Acquiring hosts for clusters already owned by this management server
2021-10-07 08:57:09,733 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ab77452c) (logid:05331fda) Completed acquiring hosts for clusters already owned by this management server
2021-10-07 08:57:09,733 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ab77452c) (logid:05331fda) Acquiring hosts for clusters not owned by any management server
2021-10-07 08:57:09,735 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-ab77452c) (logid:05331fda) Completed acquiring hosts for clusters not owned by any management server
2021-10-07 08:57:11,163 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 1-3253569255798472727: Processing: { Ans: , MgmtId: 90520741394843, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:57:11,163 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 1-3253569255798472727: No more commands found
2021-10-07 08:57:11,163 DEBUG [c.c.a.t.Request] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Seq 1-3253569255798472727: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2021-10-07 08:57:11,298 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) releasing lock for VMTemplateStoragePool 513
2021-10-07 08:57:11,362 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Unable to create Vol[1536|vm=1536|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
2021-10-07 08:57:11,362 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[1536|vm=1536|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1491)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1566)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1160)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:57:11,517 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Cleaning up resources for the vm VM[ConsoleProxy|v-1536-VM] in Starting state
2021-10-07 08:57:11,553 DEBUG [c.c.a.t.Request] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Seq 1-3253569255798472731: Sending { Cmd , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.216.138","forceStop":"false","volumesToDisconnect":[],"vmName":"v-1536-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:57:13,047 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Unable to acquire lock on VMTemplateStoragePool 513
2021-10-07 08:57:13,093 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Cleaning up resources for the vm VM[SecondaryStorageVm|s-1537-VM] in Starting state
2021-10-07 08:57:13,111 DEBUG [c.c.a.t.Request] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Seq 1-3253569255798472732: Sending { Cmd , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.101.45","forceStop":"false","volumesToDisconnect":[],"vmName":"s-1537-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:57:14,176 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 08:57:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 08:57:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 08:57:14,183 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) Run missing VM report. current time: 1633611434183
2021-10-07 08:57:14,188 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) Detected missing VM. host: 1, vm id: 1536(04319276-972c-424d-926d-86c5dd6af1b8), power state: PowerReportMissing, last state update: 2021/10/07T08:54:58.000Z
2021-10-07 08:57:14,188 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) vm id: 1536 - time since last state update(136183ms) has passed graceful period
2021-10-07 08:57:14,287 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) VM state report is updated. host: 1, vm id: 1536, power state: PowerReportMissing
2021-10-07 08:57:14,295 INFO [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-3:null) (logid:) There is pending job or HA tasks working on the VM. vm id: 1536, postpone power-change report by resetting power-change counters
2021-10-07 08:57:14,609 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1537
2021-10-07 08:57:14,609 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) Detected missing VM. host: 1, vm id: 1537(d008b3ee-dd01-4c46-a10f-05593bb59bb5), power state: PowerReportMissing, last state update: 2021/10/07T08:55:04.000Z
2021-10-07 08:57:14,609 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) vm id: 1537 - time since last state update(130183ms) has passed graceful period
2021-10-07 08:57:14,658 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) VM state report is updated. host: 1, vm id: 1537, power state: PowerReportMissing
2021-10-07 08:57:14,664 INFO [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-3:null) (logid:) There is pending job or HA tasks working on the VM. vm id: 1537, postpone power-change report by resetting power-change counters
2021-10-07 08:57:14,703 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-3:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 08:57:14,820 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 1-3253569255798472731: Processing: { Ans: , MgmtId: 90520741394843, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:57:14,820 DEBUG [c.c.a.t.Request] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Seq 1-3253569255798472731: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { StopAnswer } }
2021-10-07 08:57:14,895 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Released nic: NicProfile[5889-1536-null-null-null]
2021-10-07 08:57:14,961 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Releasing ip address for reservationId=ba6ec20b-2962-412e-8d82-3bac7791dc1a, instance=5890
2021-10-07 08:57:14,962 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Released nic: NicProfile[5890-1536-null-null-null]
2021-10-07 08:57:15,002 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Successfully released network resources for the vm VM[ConsoleProxy|v-1536-VM]
2021-10-07 08:57:15,002 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Successfully cleaned up resources for the VM VM[ConsoleProxy|v-1536-VM] in Starting state
2021-10-07 08:57:15,014 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) DeploymentPlanner allocation algorithm: null
2021-10-07 08:57:15,014 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: (1.00 GB) 1073741824
2021-10-07 08:57:15,014 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 08:57:15,019 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: [1]
2021-10-07 08:57:15,031 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Searching resources only under specified Pod: 1
2021-10-07 08:57:15,044 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Pod: 1
2021-10-07 08:57:15,101 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 08:57:15,111 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 08:57:15,112 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 08:57:15,118 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 08:57:15,160 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 08:57:15,160 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for speed=500Mhz, Ram=1024 MB
2021-10-07 08:57:15,194 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:57:15,194 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:15,204 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:15,204 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:57:15,204 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.90 GB) 66469449728 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:57:15,204 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:57:15,204 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:57:15,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 2, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:57:15,205 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 2
2021-10-07 08:57:15,205 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Host name: gd-cs01-clst01-compute01, hostId: 1 is in avoid set, skipping this and trying other available hosts
2021-10-07 08:57:15,205 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69 FirstFitRoutingAllocator) (logid:bcef1bbe) Host Allocator returning 1 suitable hosts
2021-10-07 08:57:15,219 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Checking suitable pools for volume (Id, Type): (1536,ROOT)
2021-10-07 08:57:15,219 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) We need to allocate new storagepool for this volume
2021-10-07 08:57:15,226 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Calling StoragePoolAllocators to find suitable pools
2021-10-07 08:57:15,248 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) System VMs will use shared storage for zone id=1
2021-10-07 08:57:15,248 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 08:57:15,248 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 08:57:15,248 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 08:57:15,252 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 08:57:15,258 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 08:57:15,258 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) StoragePool is in avoid set, skipping this pool
2021-10-07 08:57:15,258 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator returning 0 suitable storage pools
2021-10-07 08:57:15,258 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) ZoneWideStoragePoolAllocator to find storage pool
2021-10-07 08:57:15,268 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) No suitable pools found for volume: Vol[1536|vm=1536|ROOT] under cluster: 1
2021-10-07 08:57:15,268 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) No suitable pools found
2021-10-07 08:57:15,268 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) No suitable storagePools found under this Cluster: 1
2021-10-07 08:57:15,279 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2021-10-07 08:57:15,281 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Searching resources only under specified Pod: 1
2021-10-07 08:57:15,285 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Pod: 1
2021-10-07 08:57:15,291 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Removing from the clusterId list these clusters from avoid set: [1]
2021-10-07 08:57:15,294 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) No clusters found after removing disabled clusters and clusters in avoid list, returning.
2021-10-07 08:57:15,402 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 1
2021-10-07 08:57:15,433 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:15,433 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Hosts's actual total RAM: (61.89 GB) 66452602880 and RAM after applying overprovisioning: (61.89 GB) 66452602880
2021-10-07 08:57:15,433 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) release cpu from host: 1, old used: 1000,reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used: 500,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:57:15,433 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) release mem from host: 1, old used: (1.50 GB) 1610612736,reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; new used: (512.00 MB) 536870912,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:57:15,491 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1536-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 08:57:15,492 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731 ctx-0a096f69) (logid:bcef1bbe) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1536-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 08:57:15,492 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 1536, job origin: 3439
2021-10-07 08:57:15,492 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Unable to complete AsyncJobVO {id:4731, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:55:01 EDT 2021, removed: null}, job origin:3439
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1536-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:57:15,498 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Complete async job-4731, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyADdjb20uY2xvdWQuZXhjZXB0aW9uLkluc3VmZmljaWVudFNlcnZlckNhcGFjaXR5RXhjZXB0aW9uAAAAAFZNT3ACAAFaABVhZmZpbml0eUdyb3Vwc0FwcGxpZWR4cgAxY29tLmNsb3VkLmV4Y2VwdGlvbi5JbnN1ZmZpY2llbnRDYXBhY2l0eUV4Y2VwdGlvbgAAAABWTU91AgACTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc2NvcGV0ABFMamF2YS9sYW5nL0NsYXNzO3hyACJjb20uY2xvdWQuZXhjZXB0aW9uLkNsb3VkRXhjZXB0aW9ueeiOG_HkRMcCAAJMAAtjc0Vycm9yQ29kZXQAE0xqYXZhL2xhbmcvSW50ZWdlcjtMAAZpZExpc3R0ABVMamF2YS91dGlsL0FycmF5TGlzdDt4cgATamF2YS5sYW5nLkV4Y2VwdGlvbtD9Hz4aOxzEAgAAeHIAE2phdmEubGFuZy5UaHJvd2FibGXVxjUnOXe4ywMABEwABWNhdXNldAAVTGphdmEvbGFuZy9UaHJvd2FibGU7TAANZGV0YWlsTWVzc2FnZXQAEkxqYXZhL2xhbmcvU3RyaW5nO1sACnN0YWNrVHJhY2V0AB5bTGphdmEvbGFuZy9TdGFja1RyYWNlRWxlbWVudDtMABRzdXBwcmVzc2VkRXhjZXB0aW9uc3QAEExqYXZhL3V0aWwvTGlzdDt4cHEAfgANdAA8VW5hYmxlIHRvIGNyZWF0ZSBhIGRlcGxveW1lbnQgZm9yIFZNW0NvbnNvbGVQcm94eXx2LTE1MzYtVk1ddXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAUc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAIQgAGZm9ybWF0SQAKbGluZU51bWJlckwAD2NsYXNzTG9hZGVyTmFtZXEAfgAKTAAOZGVjbGFyaW5nQ2xhc3NxAH4ACkwACGZpbGVOYW1lcQB-AApMAAptZXRob2ROYW1lcQB-AApMAAptb2R1bGVOYW1lcQB-AApMAA1tb2R1bGVWZXJzaW9ucQB-AAp4cAEAAARfdAADYXBwdAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0ABBvcmNoZXN0cmF0ZVN0YXJ0cHBzcQB-ABEBAAAVfnEAfgATcQB-ABRxAH4AFXEAfgAWcHBzcQB-ABEA_____3B0AC9qZGsuaW50ZXJuYWwucmVmbGVjdC5HZW5lcmF0ZWRNZXRob2RBY2Nlc3NvcjIyNXB0AAZpbnZva2VwcHNxAH4AEQIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AGnQACWphdmEuYmFzZXQABzExLjAuMTFzcQB-ABECAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAacQB-AB5xAH4AH3NxAH4AEQEAAABrcQB-ABN0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4AEQEAABYlcQB-ABNxAH4AFHEAfgAVcQB-ACZwcHNxAH4AEQEAAABmcQB-ABN0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnBwc3EAfgARAQAAAmxxAH4AE3QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgARAQAAADBxAH4AE3QAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-ABEBAAAAN3EAfgATdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-ABEBAAAAZnEAfgATdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA2dAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-ABEBAAAANHEAfgATcQB-ADlxAH4ANnQADnJ1bldpdGhDb250ZXh0cHBzcQB-ABEBAAAALXEAfgATdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ADJxAH4AM3Bwc3EAfgARAQAAAjhxAH4AE3EAfgAtcQB-AC5xAH4AM3Bwc3EAfgARAgAAAgNwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ADdxAH4AHnEAfgAfc3EAfgARAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAzcQB-AB5xAH4AH3NxAH4AEQIAAARocHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-AB5xAH4AH3NxAH4AEQIAAAJ0cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ASHEAfgAzcQB-AB5xAH4AH3NxAH4AEQIAAAM9cHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAzcQB-AB5xAH4AH3NyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHhzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAQ73NyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cQB-AFIAAAAAAAAAAXZyABdjb20uY2xvdWQuZGMuRGF0YUNlbnRlclPoJBrS07ZcAgAAeHAA
2021-10-07 08:57:15,503 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Publish async job-4731 complete on message bus
2021-10-07 08:57:15,503 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Wake up jobs related to job-4731
2021-10-07 08:57:15,503 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Update db status for job-4731
2021-10-07 08:57:15,510 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Wake up jobs joined with job-4731 and disjoin all subjobs created from job- 4731
2021-10-07 08:57:15,639 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Done executing com.cloud.vm.VmWorkStart for job-4731
2021-10-07 08:57:15,646 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-128:ctx-041b04f8 job-3439/job-4731) (logid:bcef1bbe) Remove job-4731 from job monitoring
2021-10-07 08:57:15,711 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Exception while trying to start console proxy
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1536-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:57:15,712 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Unable to start console proxy vm for standby capacity, vm id : 1536, will recycle it and start a new one
2021-10-07 08:57:15,776 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Sync job-4733 execution on object VmWorkJobQueue.1536
2021-10-07 08:57:15,969 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0cf9937d) (logid:2f8617c9) StorageCollector is running...
2021-10-07 08:57:16,016 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-0cf9937d) (logid:2f8617c9) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 08:57:16,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:ctx-0cf9937d) (logid:2f8617c9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 08:57:16,023 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:ctx-0cf9937d) (logid:2f8617c9) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 08:57:16,287 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-0cf9937d) (logid:2f8617c9) Seq 1-3253569255798472733: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 08:57:16,361 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:null) (logid:) Seq 1-3253569255798472732: Processing: { Ans: , MgmtId: 90520741394843, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:57:16,362 DEBUG [c.c.a.t.Request] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Seq 1-3253569255798472732: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { StopAnswer } }
2021-10-07 08:57:16,367 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a4b237a2) (logid:92c4cd2f) Execute sync-queue item: SyncQueueItemVO {id:4697, queueId: 4693, contentType: AsyncJob, contentId: 4733, lastProcessMsid: 90520741394843, lastprocessNumber: 3, lastProcessTime: Thu Oct 07 08:57:16 EDT 2021, created: Thu Oct 07 08:57:15 EDT 2021}
2021-10-07 08:57:16,371 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-a4b237a2) (logid:92c4cd2f) Schedule queued job-4733
2021-10-07 08:57:16,428 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733) (logid:6e414521) Add job-4733 into job monitoring
2021-10-07 08:57:16,438 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733) (logid:bcef1bbe) Executing AsyncJobVO {id:4733, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:57:15 EDT 2021, removed: null}
2021-10-07 08:57:16,439 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733) (logid:bcef1bbe) Run VM work job: com.cloud.vm.VmWorkStop for VM 1536, job origin: 3439
2021-10-07 08:57:16,444 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1536,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:57:16,452 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) VM is already stopped: VM[ConsoleProxy|v-1536-VM]
2021-10-07 08:57:16,452 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1536,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:57:16,454 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) Complete async job-4733, jobStatus: SUCCEEDED, resultCode: 0, result: null
2021-10-07 08:57:16,457 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Released nic: NicProfile[5892-1537-null-null-null]
2021-10-07 08:57:16,457 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) Publish async job-4733 complete on message bus
2021-10-07 08:57:16,457 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) Wake up jobs related to job-4733
2021-10-07 08:57:16,457 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) Update db status for job-4733
2021-10-07 08:57:16,459 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733 ctx-305b738b) (logid:bcef1bbe) Wake up jobs joined with job-4733 and disjoin all subjobs created from job- 4733
2021-10-07 08:57:16,530 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733) (logid:bcef1bbe) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 1536, job origin: 3439
2021-10-07 08:57:16,530 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733) (logid:bcef1bbe) Done executing com.cloud.vm.VmWorkStop for job-4733
2021-10-07 08:57:16,534 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Releasing ip address for reservationId=b2622726-a201-431c-8887-4794da95439b, instance=5893
2021-10-07 08:57:16,535 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Released nic: NicProfile[5893-1537-null-null-null]
2021-10-07 08:57:16,535 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-129:ctx-4c9992dd job-3439/job-4733) (logid:bcef1bbe) Remove job-4733 from job monitoring
2021-10-07 08:57:16,593 DEBUG [c.c.n.g.StorageNetworkGuru] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Release an storage ip 192.168.204.187
2021-10-07 08:57:16,630 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Successfully released network resources for the vm VM[SecondaryStorageVm|s-1537-VM]
2021-10-07 08:57:16,630 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Successfully cleaned up resources for the VM VM[SecondaryStorageVm|s-1537-VM] in Starting state
2021-10-07 08:57:16,651 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) VM state transitted from :Stopped to Expunging with event: ExpungeOperationvm's original host id: null new host id: null host id before state transition: null
2021-10-07 08:57:16,684 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Destroying vm VM[ConsoleProxy|v-1536-VM]
2021-10-07 08:57:16,684 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Cleaning up NICS
2021-10-07 08:57:16,684 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Cleaning network for vm: 1536
2021-10-07 08:57:16,716 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 1
2021-10-07 08:57:16,750 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:16,750 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Hosts's actual total RAM: (61.89 GB) 66452602880 and RAM after applying overprovisioning: (61.89 GB) 66452602880
2021-10-07 08:57:16,750 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) release cpu from host: 1, old used: 500,reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:57:16,750 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) release mem from host: 1, old used: (512.00 MB) 536870912,reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; new used: (0 bytes) 0,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:57:16,764 DEBUG [c.c.n.g.PublicNetworkGuru] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) public network deallocate network: networkId: 200, ip: 192.168.200.199
2021-10-07 08:57:16,788 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Caught CloudRuntimeException, returning job failed com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on VMTemplateStoragePool: 513
2021-10-07 08:57:16,802 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Done executing VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1537,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:57:16,806 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Complete async job-4732, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ABtVbmFibGUgdG8gc3RhcnQgVk0gaW5zdGFuY2V1cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABNzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAAhCAAZmb3JtYXRJAApsaW5lTnVtYmVyTAAPY2xhc3NMb2FkZXJOYW1lcQB-AAVMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABUwACm1vZHVsZU5hbWVxAH4ABUwADW1vZHVsZVZlcnNpb25xAH4ABXhwAQAAFYN0AANhcHB0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAEG9yY2hlc3RyYXRlU3RhcnRwcHNxAH4ADAD_____cHQAL2pkay5pbnRlcm5hbC5yZWZsZWN0LkdlbmVyYXRlZE1ldGhvZEFjY2Vzc29yMjI1cHQABmludm9rZXBwc3EAfgAMAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAUdAAJamF2YS5iYXNldAAHMTEuMC4xMXNxAH4ADAIAAAI2cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ABRxAH4AGHEAfgAZc3EAfgAMAQAAAGtxAH4ADnQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgAMAQAAFiVxAH4ADnEAfgAPcQB-ABBxAH4AIHBwc3EAfgAMAQAAAGZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACbHEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMHEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA3cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABmcQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADB0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA0cQB-AA5xAH4AM3EAfgAwdAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAtcQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ALHEAfgAtcHBzcQB-AAwBAAACOHEAfgAOcQB-ACdxAH4AKHEAfgAtcHBzcQB-AAwCAAACA3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AMXEAfgAYcQB-ABlzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AGHEAfgAZc3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBCcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAAAz1wdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AC1xAH4AGHEAfgAZc3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2021-10-07 08:57:16,810 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Publish async job-4732 complete on message bus
2021-10-07 08:57:16,810 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Wake up jobs related to job-4732
2021-10-07 08:57:16,810 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Update db status for job-4732
2021-10-07 08:57:16,814 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732 ctx-5a81b320) (logid:e381c0f1) Wake up jobs joined with job-4732 and disjoin all subjobs created from job- 4732
2021-10-07 08:57:16,816 DEBUG [c.c.n.g.PublicNetworkGuru] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Deallocated nic: NicProfile[5888-1536-null-null-null]
2021-10-07 08:57:16,851 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Removed nic id=5888
2021-10-07 08:57:16,859 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Revoving nic secondary ip entry ...
2021-10-07 08:57:16,885 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732) (logid:e381c0f1) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 1537, job origin: 3440
2021-10-07 08:57:16,885 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732) (logid:e381c0f1) Done executing com.cloud.vm.VmWorkStart for job-4732
2021-10-07 08:57:16,892 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-131:ctx-51b5895e job-3440/job-4732) (logid:e381c0f1) Remove job-4732 from job monitoring
2021-10-07 08:57:16,942 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Exception while trying to start secondary storage vm
com.cloud.utils.exception.CloudRuntimeException: Unable to start VM instance
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5507)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:57:16,948 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Unable to start secondary storage vm for standby capacity, vm id : 1537, will recycle it and start a new one
2021-10-07 08:57:16,958 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Removed nic id=5889
2021-10-07 08:57:16,960 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Revoving nic secondary ip entry ...
2021-10-07 08:57:17,029 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Sync job-4734 execution on object VmWorkJobQueue.1537
2021-10-07 08:57:17,059 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Removed nic id=5890
2021-10-07 08:57:17,062 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Revoving nic secondary ip entry ...
2021-10-07 08:57:17,065 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage
2021-10-07 08:57:17,069 DEBUG [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Cleaning storage for vm: 1536
2021-10-07 08:57:17,274 INFO [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Expunge volume with no data store specified
2021-10-07 08:57:17,283 INFO [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Volume 1536 is not referred anywhere, remove it from volumes table
2021-10-07 08:57:17,368 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Expunged VM[ConsoleProxy|v-1536-VM]
2021-10-07 08:57:17,434 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) received console proxy alert
2021-10-07 08:57:17,439 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) Console proxy creation failure, zone: gd-cs01
2021-10-07 08:57:17,445 WARN [c.c.a.AlertManagerImpl] (consoleproxy-1:ctx-d67a82a5) (logid:5fb5524f) AlertType:: 10 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Console proxy creation failure. zone: gd-cs01, error details: null
2021-10-07 08:57:17,562 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Zone 1 is ready to launch console proxy
2021-10-07 08:57:17,562 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Expand console proxy standby capacity for zone gd-cs01
2021-10-07 08:57:17,565 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) No stopped console proxy is available, need to allocate a new console proxy
2021-10-07 08:57:17,568 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Assign console proxy from a newly started instance for request from data center : 1
2021-10-07 08:57:17,688 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Found existing network configuration for offering [Network Offering [1-Public-System-Public-Network]: Ntwk[200|Public|1]
2021-10-07 08:57:17,689 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 08:57:17,795 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2021-10-07 08:57:17,795 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 08:57:17,886 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]: Ntwk[201|Management|2]
2021-10-07 08:57:17,886 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 08:57:18,102 INFO [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) allocating virtual machine from template:8c443ce1-22f9-11ec-a65c-52540037f312 with hostname:v-1538-VM and 3 networks
2021-10-07 08:57:18,113 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Allocating entries for VM: VM[ConsoleProxy|v-1538-VM]
2021-10-07 08:57:18,122 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Allocating nics for VM[ConsoleProxy|v-1538-VM]
2021-10-07 08:57:18,129 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Allocating nic for vm VM[ConsoleProxy|v-1538-VM] in network Ntwk[200|Public|1] with requested profile NicProfile[0-0-null-null-null]
2021-10-07 08:57:18,228 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Allocating nic for vm VM[ConsoleProxy|v-1538-VM] in network Ntwk[202|Control|3] with requested profile null
2021-10-07 08:57:18,245 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Allocating nic for vm VM[ConsoleProxy|v-1538-VM] in network Ntwk[201|Management|2] with requested profile null
2021-10-07 08:57:18,266 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Allocating disks for VM[ConsoleProxy|v-1538-VM]
2021-10-07 08:57:18,267 INFO [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) adding disk object ROOT-1538 to v-1538-VM
2021-10-07 08:57:18,269 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-011af25d) (logid:1f651004) Execute sync-queue item: SyncQueueItemVO {id:4698, queueId: 4694, contentType: AsyncJob, contentId: 4734, lastProcessMsid: 90520741394843, lastprocessNumber: 3, lastProcessTime: Thu Oct 07 08:57:18 EDT 2021, created: Thu Oct 07 08:57:17 EDT 2021}
2021-10-07 08:57:18,272 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-011af25d) (logid:1f651004) Schedule queued job-4734
2021-10-07 08:57:18,318 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734) (logid:d081627f) Add job-4734 into job monitoring
2021-10-07 08:57:18,329 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734) (logid:e381c0f1) Executing AsyncJobVO {id:4734, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:57:17 EDT 2021, removed: null}
2021-10-07 08:57:18,329 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734) (logid:e381c0f1) Run VM work job: com.cloud.vm.VmWorkStop for VM 1537, job origin: 3440
2021-10-07 08:57:18,337 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1537,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:57:18,353 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) VM is already stopped: VM[SecondaryStorageVm|s-1537-VM]
2021-10-07 08:57:18,353 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1537,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:57:18,358 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) Complete async job-4734, jobStatus: SUCCEEDED, resultCode: 0, result: null
2021-10-07 08:57:18,365 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) Publish async job-4734 complete on message bus
2021-10-07 08:57:18,365 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) Wake up jobs related to job-4734
2021-10-07 08:57:18,365 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) Update db status for job-4734
2021-10-07 08:57:18,371 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734 ctx-42b33adc) (logid:e381c0f1) Wake up jobs joined with job-4734 and disjoin all subjobs created from job- 4734
2021-10-07 08:57:18,386 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Allocation completed for VM: VM[ConsoleProxy|v-1538-VM]
2021-10-07 08:57:18,392 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) received console proxy alert
2021-10-07 08:57:18,394 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) New console proxy created, zone: gd-cs01, proxy: v-1538-VM, public IP: null, private IP: null
2021-10-07 08:57:18,471 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734) (logid:e381c0f1) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 1537, job origin: 3440
2021-10-07 08:57:18,471 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734) (logid:e381c0f1) Done executing com.cloud.vm.VmWorkStop for job-4734
2021-10-07 08:57:18,475 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-132:ctx-7340f1d7 job-3440/job-4734) (logid:e381c0f1) Remove job-4734 from job monitoring
2021-10-07 08:57:18,510 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Sync job-4735 execution on object VmWorkJobQueue.1538
2021-10-07 08:57:18,604 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) VM state transitted from :Stopped to Expunging with event: ExpungeOperationvm's original host id: null new host id: null host id before state transition: null
2021-10-07 08:57:18,626 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Destroying vm VM[SecondaryStorageVm|s-1537-VM]
2021-10-07 08:57:18,626 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Cleaning up NICS
2021-10-07 08:57:18,626 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Cleaning network for vm: 1537
2021-10-07 08:57:18,819 DEBUG [c.c.n.g.PublicNetworkGuru] (secstorage-1:ctx-9917b96e) (logid:e07f807d) public network deallocate network: networkId: 200, ip: 192.168.200.200
2021-10-07 08:57:18,872 DEBUG [c.c.n.g.PublicNetworkGuru] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Deallocated nic: NicProfile[5891-1537-null-null-null]
2021-10-07 08:57:18,913 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Removed nic id=5891
2021-10-07 08:57:18,916 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Revoving nic secondary ip entry ...
2021-10-07 08:57:18,988 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Removed nic id=5892
2021-10-07 08:57:18,991 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Revoving nic secondary ip entry ...
2021-10-07 08:57:19,089 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Removed nic id=5893
2021-10-07 08:57:19,092 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Revoving nic secondary ip entry ...
2021-10-07 08:57:19,196 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Removed nic id=5894
2021-10-07 08:57:19,199 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Revoving nic secondary ip entry ...
2021-10-07 08:57:19,201 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage
2021-10-07 08:57:19,203 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Cleaning storage for vm: 1537
2021-10-07 08:57:19,281 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Expunge volume with no data store specified
2021-10-07 08:57:19,286 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Volume 1537 is not referred anywhere, remove it from volumes table
2021-10-07 08:57:19,358 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Expunged VM[SecondaryStorageVm|s-1537-VM]
2021-10-07 08:57:19,560 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-9917b96e) (logid:e07f807d) received secondary storage vm alert
2021-10-07 08:57:19,563 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Secondary Storage Vm creation failure, zone: gd-cs01
2021-10-07 08:57:19,565 WARN [c.c.a.AlertManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: gd-cs01, error details: null
2021-10-07 08:57:19,685 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-9917b96e) (logid:e07f807d) Primary secondary storage is not even started, wait until next turn
2021-10-07 08:57:19,727 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Zone 1 is ready to launch secondary storage VM
2021-10-07 08:57:19,737 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) No running secondary storage vms found in datacenter id=1, starting one
2021-10-07 08:57:19,746 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
2021-10-07 08:57:19,751 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Assign secondary storage vm from a newly started instance for request from data center : 1
2021-10-07 08:57:19,966 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Found existing network configuration for offering [Network Offering [1-Public-System-Public-Network]: Ntwk[200|Public|1]
2021-10-07 08:57:19,966 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 08:57:20,050 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2021-10-07 08:57:20,050 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 08:57:20,142 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]: Ntwk[201|Management|2]
2021-10-07 08:57:20,142 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 08:57:20,234 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Found existing network configuration for offering [Network Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
2021-10-07 08:57:20,234 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 08:57:20,258 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-92878edb) (logid:164634b8) Execute sync-queue item: SyncQueueItemVO {id:4699, queueId: 4699, contentType: AsyncJob, contentId: 4735, lastProcessMsid: 90520741394843, lastprocessNumber: 1, lastProcessTime: Thu Oct 07 08:57:20 EDT 2021, created: Thu Oct 07 08:57:18 EDT 2021}
2021-10-07 08:57:20,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-92878edb) (logid:164634b8) Schedule queued job-4735
2021-10-07 08:57:20,314 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:8032829d) Add job-4735 into job monitoring
2021-10-07 08:57:20,327 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Executing AsyncJobVO {id:4735, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:57:18 EDT 2021, removed: null}
2021-10-07 08:57:20,327 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Run VM work job: com.cloud.vm.VmWorkStart for VM 1538, job origin: 3439
2021-10-07 08:57:20,332 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Execute VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1538,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:57:20,349 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2021-10-07 08:57:20,351 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Successfully transitioned to start state for VM[ConsoleProxy|v-1538-VM] reservation id = 6f5e31ff-d834-4e4c-8260-8324b243ff39
2021-10-07 08:57:20,380 INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) allocating virtual machine from template:8c443ce1-22f9-11ec-a65c-52540037f312 with hostname:s-1539-VM and 4 networks
2021-10-07 08:57:20,389 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Allocating entries for VM: VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:57:20,393 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Allocating nics for VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:57:20,396 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Allocating nic for vm VM[SecondaryStorageVm|s-1539-VM] in network Ntwk[200|Public|1] with requested profile NicProfile[0-0-null-null-null]
2021-10-07 08:57:20,406 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:57:20,413 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) DeploymentPlanner allocation algorithm: null
2021-10-07 08:57:20,413 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB) 1073741824
2021-10-07 08:57:20,413 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 08:57:20,416 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:57:20,421 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Searching all possible resources under this Zone: 1
2021-10-07 08:57:20,426 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 08:57:20,446 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Allocating nic for vm VM[SecondaryStorageVm|s-1539-VM] in network Ntwk[202|Control|3] with requested profile null
2021-10-07 08:57:20,458 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 08:57:20,459 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Allocating nic for vm VM[SecondaryStorageVm|s-1539-VM] in network Ntwk[201|Management|2] with requested profile null
2021-10-07 08:57:20,463 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 08:57:20,463 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 08:57:20,470 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) FirstFitAllocator has 2 hosts to check for allocation: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:57:20,482 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) Allocating nic for vm VM[SecondaryStorageVm|s-1539-VM] in network Ntwk[203|Storage|4] with requested profile null
2021-10-07 08:57:20,489 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Found 2 hosts for allocation after prioritization: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:57:20,489 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for speed=500Mhz, Ram=1024 MB
2021-10-07 08:57:20,499 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Allocating disks for VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:57:20,499 INFO [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-995d956e) (logid:06956492) adding disk object ROOT-1539 to s-1539-VM
2021-10-07 08:57:20,510 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:57:20,510 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:20,521 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:20,521 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:57:20,521 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:57:20,521 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:57:20,521 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:57:20,521 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:57:20,521 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 1
2021-10-07 08:57:20,545 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:57:20,545 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:20,553 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:20,554 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:57:20,554 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.90 GB) 66469449728 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:57:20,554 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:57:20,554 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:57:20,554 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 2, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:57:20,554 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 2
2021-10-07 08:57:20,554 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host Allocator returning 2 suitable hosts
2021-10-07 08:57:20,577 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking suitable pools for volume (Id, Type): (1538,ROOT)
2021-10-07 08:57:20,577 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) We need to allocate new storagepool for this volume
2021-10-07 08:57:20,582 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Calling StoragePoolAllocators to find suitable pools
2021-10-07 08:57:20,584 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Allocation completed for VM: VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:57:20,621 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) System VMs will use shared storage for zone id=1
2021-10-07 08:57:20,622 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 08:57:20,622 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 08:57:20,622 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 08:57:20,637 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 08:57:20,638 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-995d956e) (logid:06956492) received secondary storage vm alert
2021-10-07 08:57:20,640 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-995d956e) (logid:06956492) New secondary storage vm created, zone: gd-cs01, secStorageVm: s-1539-VM, public IP: null, private IP: null
2021-10-07 08:57:20,641 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 08:57:20,645 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Storage pool null (2) does not supply IOPS capacity, assuming enough capacity
2021-10-07 08:57:20,649 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking pool 2 for storage, totalSize: 30005954347008, usedBytes: 11020345267, usedPct: 3.6727194674609234E-4, disable threshold: 0.85
2021-10-07 08:57:20,649 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Destination pool id: 2
2021-10-07 08:57:20,699 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Pool ID for the volume with ID 1538 is null
2021-10-07 08:57:20,708 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Sync job-4736 execution on object VmWorkJobQueue.1539
2021-10-07 08:57:20,716 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Found storage pool CS_Ceph of type RBD with over-provisioning factor 2
2021-10-07 08:57:20,716 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Total over-provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 08:57:20,716 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Total capacity of the pool CS_Ceph with ID 2 is (54.5805 TB) 60011908694016
2021-10-07 08:57:20,721 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking pool: 2 for storage allocation , maxSize : (54.5805 TB) 60011908694016, totalAllocatedSize : (0 bytes) 0, askingSize : (0 bytes) 0, allocated disable threshold: 0.85
2021-10-07 08:57:20,721 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
2021-10-07 08:57:20,721 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
2021-10-07 08:57:20,724 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking if host: 1 can access any suitable storage pool for volume: ROOT
2021-10-07 08:57:20,727 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Host: 1 can access pool: 2
2021-10-07 08:57:20,730 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Found a potential host id: 1 name: gd-cs01-clst01-compute01 and associated storage pools for this VM
2021-10-07 08:57:20,732 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1538|ROOT-->Pool(2))]
2021-10-07 08:57:20,732 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Deployment found - P0=VM[ConsoleProxy|v-1538-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1538|ROOT-->Pool(2))]
2021-10-07 08:57:20,802 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 1 host id before state transition: null
2021-10-07 08:57:20,858 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:20,858 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) We are allocating VM, increasing the used capacity of this host:1
2021-10-07 08:57:20,858 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Current Used CPU: 0 , Free CPU:73600 ,Requested CPU: 500
2021-10-07 08:57:20,858 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Current Used RAM: (0 bytes) 0 , Free RAM:(61.89 GB) 66452602880 ,Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:57:20,859 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) CPU STATS after allocation: for host: 1, old used: 0, old reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
2021-10-07 08:57:20,859 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) RAM STATS after allocation: for host: 1, old used: (0 bytes) 0, old reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; new used: (1.00 GB) 1073741824, reserved: (0 bytes) 0; requested mem: (1.00 GB) 1073741824,alloc_from_last:false
2021-10-07 08:57:20,869 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:57:20,869 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:20,876 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:20,876 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) We need to allocate to the last host again, so checking if there is enough reserved capacity
2021-10-07 08:57:20,876 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Reserved CPU: 0 , Requested CPU: 500
2021-10-07 08:57:20,876 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Reserved RAM: (0 bytes) 0 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:57:20,877 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) STATS: Failed to alloc resource from host: 1 reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (1.00 GB) 1073741824
2021-10-07 08:57:20,877 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Host does not have enough reserved CPU available, cannot allocate to this host.
2021-10-07 08:57:20,877 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:20,884 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:20,884 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:57:20,884 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:57:20,884 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:57:20,884 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:57:20,884 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:57:20,960 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Network id=200 is already implemented
2021-10-07 08:57:21,028 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Network id=202 is already implemented
2021-10-07 08:57:21,647 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Reserved NIC for v-1538-VM [ipv4:169.254.26.151 netmask:255.255.0.0 gateway:169.254.0.1]
2021-10-07 08:57:21,801 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Network id=201 is already implemented
2021-10-07 08:57:22,158 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Releasing ip address for instance=5897
2021-10-07 08:57:22,245 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Allocated a nic NicProfile[5897-1538-6f5e31ff-d834-4e4c-8260-8324b243ff39-192.168.204.159-null] for VM[ConsoleProxy|v-1538-VM]
2021-10-07 08:57:22,272 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9fbce8d9) (logid:2ec031c0) Execute sync-queue item: SyncQueueItemVO {id:4700, queueId: 4700, contentType: AsyncJob, contentId: 4736, lastProcessMsid: 90520741394843, lastprocessNumber: 1, lastProcessTime: Thu Oct 07 08:57:22 EDT 2021, created: Thu Oct 07 08:57:20 EDT 2021}
2021-10-07 08:57:22,275 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-9fbce8d9) (logid:2ec031c0) Schedule queued job-4736
2021-10-07 08:57:22,343 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736) (logid:1f32dad4) Add job-4736 into job monitoring
2021-10-07 08:57:22,355 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736) (logid:e381c0f1) Executing AsyncJobVO {id:4736, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGA3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:57:20 EDT 2021, removed: null}
2021-10-07 08:57:22,355 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736) (logid:e381c0f1) Run VM work job: com.cloud.vm.VmWorkStart for VM 1539, job origin: 3440
2021-10-07 08:57:22,359 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Execute VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1539,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:57:22,369 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) template 3 is already in store:5, type:Image
2021-10-07 08:57:22,383 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2021-10-07 08:57:22,384 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Successfully transitioned to start state for VM[SecondaryStorageVm|s-1539-VM] reservation id = 08316455-6db7-4d2c-bf6d-97762afb240b
2021-10-07 08:57:22,392 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Not found (templateId:3poolId:2conf:null) in template_spool_ref, persisting it
2021-10-07 08:57:22,432 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:57:22,439 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) DeploymentPlanner allocation algorithm: null
2021-10-07 08:57:22,439 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: (512.00 MB) 536870912
2021-10-07 08:57:22,439 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 08:57:22,445 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:57:22,453 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Searching all possible resources under this Zone: 1
2021-10-07 08:57:22,458 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 08:57:22,464 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) template 3 is already in store:2, type:Primary
2021-10-07 08:57:22,468 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 514
2021-10-07 08:57:22,470 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Acquire lock on VMTemplateStoragePool 514 with timeout 3600 seconds
2021-10-07 08:57:22,493 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 08:57:22,498 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 08:57:22,499 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 08:57:22,506 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 08:57:22,506 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) lock is acquired for VMTemplateStoragePool 514
2021-10-07 08:57:22,522 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 08:57:22,522 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Looking for speed=500Mhz, Ram=512 MB
2021-10-07 08:57:22,538 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:57:22,538 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:22,544 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:22,544 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:57:22,544 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (61.90 GB) 66469449728 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:57:22,544 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:57:22,544 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:57:22,544 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 2, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:57:22,544 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 2
2021-10-07 08:57:22,560 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2021-10-07 08:57:22,560 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:57:22,560 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:22,565 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:22,565 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73100 , Requested CPU: 500
2021-10-07 08:57:22,565 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (60.89 GB) 65378861056 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:57:22,565 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:57:22,565 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 1, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:57:22,565 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 1, used: (1.00 GB) 1073741824, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:57:22,565 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 1
2021-10-07 08:57:22,565 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc FirstFitRoutingAllocator) (logid:e381c0f1) Host Allocator returning 2 suitable hosts
2021-10-07 08:57:22,571 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking suitable pools for volume (Id, Type): (1539,ROOT)
2021-10-07 08:57:22,571 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) We need to allocate new storagepool for this volume
2021-10-07 08:57:22,572 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Calling StoragePoolAllocators to find suitable pools
2021-10-07 08:57:22,578 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) System VMs will use shared storage for zone id=1
2021-10-07 08:57:22,578 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 08:57:22,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 08:57:22,578 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 08:57:22,580 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 08:57:22,583 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 08:57:22,586 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Storage pool null (2) does not supply IOPS capacity, assuming enough capacity
2021-10-07 08:57:22,589 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking pool 2 for storage, totalSize: 30005954347008, usedBytes: 11020345267, usedPct: 3.6727194674609234E-4, disable threshold: 0.85
2021-10-07 08:57:22,589 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Destination pool id: 2
2021-10-07 08:57:22,590 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand
2021-10-07 08:57:22,596 DEBUG [c.c.a.t.Request] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Seq 1-3253569255798472734: Sending { Cmd , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.204.62/cloudstack","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6db2511f-6992-3120-b49e-fafdfa8ec257","id":"2","poolType":"RBD","host":"192.168.206.63","path":"cloudstack","port":"0","url":"RBD://192.168.206.63/cloudstack/?ROLE=Primary&STOREUUID=6db2511f-6992-3120-b49e-fafdfa8ec257","isManaged":"false"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"10800","bypassHostMaintenance":"false"}}] }
2021-10-07 08:57:22,619 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Pool ID for the volume with ID 1539 is null
2021-10-07 08:57:22,631 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Found storage pool CS_Ceph of type RBD with over-provisioning factor 2
2021-10-07 08:57:22,631 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Total over-provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 08:57:22,632 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Total capacity of the pool CS_Ceph with ID 2 is (54.5805 TB) 60011908694016
2021-10-07 08:57:22,634 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking pool: 2 for storage allocation , maxSize : (54.5805 TB) 60011908694016, totalAllocatedSize : (0 bytes) 0, askingSize : (0 bytes) 0, allocated disable threshold: 0.85
2021-10-07 08:57:22,634 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
2021-10-07 08:57:22,634 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
2021-10-07 08:57:22,635 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking if host: 2 can access any suitable storage pool for volume: ROOT
2021-10-07 08:57:22,640 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Host: 2 can access pool: 2
2021-10-07 08:57:22,642 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Found a potential host id: 2 name: gd-cs01-clst01-compute02 and associated storage pools for this VM
2021-10-07 08:57:22,643 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1539|ROOT-->Pool(2))]
2021-10-07 08:57:22,643 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Deployment found - P0=VM[SecondaryStorageVm|s-1539-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1539|ROOT-->Pool(2))]
2021-10-07 08:57:22,706 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 2 host id before state transition: null
2021-10-07 08:57:22,730 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:22,730 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) We are allocating VM, increasing the used capacity of this host:2
2021-10-07 08:57:22,730 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Current Used CPU: 0 , Free CPU:73600 ,Requested CPU: 500
2021-10-07 08:57:22,730 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Current Used RAM: (0 bytes) 0 , Free RAM:(61.90 GB) 66469449728 ,Requested RAM: (512.00 MB) 536870912
2021-10-07 08:57:22,730 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) CPU STATS after allocation: for host: 2, old used: 0, old reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
2021-10-07 08:57:22,730 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) RAM STATS after allocation: for host: 2, old used: (0 bytes) 0, old reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; new used: (512.00 MB) 536870912, reserved: (0 bytes) 0; requested mem: (512.00 MB) 536870912,alloc_from_last:false
2021-10-07 08:57:22,738 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:57:22,738 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:22,741 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:22,741 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) We need to allocate to the last host again, so checking if there is enough reserved capacity
2021-10-07 08:57:22,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Reserved CPU: 0 , Requested CPU: 500
2021-10-07 08:57:22,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Reserved RAM: (0 bytes) 0 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:57:22,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) STATS: Failed to alloc resource from host: 2 reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (512.00 MB) 536870912
2021-10-07 08:57:22,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Host does not have enough reserved CPU available, cannot allocate to this host.
2021-10-07 08:57:22,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:57:22,746 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:57:22,746 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:57:22,746 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Free RAM: (61.90 GB) 66469449728 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:57:22,746 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:57:22,746 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:57:22,746 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) STATS: Can alloc MEM from host: 2, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:57:22,795 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Network id=200 is already implemented
2021-10-07 08:57:22,863 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Network id=202 is already implemented
2021-10-07 08:57:23,432 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Reserved NIC for s-1539-VM [ipv4:169.254.241.63 netmask:255.255.0.0 gateway:169.254.0.1]
2021-10-07 08:57:23,477 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Network id=201 is already implemented
2021-10-07 08:57:23,527 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Releasing ip address for instance=5900
2021-10-07 08:57:23,567 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Allocated a nic NicProfile[5900-1539-08316455-6db7-4d2c-bf6d-97762afb240b-192.168.204.157-null] for VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:57:23,610 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Network id=203 is already implemented
2021-10-07 08:57:23,843 DEBUG [c.c.n.g.StorageNetworkGuru] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Allocated a storage nic NicProfile[5901-1539-08316455-6db7-4d2c-bf6d-97762afb240b-192.168.204.196-null] for VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:57:23,907 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) template 3 is already in store:5, type:Image
2021-10-07 08:57:23,929 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) template 3 is already in store:2, type:Primary
2021-10-07 08:57:23,931 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 514
2021-10-07 08:57:23,933 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Acquire lock on VMTemplateStoragePool 514 with timeout 3600 seconds
2021-10-07 08:57:25,143 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-f102b5c1) (logid:dce0b68d) AutoScaling Monitor is running...
2021-10-07 08:57:35,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5e20fd19) (logid:7f558c94) Found 0 routers to update status.
2021-10-07 08:57:35,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5e20fd19) (logid:7f558c94) Found 0 VPC's to update Redundant State.
2021-10-07 08:57:35,358 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5e20fd19) (logid:7f558c94) Found 0 networks to update RvR status.
2021-10-07 08:57:38,773 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-9f23dca2) (logid:3dc1bc1a) HostStatsCollector is running...
2021-10-07 08:57:38,804 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-9f23dca2) (logid:3dc1bc1a) Seq 1-3253569255798472735: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:57:38,874 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-9f23dca2) (logid:3dc1bc1a) Seq 2-4126986108531640366: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:57:57,946 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 08:57:57,947 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 08:57:57,947 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 08:57:57,956 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Run missing VM report. current time: 1633611477956
2021-10-07 08:57:57,962 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1539
2021-10-07 08:57:57,962 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Detected missing VM. host: 2, vm id: 1539(478e8eb9-9621-40e4-8a38-f3f6d42c372e), power state: PowerReportMissing, last state update: 2021/10/07T08:57:22.000Z
2021-10-07 08:57:57,962 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) vm id: 1539 - time since last state update(35956ms) has not passed graceful period yet
2021-10-07 08:57:57,962 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-4:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 08:58:04,607 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:ctx-31e4283e) (logid:77c13a56) Backup backup.poll is being called at 2021-10-07 12:58:04 GMT
2021-10-07 08:58:05,123 INFO [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:ctx-25f88f10) (logid:b5a2f139) Started resource counters recalculation periodic task.
2021-10-07 08:58:05,126 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-fcba2abe) (logid:f8264f22) External devices are not used. Skipping external device usage collection
2021-10-07 08:58:05,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-0cd631f6) (logid:8c7301df) Found 0 running routers.
2021-10-07 08:58:05,354 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-475e749c) (logid:bf9694dc) Found 0 routers to update status.
2021-10-07 08:58:05,359 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-475e749c) (logid:bf9694dc) Found 0 VPC's to update Redundant State.
2021-10-07 08:58:05,363 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-475e749c) (logid:bf9694dc) Found 0 networks to update RvR status.
2021-10-07 08:58:05,645 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-699449c0) (logid:4ea00e34) Snapshot scheduler.poll is being called at 2021-10-07 12:58:05 GMT
2021-10-07 08:58:05,652 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-699449c0) (logid:4ea00e34) Got 0 snapshots to be executed at 2021-10-07 12:58:05 GMT
2021-10-07 08:58:14,176 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 08:58:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 08:58:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 08:58:14,184 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Run missing VM report. current time: 1633611494184
2021-10-07 08:58:14,190 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1538
2021-10-07 08:58:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Detected missing VM. host: 1, vm id: 1538(b574a6bf-240d-4cac-8701-369761544bf6), power state: PowerReportMissing, last state update: 2021/10/07T08:57:20.000Z
2021-10-07 08:58:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) vm id: 1538 - time since last state update(54184ms) has not passed graceful period yet
2021-10-07 08:58:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-15:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 08:58:16,288 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-8ca6d8e7) (logid:b5d1f752) StorageCollector is running...
2021-10-07 08:58:16,306 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-8ca6d8e7) (logid:b5d1f752) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 08:58:16,310 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:ctx-8ca6d8e7) (logid:b5d1f752) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 08:58:16,310 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:ctx-8ca6d8e7) (logid:b5d1f752) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 08:58:16,591 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-8ca6d8e7) (logid:b5d1f752) Seq 1-3253569255798472736: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 08:58:25,148 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-030462f8) (logid:4c28224f) AutoScaling Monitor is running...
2021-10-07 08:58:34,547 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Running Capacity Checker ...
2021-10-07 08:58:34,548 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) recalculating system capacity
2021-10-07 08:58:34,548 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Executing cpu/ram capacity update
2021-10-07 08:58:34,561 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Found 1 VMs on host 1
2021-10-07 08:58:34,565 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Found 0 VMs are Migrating from host 1
2021-10-07 08:58:34,580 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Found 0 VM, not running on host 1
2021-10-07 08:58:34,594 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) No need to calibrate cpu capacity, host:1 usedCpuCore: 1 reservedCpuCore: 0
2021-10-07 08:58:34,602 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) No need to calibrate cpu capacity, host:1 usedCpu: 500 reservedCpu: 0
2021-10-07 08:58:34,602 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) No need to calibrate memory capacity, host:1 usedMem: (1.00 GB) 1073741824 reservedMem: (0 bytes) 0
2021-10-07 08:58:34,669 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Found 1 VMs on host 2
2021-10-07 08:58:34,675 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Found 0 VMs are Migrating from host 2
2021-10-07 08:58:34,694 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Found 0 VM, not running on host 2
2021-10-07 08:58:34,712 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) No need to calibrate cpu capacity, host:2 usedCpuCore: 1 reservedCpuCore: 0
2021-10-07 08:58:34,713 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) No need to calibrate cpu capacity, host:2 usedCpu: 500 reservedCpu: 0
2021-10-07 08:58:34,713 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) No need to calibrate memory capacity, host:2 usedMem: (512.00 MB) 536870912 reservedMem: (0 bytes) 0
2021-10-07 08:58:34,756 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Done executing cpu/ram capacity update
2021-10-07 08:58:34,756 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Executing storage capacity update
2021-10-07 08:58:34,798 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Found storage pool CS_Ceph of type RBD with overprovisioning factor 2
2021-10-07 08:58:34,798 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Total over provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 08:58:34,798 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Total over provisioned capacity of the pool CS_Ceph id: 2 is (54.5805 TB) 60011908694016
2021-10-07 08:58:34,809 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Successfully set Capacity - (54.5805 TB) 60011908694016 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 2, PodId 1
2021-10-07 08:58:34,809 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Done executing storage capacity update
2021-10-07 08:58:34,809 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Executing capacity updates for public ip and Vlans
2021-10-07 08:58:34,855 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Done capacity updates for public ip and Vlans
2021-10-07 08:58:34,855 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Executing capacity updates for private ip
2021-10-07 08:58:34,876 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Done executing capacity updates for private ip
2021-10-07 08:58:34,876 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Done recalculating system capacity
2021-10-07 08:58:34,960 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-49d10b7a) (logid:767e57cc) Done running Capacity Checker ...
2021-10-07 08:58:35,351 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-c068292a) (logid:3bbc01bd) Found 0 routers to update status.
2021-10-07 08:58:35,354 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-c068292a) (logid:3bbc01bd) Found 0 VPC's to update Redundant State.
2021-10-07 08:58:35,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-c068292a) (logid:3bbc01bd) Found 0 networks to update RvR status.
2021-10-07 08:58:35,488 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-98c4569c) (logid:5e355c34) Checking if any host reservation can be released ...
2021-10-07 08:58:35,500 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-98c4569c) (logid:5e355c34) Cannot release reservation, Found 1 VMs Running on host 1
2021-10-07 08:58:35,509 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-98c4569c) (logid:5e355c34) Cannot release reservation, Found 1 VMs Running on host 2
2021-10-07 08:58:35,509 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-98c4569c) (logid:5e355c34) Done running HostReservationReleaseChecker ...
2021-10-07 08:58:38,881 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-d9b76bad) (logid:d99f34ef) HostStatsCollector is running...
2021-10-07 08:58:38,934 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-d9b76bad) (logid:d99f34ef) Seq 1-3253569255798472737: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:58:38,969 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-d9b76bad) (logid:d99f34ef) Seq 2-4126986108531640367: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:58:39,726 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5dd44379) (logid:a85c18e8) Resetting hosts suitable for reconnect
2021-10-07 08:58:39,730 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5dd44379) (logid:a85c18e8) Completed resetting hosts suitable for reconnect
2021-10-07 08:58:39,730 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5dd44379) (logid:a85c18e8) Acquiring hosts for clusters already owned by this management server
2021-10-07 08:58:39,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5dd44379) (logid:a85c18e8) Completed acquiring hosts for clusters already owned by this management server
2021-10-07 08:58:39,732 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5dd44379) (logid:a85c18e8) Acquiring hosts for clusters not owned by any management server
2021-10-07 08:58:39,733 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5dd44379) (logid:a85c18e8) Completed acquiring hosts for clusters not owned by any management server
2021-10-07 08:58:57,942 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 08:58:57,943 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 08:58:57,943 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 08:58:57,951 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) Run missing VM report. current time: 1633611537951
2021-10-07 08:58:57,958 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1539
2021-10-07 08:58:57,959 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) Detected missing VM. host: 2, vm id: 1539(478e8eb9-9621-40e4-8a38-f3f6d42c372e), power state: PowerReportMissing, last state update: 2021/10/07T08:57:22.000Z
2021-10-07 08:58:57,959 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) vm id: 1539 - time since last state update(95951ms) has not passed graceful period yet
2021-10-07 08:58:57,959 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-1:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 08:59:05,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-73168344) (logid:81aca751) Found 0 routers to update status.
2021-10-07 08:59:05,355 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-73168344) (logid:81aca751) Found 0 VPC's to update Redundant State.
2021-10-07 08:59:05,358 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-73168344) (logid:81aca751) Found 0 networks to update RvR status.
2021-10-07 08:59:14,176 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 08:59:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 08:59:14,177 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 08:59:14,182 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:null) (logid:) Run missing VM report. current time: 1633611554182
2021-10-07 08:59:14,189 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1538
2021-10-07 08:59:14,189 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:null) (logid:) Detected missing VM. host: 1, vm id: 1538(b574a6bf-240d-4cac-8701-369761544bf6), power state: PowerReportMissing, last state update: 2021/10/07T08:57:20.000Z
2021-10-07 08:59:14,189 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:null) (logid:) vm id: 1538 - time since last state update(114182ms) has not passed graceful period yet
2021-10-07 08:59:14,189 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-6:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 08:59:16,592 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-324a2113) (logid:94d82bdc) StorageCollector is running...
2021-10-07 08:59:16,615 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-324a2113) (logid:94d82bdc) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 08:59:16,621 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:ctx-324a2113) (logid:94d82bdc) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 08:59:16,621 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:ctx-324a2113) (logid:94d82bdc) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 08:59:16,996 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-324a2113) (logid:94d82bdc) Seq 1-3253569255798472738: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 08:59:25,153 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-b98f2498) (logid:be2d357f) AutoScaling Monitor is running...
2021-10-07 08:59:29,172 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 1-3253569255798472734: Processing: { Ans: , MgmtId: 90520741394843, via: 1, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:59:29,173 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 1-3253569255798472734: No more commands found
2021-10-07 08:59:29,173 DEBUG [c.c.a.t.Request] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Seq 1-3253569255798472734: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2021-10-07 08:59:29,270 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) releasing lock for VMTemplateStoragePool 514
2021-10-07 08:59:29,329 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Unable to create Vol[1538|vm=1538|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
2021-10-07 08:59:29,330 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[1538|vm=1538|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1491)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1566)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1160)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:59:29,570 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Cleaning up resources for the vm VM[ConsoleProxy|v-1538-VM] in Starting state
2021-10-07 08:59:29,589 DEBUG [c.c.a.t.Request] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Seq 1-3253569255798472739: Sending { Cmd , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.26.151","forceStop":"false","volumesToDisconnect":[],"vmName":"v-1538-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:59:32,857 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) (logid:) Seq 1-3253569255798472739: Processing: { Ans: , MgmtId: 90520741394843, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:59:32,857 DEBUG [c.c.a.t.Request] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Seq 1-3253569255798472739: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { StopAnswer } }
2021-10-07 08:59:32,947 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Released nic: NicProfile[5896-1538-null-null-null]
2021-10-07 08:59:33,029 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Releasing ip address for reservationId=6f5e31ff-d834-4e4c-8260-8324b243ff39, instance=5897
2021-10-07 08:59:33,030 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Released nic: NicProfile[5897-1538-null-null-null]
2021-10-07 08:59:33,074 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Successfully released network resources for the vm VM[ConsoleProxy|v-1538-VM]
2021-10-07 08:59:33,075 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Successfully cleaned up resources for the VM VM[ConsoleProxy|v-1538-VM] in Starting state
2021-10-07 08:59:33,082 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) DeploymentPlanner allocation algorithm: null
2021-10-07 08:59:33,083 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB) 1073741824
2021-10-07 08:59:33,083 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 08:59:33,085 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: [1]
2021-10-07 08:59:33,091 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Searching all possible resources under this Zone: 1
2021-10-07 08:59:33,100 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 08:59:33,141 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 08:59:33,146 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 08:59:33,146 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 08:59:33,153 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) FirstFitAllocator has 2 hosts to check for allocation: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:59:33,172 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Found 2 hosts for allocation after prioritization: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:59:33,172 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for speed=500Mhz, Ram=1024 MB
2021-10-07 08:59:33,172 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host name: gd-cs01-clst01-compute01, hostId: 1 is in avoid set, skipping this and trying other available hosts
2021-10-07 08:59:33,194 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:59:33,194 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:33,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:33,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73100 , Requested CPU: 500
2021-10-07 08:59:33,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.40 GB) 65932578816 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:59:33,206 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:59:33,206 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:59:33,206 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 2, used: (512.00 MB) 536870912, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:59:33,206 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 2
2021-10-07 08:59:33,206 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545 FirstFitRoutingAllocator) (logid:bcef1bbe) Host Allocator returning 1 suitable hosts
2021-10-07 08:59:33,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking suitable pools for volume (Id, Type): (1538,ROOT)
2021-10-07 08:59:33,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) We need to allocate new storagepool for this volume
2021-10-07 08:59:33,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Calling StoragePoolAllocators to find suitable pools
2021-10-07 08:59:33,222 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) System VMs will use shared storage for zone id=1
2021-10-07 08:59:33,222 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 08:59:33,222 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 08:59:33,222 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 08:59:33,225 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 08:59:33,230 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 08:59:33,230 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) StoragePool is in avoid set, skipping this pool
2021-10-07 08:59:33,230 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator returning 0 suitable storage pools
2021-10-07 08:59:33,230 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) ZoneWideStoragePoolAllocator to find storage pool
2021-10-07 08:59:33,235 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) No suitable pools found for volume: Vol[1538|vm=1538|ROOT] under cluster: 1
2021-10-07 08:59:33,235 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) No suitable pools found
2021-10-07 08:59:33,235 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) No suitable storagePools found under this Cluster: 1
2021-10-07 08:59:33,242 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2021-10-07 08:59:33,244 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Searching all possible resources under this Zone: 1
2021-10-07 08:59:33,247 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 08:59:33,252 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Removing from the clusterId list these clusters from avoid set: [1]
2021-10-07 08:59:33,255 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) No clusters found after removing disabled clusters and clusters in avoid list, returning.
2021-10-07 08:59:33,374 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 1
2021-10-07 08:59:33,405 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:33,405 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Hosts's actual total RAM: (61.89 GB) 66452602880 and RAM after applying overprovisioning: (61.89 GB) 66452602880
2021-10-07 08:59:33,405 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) release cpu from host: 1, old used: 500,reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:59:33,405 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) release mem from host: 1, old used: (1.00 GB) 1073741824,reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; new used: (0 bytes) 0,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:59:33,454 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 08:59:33,455 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735 ctx-8046e545) (logid:bcef1bbe) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 08:59:33,455 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 1538, job origin: 3439
2021-10-07 08:59:33,455 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Unable to complete AsyncJobVO {id:4735, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:57:18 EDT 2021, removed: null}, job origin:3439
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:59:33,461 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Complete async job-4735, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyADdjb20uY2xvdWQuZXhjZXB0aW9uLkluc3VmZmljaWVudFNlcnZlckNhcGFjaXR5RXhjZXB0aW9uAAAAAFZNT3ACAAFaABVhZmZpbml0eUdyb3Vwc0FwcGxpZWR4cgAxY29tLmNsb3VkLmV4Y2VwdGlvbi5JbnN1ZmZpY2llbnRDYXBhY2l0eUV4Y2VwdGlvbgAAAABWTU91AgACTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc2NvcGV0ABFMamF2YS9sYW5nL0NsYXNzO3hyACJjb20uY2xvdWQuZXhjZXB0aW9uLkNsb3VkRXhjZXB0aW9ueeiOG_HkRMcCAAJMAAtjc0Vycm9yQ29kZXQAE0xqYXZhL2xhbmcvSW50ZWdlcjtMAAZpZExpc3R0ABVMamF2YS91dGlsL0FycmF5TGlzdDt4cgATamF2YS5sYW5nLkV4Y2VwdGlvbtD9Hz4aOxzEAgAAeHIAE2phdmEubGFuZy5UaHJvd2FibGXVxjUnOXe4ywMABEwABWNhdXNldAAVTGphdmEvbGFuZy9UaHJvd2FibGU7TAANZGV0YWlsTWVzc2FnZXQAEkxqYXZhL2xhbmcvU3RyaW5nO1sACnN0YWNrVHJhY2V0AB5bTGphdmEvbGFuZy9TdGFja1RyYWNlRWxlbWVudDtMABRzdXBwcmVzc2VkRXhjZXB0aW9uc3QAEExqYXZhL3V0aWwvTGlzdDt4cHEAfgANdAA8VW5hYmxlIHRvIGNyZWF0ZSBhIGRlcGxveW1lbnQgZm9yIFZNW0NvbnNvbGVQcm94eXx2LTE1MzgtVk1ddXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAUc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAIQgAGZm9ybWF0SQAKbGluZU51bWJlckwAD2NsYXNzTG9hZGVyTmFtZXEAfgAKTAAOZGVjbGFyaW5nQ2xhc3NxAH4ACkwACGZpbGVOYW1lcQB-AApMAAptZXRob2ROYW1lcQB-AApMAAptb2R1bGVOYW1lcQB-AApMAA1tb2R1bGVWZXJzaW9ucQB-AAp4cAEAAARfdAADYXBwdAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0ABBvcmNoZXN0cmF0ZVN0YXJ0cHBzcQB-ABEBAAAVfnEAfgATcQB-ABRxAH4AFXEAfgAWcHBzcQB-ABEA_____3B0AC9qZGsuaW50ZXJuYWwucmVmbGVjdC5HZW5lcmF0ZWRNZXRob2RBY2Nlc3NvcjIyNXB0AAZpbnZva2VwcHNxAH4AEQIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AGnQACWphdmEuYmFzZXQABzExLjAuMTFzcQB-ABECAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAacQB-AB5xAH4AH3NxAH4AEQEAAABrcQB-ABN0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4AEQEAABYlcQB-ABNxAH4AFHEAfgAVcQB-ACZwcHNxAH4AEQEAAABmcQB-ABN0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnBwc3EAfgARAQAAAmxxAH4AE3QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgARAQAAADBxAH4AE3QAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-ABEBAAAAN3EAfgATdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-ABEBAAAAZnEAfgATdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA2dAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-ABEBAAAANHEAfgATcQB-ADlxAH4ANnQADnJ1bldpdGhDb250ZXh0cHBzcQB-ABEBAAAALXEAfgATdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ADJxAH4AM3Bwc3EAfgARAQAAAjhxAH4AE3EAfgAtcQB-AC5xAH4AM3Bwc3EAfgARAgAAAgNwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ADdxAH4AHnEAfgAfc3EAfgARAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAzcQB-AB5xAH4AH3NxAH4AEQIAAARocHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-AB5xAH4AH3NxAH4AEQIAAAJ0cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ASHEAfgAzcQB-AB5xAH4AH3NxAH4AEQIAAAM9cHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAzcQB-AB5xAH4AH3NyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHhzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAQ73NyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cQB-AFIAAAAAAAAAAXZyABdjb20uY2xvdWQuZGMuRGF0YUNlbnRlclPoJBrS07ZcAgAAeHAA
2021-10-07 08:59:33,465 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Publish async job-4735 complete on message bus
2021-10-07 08:59:33,465 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Wake up jobs related to job-4735
2021-10-07 08:59:33,465 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Update db status for job-4735
2021-10-07 08:59:33,470 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Wake up jobs joined with job-4735 and disjoin all subjobs created from job- 4735
2021-10-07 08:59:33,531 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Done executing com.cloud.vm.VmWorkStart for job-4735
2021-10-07 08:59:33,535 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-133:ctx-f32d303e job-3439/job-4735) (logid:bcef1bbe) Remove job-4735 from job monitoring
2021-10-07 08:59:33,587 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Exception while trying to start console proxy
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:59:33,588 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Unable to start console proxy vm for standby capacity, vm id : 1538, will recycle it and start a new one
2021-10-07 08:59:33,588 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) received console proxy alert
2021-10-07 08:59:33,592 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) Console proxy creation failure, zone: gd-cs01
2021-10-07 08:59:33,594 WARN [c.c.a.AlertManagerImpl] (consoleproxy-1:ctx-935abca1) (logid:b4bd78eb) AlertType:: 10 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Console proxy creation failure. zone: gd-cs01, error details: null
2021-10-07 08:59:33,706 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Zone 1 is ready to launch console proxy
2021-10-07 08:59:33,706 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Expand console proxy standby capacity for zone gd-cs01
2021-10-07 08:59:33,710 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Found a stopped console proxy, starting it. Vm id : 1538
2021-10-07 08:59:33,766 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Sync job-4737 execution on object VmWorkJobQueue.1538
2021-10-07 08:59:34,247 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Unable to acquire lock on VMTemplateStoragePool 514
2021-10-07 08:59:34,441 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Cleaning up resources for the vm VM[SecondaryStorageVm|s-1539-VM] in Starting state
2021-10-07 08:59:34,456 DEBUG [c.c.a.t.Request] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Seq 2-4126986108531640368: Sending { Cmd , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.241.63","forceStop":"false","volumesToDisconnect":[],"vmName":"s-1539-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:59:34,473 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-61919b0d) (logid:28b78bbb) Execute sync-queue item: SyncQueueItemVO {id:4701, queueId: 4699, contentType: AsyncJob, contentId: 4737, lastProcessMsid: 90520741394843, lastprocessNumber: 2, lastProcessTime: Thu Oct 07 08:59:34 EDT 2021, created: Thu Oct 07 08:59:33 EDT 2021}
2021-10-07 08:59:34,476 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-61919b0d) (logid:28b78bbb) Schedule queued job-4737
2021-10-07 08:59:34,520 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:4f77b3a9) Add job-4737 into job monitoring
2021-10-07 08:59:34,532 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Executing AsyncJobVO {id:4737, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:59:33 EDT 2021, removed: null}
2021-10-07 08:59:34,532 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Run VM work job: com.cloud.vm.VmWorkStart for VM 1538, job origin: 3439
2021-10-07 08:59:34,536 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Execute VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1538,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:59:34,555 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2021-10-07 08:59:34,557 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Successfully transitioned to start state for VM[ConsoleProxy|v-1538-VM] reservation id = 4ee9913f-7ce8-4f4c-8e9d-2bb03c903e91
2021-10-07 08:59:34,598 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:59:34,604 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) DeploymentPlanner allocation algorithm: null
2021-10-07 08:59:34,604 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: (1.00 GB) 1073741824
2021-10-07 08:59:34,604 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 08:59:34,606 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:59:34,611 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Searching resources only under specified Pod: 1
2021-10-07 08:59:34,615 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Pod: 1
2021-10-07 08:59:34,645 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 08:59:34,650 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 08:59:34,650 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 08:59:34,659 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 08:59:34,702 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 08:59:34,702 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for speed=500Mhz, Ram=1024 MB
2021-10-07 08:59:34,733 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:59:34,733 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:34,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:34,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73100 , Requested CPU: 500
2021-10-07 08:59:34,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.40 GB) 65932578816 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:59:34,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:59:34,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:59:34,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 2, used: (512.00 MB) 536870912, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:59:34,742 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 2
2021-10-07 08:59:34,782 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:59:34,782 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:34,794 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:34,794 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:59:34,794 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:59:34,794 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:59:34,794 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:59:34,794 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:59:34,794 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 1
2021-10-07 08:59:34,794 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host Allocator returning 2 suitable hosts
2021-10-07 08:59:34,805 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking suitable pools for volume (Id, Type): (1538,ROOT)
2021-10-07 08:59:34,805 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) We need to allocate new storagepool for this volume
2021-10-07 08:59:34,809 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Calling StoragePoolAllocators to find suitable pools
2021-10-07 08:59:34,825 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) System VMs will use shared storage for zone id=1
2021-10-07 08:59:34,825 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 08:59:34,826 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 08:59:34,826 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 08:59:34,830 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 08:59:34,834 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 08:59:34,837 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Storage pool null (2) does not supply IOPS capacity, assuming enough capacity
2021-10-07 08:59:34,842 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking pool 2 for storage, totalSize: 30005954347008, usedBytes: 11020345267, usedPct: 3.6727194674609234E-4, disable threshold: 0.85
2021-10-07 08:59:34,842 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Destination pool id: 2
2021-10-07 08:59:34,890 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Pool ID for the volume with ID 1538 is null
2021-10-07 08:59:34,910 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Found storage pool CS_Ceph of type RBD with over-provisioning factor 2
2021-10-07 08:59:34,910 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Total over-provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 08:59:34,910 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Total capacity of the pool CS_Ceph with ID 2 is (54.5805 TB) 60011908694016
2021-10-07 08:59:34,913 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking pool: 2 for storage allocation , maxSize : (54.5805 TB) 60011908694016, totalAllocatedSize : (0 bytes) 0, askingSize : (0 bytes) 0, allocated disable threshold: 0.85
2021-10-07 08:59:34,913 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
2021-10-07 08:59:34,913 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
2021-10-07 08:59:34,915 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking if host: 2 can access any suitable storage pool for volume: ROOT
2021-10-07 08:59:34,918 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Host: 2 can access pool: 2
2021-10-07 08:59:34,920 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Found a potential host id: 2 name: gd-cs01-clst01-compute02 and associated storage pools for this VM
2021-10-07 08:59:34,922 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1538|ROOT-->Pool(2))]
2021-10-07 08:59:34,922 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Deployment found - P0=VM[ConsoleProxy|v-1538-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1538|ROOT-->Pool(2))]
2021-10-07 08:59:35,083 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 2 host id before state transition: null
2021-10-07 08:59:35,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:35,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) We are allocating VM, increasing the used capacity of this host:2
2021-10-07 08:59:35,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Current Used CPU: 500 , Free CPU:73100 ,Requested CPU: 500
2021-10-07 08:59:35,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Current Used RAM: (512.00 MB) 536870912 , Free RAM:(61.40 GB) 65932578816 ,Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:59:35,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) CPU STATS after allocation: for host: 2, old used: 500, old reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used:1000, reserved:0; requested cpu:500,alloc_from_last:false
2021-10-07 08:59:35,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) RAM STATS after allocation: for host: 2, old used: (512.00 MB) 536870912, old reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; new used: (1.50 GB) 1610612736, reserved: (0 bytes) 0; requested mem: (1.00 GB) 1073741824,alloc_from_last:false
2021-10-07 08:59:35,123 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:59:35,123 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:35,128 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:35,128 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) We need to allocate to the last host again, so checking if there is enough reserved capacity
2021-10-07 08:59:35,128 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Reserved CPU: 0 , Requested CPU: 500
2021-10-07 08:59:35,128 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Reserved RAM: (0 bytes) 0 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:59:35,128 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) STATS: Failed to alloc resource from host: 2 reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (1.00 GB) 1073741824
2021-10-07 08:59:35,128 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Host does not have enough reserved CPU available, cannot allocate to this host.
2021-10-07 08:59:35,128 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:35,132 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:35,133 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Free CPU: 73100 , Requested CPU: 500
2021-10-07 08:59:35,133 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Free RAM: (61.40 GB) 65932578816 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 08:59:35,133 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 08:59:35,133 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:59:35,133 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) STATS: Can alloc MEM from host: 2, used: (512.00 MB) 536870912, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:59:35,201 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Network id=200 is already implemented
2021-10-07 08:59:35,296 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Network id=202 is already implemented
2021-10-07 08:59:35,352 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8f80736c) (logid:d486bfff) Found 0 routers to update status.
2021-10-07 08:59:35,354 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8f80736c) (logid:d486bfff) Found 0 VPC's to update Redundant State.
2021-10-07 08:59:35,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8f80736c) (logid:d486bfff) Found 0 networks to update RvR status.
2021-10-07 08:59:35,860 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Reserved NIC for v-1538-VM [ipv4:169.254.163.36 netmask:255.255.0.0 gateway:169.254.0.1]
2021-10-07 08:59:35,926 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Network id=201 is already implemented
2021-10-07 08:59:35,996 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Releasing ip address for instance=5897
2021-10-07 08:59:36,062 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Allocated a nic NicProfile[5897-1538-4ee9913f-7ce8-4f4c-8e9d-2bb03c903e91-192.168.204.150-null] for VM[ConsoleProxy|v-1538-VM]
2021-10-07 08:59:36,215 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) template 3 is already in store:5, type:Image
2021-10-07 08:59:36,233 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Not found (templateId:3poolId:2conf:null) in template_spool_ref, persisting it
2021-10-07 08:59:36,330 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) template 3 is already in store:2, type:Primary
2021-10-07 08:59:36,334 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 515
2021-10-07 08:59:36,340 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Acquire lock on VMTemplateStoragePool 515 with timeout 3600 seconds
2021-10-07 08:59:36,374 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) lock is acquired for VMTemplateStoragePool 515
2021-10-07 08:59:36,440 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2021-10-07 08:59:36,500 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand
2021-10-07 08:59:36,511 DEBUG [c.c.a.t.Request] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Seq 2-4126986108531640369: Sending { Cmd , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.204.62/cloudstack","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6db2511f-6992-3120-b49e-fafdfa8ec257","id":"2","poolType":"RBD","host":"192.168.206.63","path":"cloudstack","port":"0","url":"RBD://192.168.206.63/cloudstack/?ROLE=Primary&STOREUUID=6db2511f-6992-3120-b49e-fafdfa8ec257","isManaged":"false"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"10800","bypassHostMaintenance":"false"}}] }
2021-10-07 08:59:37,729 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 2-4126986108531640368: Processing: { Ans: , MgmtId: 90520741394843, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 08:59:37,729 DEBUG [c.c.a.t.Request] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Seq 2-4126986108531640368: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { StopAnswer } }
2021-10-07 08:59:38,976 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-f8d2fab9) (logid:7acbe3f6) HostStatsCollector is running...
2021-10-07 08:59:39,005 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-f8d2fab9) (logid:7acbe3f6) Seq 1-3253569255798472740: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:59:39,028 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-f8d2fab9) (logid:7acbe3f6) Seq 2-4126986108531640370: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 08:59:39,142 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Released nic: NicProfile[5899-1539-null-null-null]
2021-10-07 08:59:39,296 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Releasing ip address for reservationId=08316455-6db7-4d2c-bf6d-97762afb240b, instance=5900
2021-10-07 08:59:39,298 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Released nic: NicProfile[5900-1539-null-null-null]
2021-10-07 08:59:39,356 DEBUG [c.c.n.g.StorageNetworkGuru] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Release an storage ip 192.168.204.196
2021-10-07 08:59:39,399 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Successfully released network resources for the vm VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:59:39,399 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Successfully cleaned up resources for the VM VM[SecondaryStorageVm|s-1539-VM] in Starting state
2021-10-07 08:59:39,480 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2
2021-10-07 08:59:39,502 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:39,502 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Hosts's actual total RAM: (61.90 GB) 66469449728 and RAM after applying overprovisioning: (61.90 GB) 66469449728
2021-10-07 08:59:39,502 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) release cpu from host: 2, old used: 1000,reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used: 500,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:59:39,502 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) release mem from host: 2, old used: (1.50 GB) 1610612736,reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; new used: (1.00 GB) 1073741824,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 08:59:39,546 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Caught CloudRuntimeException, returning job failed com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on VMTemplateStoragePool: 514
2021-10-07 08:59:39,548 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Done executing VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1539,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:59:39,552 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Complete async job-4736, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ABtVbmFibGUgdG8gc3RhcnQgVk0gaW5zdGFuY2V1cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABNzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAAhCAAZmb3JtYXRJAApsaW5lTnVtYmVyTAAPY2xhc3NMb2FkZXJOYW1lcQB-AAVMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABUwACm1vZHVsZU5hbWVxAH4ABUwADW1vZHVsZVZlcnNpb25xAH4ABXhwAQAAFYN0AANhcHB0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAEG9yY2hlc3RyYXRlU3RhcnRwcHNxAH4ADAD_____cHQAL2pkay5pbnRlcm5hbC5yZWZsZWN0LkdlbmVyYXRlZE1ldGhvZEFjY2Vzc29yMjI1cHQABmludm9rZXBwc3EAfgAMAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAUdAAJamF2YS5iYXNldAAHMTEuMC4xMXNxAH4ADAIAAAI2cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ABRxAH4AGHEAfgAZc3EAfgAMAQAAAGtxAH4ADnQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgAMAQAAFiVxAH4ADnEAfgAPcQB-ABBxAH4AIHBwc3EAfgAMAQAAAGZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACbHEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMHEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA3cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABmcQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADB0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA0cQB-AA5xAH4AM3EAfgAwdAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAtcQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ALHEAfgAtcHBzcQB-AAwBAAACOHEAfgAOcQB-ACdxAH4AKHEAfgAtcHBzcQB-AAwCAAACA3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AMXEAfgAYcQB-ABlzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AGHEAfgAZc3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBCcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAAAz1wdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AC1xAH4AGHEAfgAZc3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2021-10-07 08:59:39,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Publish async job-4736 complete on message bus
2021-10-07 08:59:39,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Wake up jobs related to job-4736
2021-10-07 08:59:39,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Update db status for job-4736
2021-10-07 08:59:39,562 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736 ctx-f75b1bbc) (logid:e381c0f1) Wake up jobs joined with job-4736 and disjoin all subjobs created from job- 4736
2021-10-07 08:59:39,751 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736) (logid:e381c0f1) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 1539, job origin: 3440
2021-10-07 08:59:39,751 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736) (logid:e381c0f1) Done executing com.cloud.vm.VmWorkStart for job-4736
2021-10-07 08:59:39,756 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-135:ctx-daf2a929 job-3440/job-4736) (logid:e381c0f1) Remove job-4736 from job monitoring
2021-10-07 08:59:39,808 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Exception while trying to start secondary storage vm
com.cloud.utils.exception.CloudRuntimeException: Unable to start VM instance
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5507)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 08:59:39,813 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Unable to start secondary storage vm for standby capacity, vm id : 1539, will recycle it and start a new one
2021-10-07 08:59:39,813 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-995d956e) (logid:06956492) received secondary storage vm alert
2021-10-07 08:59:39,815 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-995d956e) (logid:06956492) Secondary Storage Vm creation failure, zone: gd-cs01
2021-10-07 08:59:39,818 WARN [c.c.a.AlertManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: gd-cs01, error details: null
2021-10-07 08:59:39,858 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-995d956e) (logid:06956492) Primary secondary storage is not even started, wait until next turn
2021-10-07 08:59:39,902 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Zone 1 is ready to launch secondary storage VM
2021-10-07 08:59:39,910 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) No running secondary storage vms found in datacenter id=1, starting one
2021-10-07 08:59:39,922 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Found a stopped secondary storage vm, starting it. Vm id : 1539
2021-10-07 08:59:39,978 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Sync job-4738 execution on object VmWorkJobQueue.1539
2021-10-07 08:59:40,334 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-1c8fc570) (logid:21128743) Execute sync-queue item: SyncQueueItemVO {id:4702, queueId: 4700, contentType: AsyncJob, contentId: 4738, lastProcessMsid: 90520741394843, lastprocessNumber: 2, lastProcessTime: Thu Oct 07 08:59:40 EDT 2021, created: Thu Oct 07 08:59:39 EDT 2021}
2021-10-07 08:59:40,338 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-1c8fc570) (logid:21128743) Schedule queued job-4738
2021-10-07 08:59:40,388 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738) (logid:dc24fd0d) Add job-4738 into job monitoring
2021-10-07 08:59:40,400 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738) (logid:e381c0f1) Executing AsyncJobVO {id:4738, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGA3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:59:39 EDT 2021, removed: null}
2021-10-07 08:59:40,401 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738) (logid:e381c0f1) Run VM work job: com.cloud.vm.VmWorkStart for VM 1539, job origin: 3440
2021-10-07 08:59:40,412 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Execute VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1539,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 08:59:40,440 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2021-10-07 08:59:40,443 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Successfully transitioned to start state for VM[SecondaryStorageVm|s-1539-VM] reservation id = 5048be5c-eb7d-4f33-bee7-67485456483c
2021-10-07 08:59:40,502 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:59:40,520 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) DeploymentPlanner allocation algorithm: null
2021-10-07 08:59:40,520 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: (512.00 MB) 536870912
2021-10-07 08:59:40,520 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 08:59:40,527 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 08:59:40,541 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Searching resources only under specified Pod: 1
2021-10-07 08:59:40,553 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Pod: 1
2021-10-07 08:59:40,602 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 08:59:40,608 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 08:59:40,608 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 08:59:40,621 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) FirstFitAllocator has 2 hosts to check for allocation: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:59:40,649 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Found 2 hosts for allocation after prioritization: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 08:59:40,649 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Looking for speed=500Mhz, Ram=512 MB
2021-10-07 08:59:40,688 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:59:40,688 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:40,700 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:40,700 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:59:40,700 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:59:40,700 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:59:40,700 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:59:40,700 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:59:40,700 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 1
2021-10-07 08:59:40,728 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:59:40,729 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:40,735 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:40,735 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73100 , Requested CPU: 500
2021-10-07 08:59:40,735 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (60.90 GB) 65395707904 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:59:40,735 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:59:40,735 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:59:40,735 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 2, used: (1.00 GB) 1073741824, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:59:40,735 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 2
2021-10-07 08:59:40,735 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a FirstFitRoutingAllocator) (logid:e381c0f1) Host Allocator returning 2 suitable hosts
2021-10-07 08:59:40,745 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking suitable pools for volume (Id, Type): (1539,ROOT)
2021-10-07 08:59:40,745 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) We need to allocate new storagepool for this volume
2021-10-07 08:59:40,750 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Calling StoragePoolAllocators to find suitable pools
2021-10-07 08:59:40,759 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) System VMs will use shared storage for zone id=1
2021-10-07 08:59:40,760 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 08:59:40,760 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 08:59:40,760 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 08:59:40,763 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 08:59:40,770 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 08:59:40,776 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Storage pool null (2) does not supply IOPS capacity, assuming enough capacity
2021-10-07 08:59:40,780 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking pool 2 for storage, totalSize: 30005954347008, usedBytes: 11020345267, usedPct: 3.6727194674609234E-4, disable threshold: 0.85
2021-10-07 08:59:40,780 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Destination pool id: 2
2021-10-07 08:59:40,831 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Pool ID for the volume with ID 1539 is null
2021-10-07 08:59:40,848 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Found storage pool CS_Ceph of type RBD with over-provisioning factor 2
2021-10-07 08:59:40,848 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Total over-provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 08:59:40,848 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Total capacity of the pool CS_Ceph with ID 2 is (54.5805 TB) 60011908694016
2021-10-07 08:59:40,852 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking pool: 2 for storage allocation , maxSize : (54.5805 TB) 60011908694016, totalAllocatedSize : (0 bytes) 0, askingSize : (0 bytes) 0, allocated disable threshold: 0.85
2021-10-07 08:59:40,852 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
2021-10-07 08:59:40,852 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
2021-10-07 08:59:40,855 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking if host: 1 can access any suitable storage pool for volume: ROOT
2021-10-07 08:59:40,858 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Host: 1 can access pool: 2
2021-10-07 08:59:40,861 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Found a potential host id: 1 name: gd-cs01-clst01-compute01 and associated storage pools for this VM
2021-10-07 08:59:40,863 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1539|ROOT-->Pool(2))]
2021-10-07 08:59:40,863 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Deployment found - P0=VM[SecondaryStorageVm|s-1539-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage(Volume(1539|ROOT-->Pool(2))]
2021-10-07 08:59:40,926 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 1 host id before state transition: null
2021-10-07 08:59:40,975 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:40,975 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) We are allocating VM, increasing the used capacity of this host:1
2021-10-07 08:59:40,975 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Current Used CPU: 0 , Free CPU:73600 ,Requested CPU: 500
2021-10-07 08:59:40,975 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Current Used RAM: (0 bytes) 0 , Free RAM:(61.89 GB) 66452602880 ,Requested RAM: (512.00 MB) 536870912
2021-10-07 08:59:40,976 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) CPU STATS after allocation: for host: 1, old used: 0, old reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
2021-10-07 08:59:40,976 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) RAM STATS after allocation: for host: 1, old used: (0 bytes) 0, old reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; new used: (512.00 MB) 536870912, reserved: (0 bytes) 0; requested mem: (512.00 MB) 536870912,alloc_from_last:false
2021-10-07 08:59:40,985 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 08:59:40,986 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:40,992 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:40,992 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) We need to allocate to the last host again, so checking if there is enough reserved capacity
2021-10-07 08:59:40,992 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Reserved CPU: 0 , Requested CPU: 500
2021-10-07 08:59:40,992 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Reserved RAM: (0 bytes) 0 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:59:40,992 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) STATS: Failed to alloc resource from host: 1 reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (512.00 MB) 536870912
2021-10-07 08:59:40,992 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Host does not have enough reserved CPU available, cannot allocate to this host.
2021-10-07 08:59:40,993 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 08:59:40,999 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 08:59:40,999 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Free CPU: 73600 , Requested CPU: 500
2021-10-07 08:59:41,000 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (512.00 MB) 536870912
2021-10-07 08:59:41,000 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 08:59:41,000 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 08:59:41,000 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 08:59:41,071 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Network id=200 is already implemented
2021-10-07 08:59:41,171 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Network id=202 is already implemented
2021-10-07 08:59:42,393 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Reserved NIC for s-1539-VM [ipv4:169.254.28.97 netmask:255.255.0.0 gateway:169.254.0.1]
2021-10-07 08:59:43,020 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Network id=201 is already implemented
2021-10-07 08:59:43,084 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Releasing ip address for instance=5900
2021-10-07 08:59:43,181 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Allocated a nic NicProfile[5900-1539-5048be5c-eb7d-4f33-bee7-67485456483c-192.168.204.172-null] for VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:59:43,245 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Network id=203 is already implemented
2021-10-07 08:59:43,475 DEBUG [c.c.n.g.StorageNetworkGuru] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Allocated a storage nic NicProfile[5901-1539-5048be5c-eb7d-4f33-bee7-67485456483c-192.168.204.198-null] for VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 08:59:43,597 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) template 3 is already in store:5, type:Image
2021-10-07 08:59:43,648 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) template 3 is already in store:2, type:Primary
2021-10-07 08:59:43,655 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 515
2021-10-07 08:59:43,658 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Acquire lock on VMTemplateStoragePool 515 with timeout 3600 seconds
2021-10-07 08:59:57,946 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 08:59:57,946 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 08:59:57,947 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 08:59:57,953 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Run missing VM report. current time: 1633611597953
2021-10-07 08:59:57,959 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1538
2021-10-07 08:59:57,959 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Detected missing VM. host: 2, vm id: 1538(b574a6bf-240d-4cac-8701-369761544bf6), power state: PowerReportMissing, last state update: 2021/10/07T08:59:34.000Z
2021-10-07 08:59:57,959 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) vm id: 1538 - time since last state update(23953ms) has not passed graceful period yet
2021-10-07 08:59:57,959 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 09:00:05,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-1b6fb511) (logid:03364de7) Found 0 routers to update status.
2021-10-07 09:00:05,355 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-1b6fb511) (logid:03364de7) Found 0 VPC's to update Redundant State.
2021-10-07 09:00:05,357 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-1b6fb511) (logid:03364de7) Found 0 networks to update RvR status.
2021-10-07 09:00:09,726 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-daa3355d) (logid:aa7dc32c) Resetting hosts suitable for reconnect
2021-10-07 09:00:09,730 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-daa3355d) (logid:aa7dc32c) Completed resetting hosts suitable for reconnect
2021-10-07 09:00:09,730 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-daa3355d) (logid:aa7dc32c) Acquiring hosts for clusters already owned by this management server
2021-10-07 09:00:09,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-daa3355d) (logid:aa7dc32c) Completed acquiring hosts for clusters already owned by this management server
2021-10-07 09:00:09,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-daa3355d) (logid:aa7dc32c) Acquiring hosts for clusters not owned by any management server
2021-10-07 09:00:09,733 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-daa3355d) (logid:aa7dc32c) Completed acquiring hosts for clusters not owned by any management server
2021-10-07 09:00:14,186 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 09:00:14,187 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 09:00:14,187 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 09:00:14,194 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Run missing VM report. current time: 1633611614194
2021-10-07 09:00:14,199 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1539
2021-10-07 09:00:14,199 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Detected missing VM. host: 1, vm id: 1539(478e8eb9-9621-40e4-8a38-f3f6d42c372e), power state: PowerReportMissing, last state update: 2021/10/07T08:59:40.000Z
2021-10-07 09:00:14,199 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) vm id: 1539 - time since last state update(34194ms) has not passed graceful period yet
2021-10-07 09:00:14,199 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-14:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 09:00:16,997 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-f9c3fd88) (logid:8c74a8b9) StorageCollector is running...
2021-10-07 09:00:17,013 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-f9c3fd88) (logid:8c74a8b9) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 09:00:17,017 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:ctx-f9c3fd88) (logid:8c74a8b9) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 09:00:17,017 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:ctx-f9c3fd88) (logid:8c74a8b9) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 09:00:17,244 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-f9c3fd88) (logid:8c74a8b9) Seq 2-4126986108531640371: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 09:00:25,157 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-90dfb2d4) (logid:eac0f424) AutoScaling Monitor is running...
2021-10-07 09:00:35,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-0f5a6ecc) (logid:758b7140) Found 0 routers to update status.
2021-10-07 09:00:35,355 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-0f5a6ecc) (logid:758b7140) Found 0 VPC's to update Redundant State.
2021-10-07 09:00:35,358 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-0f5a6ecc) (logid:758b7140) Found 0 networks to update RvR status.
2021-10-07 09:00:39,033 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-d07dd6c9) (logid:70e99093) HostStatsCollector is running...
2021-10-07 09:00:39,078 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-d07dd6c9) (logid:70e99093) Seq 1-3253569255798472741: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:00:39,098 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-d07dd6c9) (logid:70e99093) Seq 2-4126986108531640372: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:00:57,951 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 09:00:57,953 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 09:00:57,953 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 09:00:57,963 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Run missing VM report. current time: 1633611657963
2021-10-07 09:00:57,971 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1538
2021-10-07 09:00:57,971 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Detected missing VM. host: 2, vm id: 1538(b574a6bf-240d-4cac-8701-369761544bf6), power state: PowerReportMissing, last state update: 2021/10/07T08:59:34.000Z
2021-10-07 09:00:57,971 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) vm id: 1538 - time since last state update(83963ms) has not passed graceful period yet
2021-10-07 09:00:57,971 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 09:01:05,352 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a0467b87) (logid:abbcde0a) Found 0 routers to update status.
2021-10-07 09:01:05,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a0467b87) (logid:abbcde0a) Found 0 VPC's to update Redundant State.
2021-10-07 09:01:05,360 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-a0467b87) (logid:abbcde0a) Found 0 networks to update RvR status.
2021-10-07 09:01:14,179 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 09:01:14,180 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 09:01:14,180 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 09:01:14,185 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:null) (logid:) Run missing VM report. current time: 1633611674185
2021-10-07 09:01:14,190 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1539
2021-10-07 09:01:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:null) (logid:) Detected missing VM. host: 1, vm id: 1539(478e8eb9-9621-40e4-8a38-f3f6d42c372e), power state: PowerReportMissing, last state update: 2021/10/07T08:59:40.000Z
2021-10-07 09:01:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:null) (logid:) vm id: 1539 - time since last state update(94185ms) has not passed graceful period yet
2021-10-07 09:01:14,190 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-5:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 09:01:17,245 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-1e709280) (logid:b48a7db5) StorageCollector is running...
2021-10-07 09:01:17,266 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-1e709280) (logid:b48a7db5) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 09:01:17,271 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-4:ctx-1e709280) (logid:b48a7db5) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 09:01:17,271 DEBUG [c.c.h.XenServerGuru] (StatsCollector-4:ctx-1e709280) (logid:b48a7db5) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 09:01:17,579 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-1e709280) (logid:b48a7db5) Seq 1-3253569255798472742: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 09:01:18,190 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:null) (logid:) Seq 2-4126986108531638273: Processing: { Ans: , MgmtId: 90520741394843, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:01:25,162 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-d6891f60) (logid:8b89d51a) AutoScaling Monitor is running...
2021-10-07 09:01:35,352 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e3fb206c) (logid:13906c93) Found 0 routers to update status.
2021-10-07 09:01:35,354 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e3fb206c) (logid:13906c93) Found 0 VPC's to update Redundant State.
2021-10-07 09:01:35,358 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e3fb206c) (logid:13906c93) Found 0 networks to update RvR status.
2021-10-07 09:01:39,102 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-51170885) (logid:81b07cb5) HostStatsCollector is running...
2021-10-07 09:01:39,135 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-51170885) (logid:81b07cb5) Seq 1-3253569255798472743: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:01:39,159 DEBUG [c.c.a.t.Request] (StatsCollector-6:ctx-51170885) (logid:81b07cb5) Seq 2-4126986108531640373: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:01:39,727 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-383685db) (logid:742f219d) Resetting hosts suitable for reconnect
2021-10-07 09:01:39,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-383685db) (logid:742f219d) Completed resetting hosts suitable for reconnect
2021-10-07 09:01:39,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-383685db) (logid:742f219d) Acquiring hosts for clusters already owned by this management server
2021-10-07 09:01:39,732 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-383685db) (logid:742f219d) Completed acquiring hosts for clusters already owned by this management server
2021-10-07 09:01:39,732 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-383685db) (logid:742f219d) Acquiring hosts for clusters not owned by any management server
2021-10-07 09:01:39,734 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-383685db) (logid:742f219d) Completed acquiring hosts for clusters not owned by any management server
2021-10-07 09:01:43,034 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) (logid:) Seq 2-4126986108531640369: Processing: { Ans: , MgmtId: 90520741394843, via: 2, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:01:43,066 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) (logid:) Seq 2-4126986108531640369: No more commands found
2021-10-07 09:01:43,066 DEBUG [c.c.a.t.Request] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Seq 2-4126986108531640369: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2021-10-07 09:01:43,154 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) releasing lock for VMTemplateStoragePool 515
2021-10-07 09:01:43,210 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Unable to create Vol[1538|vm=1538|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
2021-10-07 09:01:43,210 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[1538|vm=1538|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1491)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1566)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1160)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 09:01:43,304 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Cleaning up resources for the vm VM[ConsoleProxy|v-1538-VM] in Starting state
2021-10-07 09:01:43,334 DEBUG [c.c.a.t.Request] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Seq 2-4126986108531640374: Sending { Cmd , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.163.36","forceStop":"false","volumesToDisconnect":[],"vmName":"v-1538-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:01:43,878 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Unable to acquire lock on VMTemplateStoragePool 515
2021-10-07 09:01:43,931 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Cleaning up resources for the vm VM[SecondaryStorageVm|s-1539-VM] in Starting state
2021-10-07 09:01:43,959 DEBUG [c.c.a.t.Request] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Seq 1-3253569255798472744: Sending { Cmd , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.28.97","forceStop":"false","volumesToDisconnect":[],"vmName":"s-1539-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:01:46,596 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 2-4126986108531640374: Processing: { Ans: , MgmtId: 90520741394843, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:01:46,596 DEBUG [c.c.a.t.Request] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Seq 2-4126986108531640374: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { StopAnswer } }
2021-10-07 09:01:46,700 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Released nic: NicProfile[5896-1538-null-null-null]
2021-10-07 09:01:46,760 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Releasing ip address for reservationId=4ee9913f-7ce8-4f4c-8e9d-2bb03c903e91, instance=5897
2021-10-07 09:01:46,763 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Released nic: NicProfile[5897-1538-null-null-null]
2021-10-07 09:01:46,801 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Successfully released network resources for the vm VM[ConsoleProxy|v-1538-VM]
2021-10-07 09:01:46,801 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Successfully cleaned up resources for the VM VM[ConsoleProxy|v-1538-VM] in Starting state
2021-10-07 09:01:46,807 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) DeploymentPlanner allocation algorithm: null
2021-10-07 09:01:46,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 500, requested ram: (1.00 GB) 1073741824
2021-10-07 09:01:46,808 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 09:01:46,810 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: [2]
2021-10-07 09:01:46,817 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Searching resources only under specified Pod: 1
2021-10-07 09:01:46,821 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Pod: 1
2021-10-07 09:01:46,855 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 09:01:46,863 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 09:01:46,863 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 09:01:46,872 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) FirstFitAllocator has 2 hosts to check for allocation: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 09:01:46,891 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Found 2 hosts for allocation after prioritization: [Host[-1-Routing], Host[-2-Routing]]
2021-10-07 09:01:46,891 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for speed=500Mhz, Ram=1024 MB
2021-10-07 09:01:46,912 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:01:46,913 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:46,921 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:46,922 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73100 , Requested CPU: 500
2021-10-07 09:01:46,922 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.39 GB) 65915731968 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 09:01:46,922 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 09:01:46,922 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 1, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:01:46,922 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 1, used: (512.00 MB) 536870912, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:01:46,922 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 1
2021-10-07 09:01:46,922 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host name: gd-cs01-clst01-compute02, hostId: 2 is in avoid set, skipping this and trying other available hosts
2021-10-07 09:01:46,922 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194 FirstFitRoutingAllocator) (logid:bcef1bbe) Host Allocator returning 1 suitable hosts
2021-10-07 09:01:46,929 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking suitable pools for volume (Id, Type): (1538,ROOT)
2021-10-07 09:01:46,929 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) We need to allocate new storagepool for this volume
2021-10-07 09:01:46,934 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Calling StoragePoolAllocators to find suitable pools
2021-10-07 09:01:46,943 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) System VMs will use shared storage for zone id=1
2021-10-07 09:01:46,943 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 09:01:46,943 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 09:01:46,943 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 09:01:46,946 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 09:01:46,951 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 09:01:46,952 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) StoragePool is in avoid set, skipping this pool
2021-10-07 09:01:46,952 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator returning 0 suitable storage pools
2021-10-07 09:01:46,952 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) ZoneWideStoragePoolAllocator to find storage pool
2021-10-07 09:01:46,957 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) No suitable pools found for volume: Vol[1538|vm=1538|ROOT] under cluster: 1
2021-10-07 09:01:46,957 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) No suitable pools found
2021-10-07 09:01:46,957 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) No suitable storagePools found under this Cluster: 1
2021-10-07 09:01:46,967 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2021-10-07 09:01:46,969 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Searching resources only under specified Pod: 1
2021-10-07 09:01:46,972 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Pod: 1
2021-10-07 09:01:46,977 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Removing from the clusterId list these clusters from avoid set: [1]
2021-10-07 09:01:46,979 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) No clusters found after removing disabled clusters and clusters in avoid list, returning.
2021-10-07 09:01:47,074 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2
2021-10-07 09:01:47,120 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:47,120 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Hosts's actual total RAM: (61.90 GB) 66469449728 and RAM after applying overprovisioning: (61.90 GB) 66469449728
2021-10-07 09:01:47,120 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) release cpu from host: 2, old used: 500,reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 09:01:47,120 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) release mem from host: 2, old used: (1.00 GB) 1073741824,reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; new used: (0 bytes) 0,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 09:01:47,168 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 09:01:47,168 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737 ctx-32f26194) (logid:bcef1bbe) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 09:01:47,168 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 1538, job origin: 3439
2021-10-07 09:01:47,169 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Unable to complete AsyncJobVO {id:4737, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 08:59:33 EDT 2021, removed: null}, job origin:3439
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 09:01:47,176 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Complete async job-4737, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyADdjb20uY2xvdWQuZXhjZXB0aW9uLkluc3VmZmljaWVudFNlcnZlckNhcGFjaXR5RXhjZXB0aW9uAAAAAFZNT3ACAAFaABVhZmZpbml0eUdyb3Vwc0FwcGxpZWR4cgAxY29tLmNsb3VkLmV4Y2VwdGlvbi5JbnN1ZmZpY2llbnRDYXBhY2l0eUV4Y2VwdGlvbgAAAABWTU91AgACTAACaWR0ABBMamF2YS9sYW5nL0xvbmc7TAAFc2NvcGV0ABFMamF2YS9sYW5nL0NsYXNzO3hyACJjb20uY2xvdWQuZXhjZXB0aW9uLkNsb3VkRXhjZXB0aW9ueeiOG_HkRMcCAAJMAAtjc0Vycm9yQ29kZXQAE0xqYXZhL2xhbmcvSW50ZWdlcjtMAAZpZExpc3R0ABVMamF2YS91dGlsL0FycmF5TGlzdDt4cgATamF2YS5sYW5nLkV4Y2VwdGlvbtD9Hz4aOxzEAgAAeHIAE2phdmEubGFuZy5UaHJvd2FibGXVxjUnOXe4ywMABEwABWNhdXNldAAVTGphdmEvbGFuZy9UaHJvd2FibGU7TAANZGV0YWlsTWVzc2FnZXQAEkxqYXZhL2xhbmcvU3RyaW5nO1sACnN0YWNrVHJhY2V0AB5bTGphdmEvbGFuZy9TdGFja1RyYWNlRWxlbWVudDtMABRzdXBwcmVzc2VkRXhjZXB0aW9uc3QAEExqYXZhL3V0aWwvTGlzdDt4cHEAfgANdAA8VW5hYmxlIHRvIGNyZWF0ZSBhIGRlcGxveW1lbnQgZm9yIFZNW0NvbnNvbGVQcm94eXx2LTE1MzgtVk1ddXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAAUc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAIQgAGZm9ybWF0SQAKbGluZU51bWJlckwAD2NsYXNzTG9hZGVyTmFtZXEAfgAKTAAOZGVjbGFyaW5nQ2xhc3NxAH4ACkwACGZpbGVOYW1lcQB-AApMAAptZXRob2ROYW1lcQB-AApMAAptb2R1bGVOYW1lcQB-AApMAA1tb2R1bGVWZXJzaW9ucQB-AAp4cAEAAARfdAADYXBwdAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0ABBvcmNoZXN0cmF0ZVN0YXJ0cHBzcQB-ABEBAAAVfnEAfgATcQB-ABRxAH4AFXEAfgAWcHBzcQB-ABEA_____3B0AC9qZGsuaW50ZXJuYWwucmVmbGVjdC5HZW5lcmF0ZWRNZXRob2RBY2Nlc3NvcjIyNXB0AAZpbnZva2VwcHNxAH4AEQIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AGnQACWphdmEuYmFzZXQABzExLjAuMTFzcQB-ABECAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAacQB-AB5xAH4AH3NxAH4AEQEAAABrcQB-ABN0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4AEQEAABYlcQB-ABNxAH4AFHEAfgAVcQB-ACZwcHNxAH4AEQEAAABmcQB-ABN0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnBwc3EAfgARAQAAAmxxAH4AE3QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgARAQAAADBxAH4AE3QAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-ABEBAAAAN3EAfgATdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-ABEBAAAAZnEAfgATdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA2dAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-ABEBAAAANHEAfgATcQB-ADlxAH4ANnQADnJ1bldpdGhDb250ZXh0cHBzcQB-ABEBAAAALXEAfgATdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ADJxAH4AM3Bwc3EAfgARAQAAAjhxAH4AE3EAfgAtcQB-AC5xAH4AM3Bwc3EAfgARAgAAAgNwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ADdxAH4AHnEAfgAfc3EAfgARAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAzcQB-AB5xAH4AH3NxAH4AEQIAAARocHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-AB5xAH4AH3NxAH4AEQIAAAJ0cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ASHEAfgAzcQB-AB5xAH4AH3NxAH4AEQIAAAM9cHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAzcQB-AB5xAH4AH3NyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHhzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAQ73NyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cQB-AFIAAAAAAAAAAXZyABdjb20uY2xvdWQuZGMuRGF0YUNlbnRlclPoJBrS07ZcAgAAeHAA
2021-10-07 09:01:47,182 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Publish async job-4737 complete on message bus
2021-10-07 09:01:47,182 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Wake up jobs related to job-4737
2021-10-07 09:01:47,182 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Update db status for job-4737
2021-10-07 09:01:47,189 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Wake up jobs joined with job-4737 and disjoin all subjobs created from job- 4737
2021-10-07 09:01:47,222 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:null) (logid:) Seq 1-3253569255798472744: Processing: { Ans: , MgmtId: 90520741394843, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:01:47,223 DEBUG [c.c.a.t.Request] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Seq 1-3253569255798472744: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { StopAnswer } }
2021-10-07 09:01:47,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Done executing com.cloud.vm.VmWorkStart for job-4737
2021-10-07 09:01:47,291 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-134:ctx-cc1bf559 job-3439/job-4737) (logid:bcef1bbe) Remove job-4737 from job monitoring
2021-10-07 09:01:47,319 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Released nic: NicProfile[5899-1539-null-null-null]
2021-10-07 09:01:47,343 WARN [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Exception while trying to start console proxy
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1538-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 09:01:47,344 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Unable to start console proxy vm for standby capacity, vm id : 1538, will recycle it and start a new one
2021-10-07 09:01:47,377 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Releasing ip address for reservationId=5048be5c-eb7d-4f33-bee7-67485456483c, instance=5900
2021-10-07 09:01:47,379 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Released nic: NicProfile[5900-1539-null-null-null]
2021-10-07 09:01:47,404 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Sync job-4739 execution on object VmWorkJobQueue.1538
2021-10-07 09:01:47,465 DEBUG [c.c.n.g.StorageNetworkGuru] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Release an storage ip 192.168.204.198
2021-10-07 09:01:47,588 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Successfully released network resources for the vm VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 09:01:47,588 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Successfully cleaned up resources for the VM VM[SecondaryStorageVm|s-1539-VM] in Starting state
2021-10-07 09:01:47,742 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 1
2021-10-07 09:01:47,767 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:47,767 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Hosts's actual total RAM: (61.89 GB) 66452602880 and RAM after applying overprovisioning: (61.89 GB) 66452602880
2021-10-07 09:01:47,767 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) release cpu from host: 1, old used: 500,reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 09:01:47,767 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) release mem from host: 1, old used: (512.00 MB) 536870912,reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; new used: (0 bytes) 0,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 09:01:47,808 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Caught CloudRuntimeException, returning job failed com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on VMTemplateStoragePool: 515
2021-10-07 09:01:47,809 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Done executing VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1539,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 09:01:47,816 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Complete async job-4738, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ABtVbmFibGUgdG8gc3RhcnQgVk0gaW5zdGFuY2V1cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABNzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAAhCAAZmb3JtYXRJAApsaW5lTnVtYmVyTAAPY2xhc3NMb2FkZXJOYW1lcQB-AAVMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABUwACm1vZHVsZU5hbWVxAH4ABUwADW1vZHVsZVZlcnNpb25xAH4ABXhwAQAAFYN0AANhcHB0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAEG9yY2hlc3RyYXRlU3RhcnRwcHNxAH4ADAD_____cHQAL2pkay5pbnRlcm5hbC5yZWZsZWN0LkdlbmVyYXRlZE1ldGhvZEFjY2Vzc29yMjI1cHQABmludm9rZXBwc3EAfgAMAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAUdAAJamF2YS5iYXNldAAHMTEuMC4xMXNxAH4ADAIAAAI2cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ABRxAH4AGHEAfgAZc3EAfgAMAQAAAGtxAH4ADnQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgAMAQAAFiVxAH4ADnEAfgAPcQB-ABBxAH4AIHBwc3EAfgAMAQAAAGZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-AAwBAAACbHEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-AAwBAAAAMHEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4ADAEAAAA3cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABmcQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADB0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA0cQB-AA5xAH4AM3EAfgAwdAAOcnVuV2l0aENvbnRleHRwcHNxAH4ADAEAAAAtcQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ALHEAfgAtcHBzcQB-AAwBAAACOHEAfgAOcQB-ACdxAH4AKHEAfgAtcHBzcQB-AAwCAAACA3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AMXEAfgAYcQB-ABlzcQB-AAwCAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AGHEAfgAZc3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBCcQB-AC1xAH4AGHEAfgAZc3EAfgAMAgAAAz1wdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-AC1xAH4AGHEAfgAZc3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAEJp3CAAAAAAAAAAAeA
2021-10-07 09:01:47,823 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Publish async job-4738 complete on message bus
2021-10-07 09:01:47,824 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Wake up jobs related to job-4738
2021-10-07 09:01:47,824 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Update db status for job-4738
2021-10-07 09:01:47,828 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738 ctx-b82dd15a) (logid:e381c0f1) Wake up jobs joined with job-4738 and disjoin all subjobs created from job- 4738
2021-10-07 09:01:47,893 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738) (logid:e381c0f1) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 1539, job origin: 3440
2021-10-07 09:01:47,893 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738) (logid:e381c0f1) Done executing com.cloud.vm.VmWorkStart for job-4738
2021-10-07 09:01:47,900 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-136:ctx-c542acef job-3440/job-4738) (logid:e381c0f1) Remove job-4738 from job monitoring
2021-10-07 09:01:47,951 WARN [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Exception while trying to start secondary storage vm
com.cloud.utils.exception.CloudRuntimeException: Unable to start VM instance
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5507)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 09:01:47,957 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Unable to start secondary storage vm for standby capacity, vm id : 1539, will recycle it and start a new one
2021-10-07 09:01:48,041 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Sync job-4740 execution on object VmWorkJobQueue.1539
2021-10-07 09:01:48,336 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cf47ae1b) (logid:d5f97561) Execute sync-queue item: SyncQueueItemVO {id:4703, queueId: 4699, contentType: AsyncJob, contentId: 4739, lastProcessMsid: 90520741394843, lastprocessNumber: 3, lastProcessTime: Thu Oct 07 09:01:48 EDT 2021, created: Thu Oct 07 09:01:47 EDT 2021}
2021-10-07 09:01:48,361 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cf47ae1b) (logid:d5f97561) Schedule queued job-4739
2021-10-07 09:01:48,395 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cf47ae1b) (logid:d5f97561) Execute sync-queue item: SyncQueueItemVO {id:4704, queueId: 4700, contentType: AsyncJob, contentId: 4740, lastProcessMsid: 90520741394843, lastprocessNumber: 3, lastProcessTime: Thu Oct 07 09:01:48 EDT 2021, created: Thu Oct 07 09:01:48 EDT 2021}
2021-10-07 09:01:48,400 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-cf47ae1b) (logid:d5f97561) Schedule queued job-4740
2021-10-07 09:01:48,401 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739) (logid:a10cb2fe) Add job-4739 into job monitoring
2021-10-07 09:01:48,410 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739) (logid:bcef1bbe) Executing AsyncJobVO {id:4739, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGAnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 09:01:47 EDT 2021, removed: null}
2021-10-07 09:01:48,411 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739) (logid:bcef1bbe) Run VM work job: com.cloud.vm.VmWorkStop for VM 1538, job origin: 3439
2021-10-07 09:01:48,421 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1538,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 09:01:48,431 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) VM is already stopped: VM[ConsoleProxy|v-1538-VM]
2021-10-07 09:01:48,432 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1538,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 09:01:48,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) Complete async job-4739, jobStatus: SUCCEEDED, resultCode: 0, result: null
2021-10-07 09:01:48,439 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) Publish async job-4739 complete on message bus
2021-10-07 09:01:48,440 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) Wake up jobs related to job-4739
2021-10-07 09:01:48,440 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) Update db status for job-4739
2021-10-07 09:01:48,442 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739 ctx-74f1a4d3) (logid:bcef1bbe) Wake up jobs joined with job-4739 and disjoin all subjobs created from job- 4739
2021-10-07 09:01:48,442 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740) (logid:8cbc4233) Add job-4740 into job monitoring
2021-10-07 09:01:48,454 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740) (logid:e381c0f1) Executing AsyncJobVO {id:4740, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGA3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 09:01:48 EDT 2021, removed: null}
2021-10-07 09:01:48,455 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740) (logid:e381c0f1) Run VM work job: com.cloud.vm.VmWorkStop for VM 1539, job origin: 3440
2021-10-07 09:01:48,459 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1539,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 09:01:48,466 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) VM is already stopped: VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 09:01:48,467 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":1539,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 09:01:48,469 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) Complete async job-4740, jobStatus: SUCCEEDED, resultCode: 0, result: null
2021-10-07 09:01:48,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) Publish async job-4740 complete on message bus
2021-10-07 09:01:48,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) Wake up jobs related to job-4740
2021-10-07 09:01:48,474 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) Update db status for job-4740
2021-10-07 09:01:48,478 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740 ctx-eae5883d) (logid:e381c0f1) Wake up jobs joined with job-4740 and disjoin all subjobs created from job- 4740
2021-10-07 09:01:48,486 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739) (logid:bcef1bbe) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 1538, job origin: 3439
2021-10-07 09:01:48,486 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739) (logid:bcef1bbe) Done executing com.cloud.vm.VmWorkStop for job-4739
2021-10-07 09:01:48,493 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-137:ctx-fee8a036 job-3439/job-4739) (logid:bcef1bbe) Remove job-4739 from job monitoring
2021-10-07 09:01:48,555 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740) (logid:e381c0f1) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 1539, job origin: 3440
2021-10-07 09:01:48,556 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740) (logid:e381c0f1) Done executing com.cloud.vm.VmWorkStop for job-4740
2021-10-07 09:01:48,561 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-139:ctx-1aa455c8 job-3440/job-4740) (logid:e381c0f1) Remove job-4740 from job monitoring
2021-10-07 09:01:48,673 DEBUG [c.c.c.CapacityManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) VM state transitted from :Stopped to Expunging with event: ExpungeOperationvm's original host id: null new host id: null host id before state transition: null
2021-10-07 09:01:48,673 DEBUG [c.c.c.CapacityManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) VM state transitted from :Stopped to Expunging with event: ExpungeOperationvm's original host id: null new host id: null host id before state transition: null
2021-10-07 09:01:48,695 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Destroying vm VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 09:01:48,695 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Cleaning up NICS
2021-10-07 09:01:48,695 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Cleaning network for vm: 1539
2021-10-07 09:01:48,697 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Destroying vm VM[ConsoleProxy|v-1538-VM]
2021-10-07 09:01:48,697 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Cleaning up NICS
2021-10-07 09:01:48,697 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Cleaning network for vm: 1538
2021-10-07 09:01:48,769 DEBUG [c.c.n.g.PublicNetworkGuru] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) public network deallocate network: networkId: 200, ip: 192.168.200.200
2021-10-07 09:01:48,782 DEBUG [c.c.n.g.PublicNetworkGuru] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) public network deallocate network: networkId: 200, ip: 192.168.200.199
2021-10-07 09:01:48,812 DEBUG [c.c.n.g.PublicNetworkGuru] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Deallocated nic: NicProfile[5898-1539-null-null-null]
2021-10-07 09:01:48,839 DEBUG [c.c.n.g.PublicNetworkGuru] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Deallocated nic: NicProfile[5895-1538-null-null-null]
2021-10-07 09:01:48,864 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Removed nic id=5898
2021-10-07 09:01:48,871 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Revoving nic secondary ip entry ...
2021-10-07 09:01:48,889 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Removed nic id=5895
2021-10-07 09:01:48,896 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Revoving nic secondary ip entry ...
2021-10-07 09:01:48,965 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Removed nic id=5899
2021-10-07 09:01:48,972 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Revoving nic secondary ip entry ...
2021-10-07 09:01:48,990 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Removed nic id=5896
2021-10-07 09:01:48,996 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Revoving nic secondary ip entry ...
2021-10-07 09:01:49,132 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Removed nic id=5900
2021-10-07 09:01:49,137 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Revoving nic secondary ip entry ...
2021-10-07 09:01:49,165 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Removed nic id=5897
2021-10-07 09:01:49,172 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Revoving nic secondary ip entry ...
2021-10-07 09:01:49,175 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage
2021-10-07 09:01:49,178 DEBUG [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Cleaning storage for vm: 1538
2021-10-07 09:01:49,249 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Removed nic id=5901
2021-10-07 09:01:49,255 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Revoving nic secondary ip entry ...
2021-10-07 09:01:49,258 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed storage
2021-10-07 09:01:49,261 DEBUG [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Cleaning storage for vm: 1539
2021-10-07 09:01:49,292 INFO [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Expunge volume with no data store specified
2021-10-07 09:01:49,298 INFO [o.a.c.s.v.VolumeServiceImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Volume 1538 is not referred anywhere, remove it from volumes table
2021-10-07 09:01:49,364 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Expunged VM[ConsoleProxy|v-1538-VM]
2021-10-07 09:01:49,386 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Expunge volume with no data store specified
2021-10-07 09:01:49,393 INFO [o.a.c.s.v.VolumeServiceImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Volume 1539 is not referred anywhere, remove it from volumes table
2021-10-07 09:01:49,425 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) received console proxy alert
2021-10-07 09:01:49,428 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) Console proxy creation failure, zone: gd-cs01
2021-10-07 09:01:49,432 WARN [c.c.a.AlertManagerImpl] (consoleproxy-1:ctx-be40e983) (logid:1b69de73) AlertType:: 10 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Console proxy creation failure. zone: gd-cs01, error details: null
2021-10-07 09:01:49,461 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Expunged VM[SecondaryStorageVm|s-1539-VM]
2021-10-07 09:01:49,517 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) received secondary storage vm alert
2021-10-07 09:01:49,520 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Secondary Storage Vm creation failure, zone: gd-cs01
2021-10-07 09:01:49,523 WARN [c.c.a.AlertManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) AlertType:: 19 | dataCenterId:: 1 | podId:: null | clusterId:: null | message:: Secondary Storage Vm creation failure. zone: gd-cs01, error details: null
2021-10-07 09:01:49,553 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Zone 1 is ready to launch console proxy
2021-10-07 09:01:49,553 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Expand console proxy standby capacity for zone gd-cs01
2021-10-07 09:01:49,557 INFO [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) No stopped console proxy is available, need to allocate a new console proxy
2021-10-07 09:01:49,563 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Assign console proxy from a newly started instance for request from data center : 1
2021-10-07 09:01:49,634 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-f78588a2) (logid:e47bea8e) Primary secondary storage is not even started, wait until next turn
2021-10-07 09:01:49,680 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Zone 1 is ready to launch secondary storage VM
2021-10-07 09:01:49,691 INFO [o.a.c.s.PremiumSecondaryStorageManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) No running secondary storage vms found in datacenter id=1, starting one
2021-10-07 09:01:49,702 INFO [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
2021-10-07 09:01:49,709 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Assign secondary storage vm from a newly started instance for request from data center : 1
2021-10-07 09:01:49,751 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Found existing network configuration for offering [Network Offering [1-Public-System-Public-Network]: Ntwk[200|Public|1]
2021-10-07 09:01:49,751 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 09:01:49,875 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2021-10-07 09:01:49,875 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 09:01:50,059 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]: Ntwk[201|Management|2]
2021-10-07 09:01:50,059 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 09:01:50,255 INFO [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) allocating virtual machine from template:8c443ce1-22f9-11ec-a65c-52540037f312 with hostname:v-1540-VM and 3 networks
2021-10-07 09:01:50,264 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Allocating entries for VM: VM[ConsoleProxy|v-1540-VM]
2021-10-07 09:01:50,271 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Allocating nics for VM[ConsoleProxy|v-1540-VM]
2021-10-07 09:01:50,276 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Allocating nic for vm VM[ConsoleProxy|v-1540-VM] in network Ntwk[200|Public|1] with requested profile NicProfile[0-0-null-null-null]
2021-10-07 09:01:50,358 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Allocating nic for vm VM[ConsoleProxy|v-1540-VM] in network Ntwk[202|Control|3] with requested profile null
2021-10-07 09:01:50,377 DEBUG [o.a.c.e.o.NetworkOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Allocating nic for vm VM[ConsoleProxy|v-1540-VM] in network Ntwk[201|Management|2] with requested profile null
2021-10-07 09:01:50,415 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Allocating disks for VM[ConsoleProxy|v-1540-VM]
2021-10-07 09:01:50,415 INFO [o.a.c.e.o.VolumeOrchestrator] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) adding disk object ROOT-1540 to v-1540-VM
2021-10-07 09:01:50,477 DEBUG [c.c.v.VirtualMachineManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Allocation completed for VM: VM[ConsoleProxy|v-1540-VM]
2021-10-07 09:01:50,485 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) received console proxy alert
2021-10-07 09:01:50,489 DEBUG [c.c.a.ConsoleProxyAlertAdapter] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) New console proxy created, zone: gd-cs01, proxy: v-1540-VM, public IP: null, private IP: null
2021-10-07 09:01:50,561 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (consoleproxy-1:ctx-3b7ac7ea) (logid:28131d73) Sync job-4741 execution on object VmWorkJobQueue.1540
2021-10-07 09:01:52,505 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-798a0d95) (logid:a2755c5b) Execute sync-queue item: SyncQueueItemVO {id:4705, queueId: 4705, contentType: AsyncJob, contentId: 4741, lastProcessMsid: 90520741394843, lastprocessNumber: 1, lastProcessTime: Thu Oct 07 09:01:52 EDT 2021, created: Thu Oct 07 09:01:50 EDT 2021}
2021-10-07 09:01:52,510 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-798a0d95) (logid:a2755c5b) Schedule queued job-4741
2021-10-07 09:01:52,613 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741) (logid:d60bf383) Add job-4741 into job monitoring
2021-10-07 09:01:52,625 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741) (logid:bcef1bbe) Executing AsyncJobVO {id:4741, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGBHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 09:01:50 EDT 2021, removed: null}
2021-10-07 09:01:52,626 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741) (logid:bcef1bbe) Run VM work job: com.cloud.vm.VmWorkStart for VM 1540, job origin: 3439
2021-10-07 09:01:52,631 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Execute VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1540,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 09:01:52,656 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2021-10-07 09:01:52,659 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Successfully transitioned to start state for VM[ConsoleProxy|v-1540-VM] reservation id = 31c69384-bc59-40af-9e1d-e0fe73a62a59
2021-10-07 09:01:52,758 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 09:01:52,769 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) DeploymentPlanner allocation algorithm: null
2021-10-07 09:01:52,769 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB) 1073741824
2021-10-07 09:01:52,769 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 09:01:52,775 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 09:01:52,782 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Searching all possible resources under this Zone: 1
2021-10-07 09:01:52,787 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 09:01:52,839 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 09:01:52,848 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 09:01:52,848 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 09:01:52,860 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 09:01:52,906 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 09:01:52,906 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for speed=500Mhz, Ram=1024 MB
2021-10-07 09:01:52,935 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:01:52,935 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:52,945 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:52,945 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 09:01:52,945 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.90 GB) 66469449728 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 09:01:52,945 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 09:01:52,945 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:01:52,946 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 2, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:01:52,946 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 2
2021-10-07 09:01:52,983 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:01:52,983 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:52,996 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:52,996 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 09:01:52,996 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 09:01:52,996 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 09:01:52,996 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:01:52,996 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:01:52,996 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 1
2021-10-07 09:01:52,996 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host Allocator returning 2 suitable hosts
2021-10-07 09:01:53,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking suitable pools for volume (Id, Type): (1540,ROOT)
2021-10-07 09:01:53,011 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) We need to allocate new storagepool for this volume
2021-10-07 09:01:53,015 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Calling StoragePoolAllocators to find suitable pools
2021-10-07 09:01:53,025 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) System VMs will use shared storage for zone id=1
2021-10-07 09:01:53,026 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 09:01:53,026 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 09:01:53,026 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 09:01:53,029 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 09:01:53,035 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 09:01:53,039 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Storage pool null (2) does not supply IOPS capacity, assuming enough capacity
2021-10-07 09:01:53,042 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking pool 2 for storage, totalSize: 30005954347008, usedBytes: 11020345267, usedPct: 3.6727194674609234E-4, disable threshold: 0.85
2021-10-07 09:01:53,042 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Destination pool id: 2
2021-10-07 09:01:53,083 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Pool ID for the volume with ID 1540 is null
2021-10-07 09:01:53,097 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Found storage pool CS_Ceph of type RBD with over-provisioning factor 2
2021-10-07 09:01:53,097 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Total over-provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 09:01:53,097 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Total capacity of the pool CS_Ceph with ID 2 is (54.5805 TB) 60011908694016
2021-10-07 09:01:53,100 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking pool: 2 for storage allocation , maxSize : (54.5805 TB) 60011908694016, totalAllocatedSize : (0 bytes) 0, askingSize : (0 bytes) 0, allocated disable threshold: 0.85
2021-10-07 09:01:53,101 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
2021-10-07 09:01:53,101 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
2021-10-07 09:01:53,104 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking if host: 2 can access any suitable storage pool for volume: ROOT
2021-10-07 09:01:53,108 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Host: 2 can access pool: 2
2021-10-07 09:01:53,111 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Found a potential host id: 2 name: gd-cs01-clst01-compute02 and associated storage pools for this VM
2021-10-07 09:01:53,115 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1540|ROOT-->Pool(2))]
2021-10-07 09:01:53,115 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Deployment found - P0=VM[ConsoleProxy|v-1540-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1540|ROOT-->Pool(2))]
2021-10-07 09:01:53,188 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 2 host id before state transition: null
2021-10-07 09:01:53,249 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:53,249 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) We are allocating VM, increasing the used capacity of this host:2
2021-10-07 09:01:53,249 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Current Used CPU: 0 , Free CPU:73600 ,Requested CPU: 500
2021-10-07 09:01:53,249 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Current Used RAM: (0 bytes) 0 , Free RAM:(61.90 GB) 66469449728 ,Requested RAM: (1.00 GB) 1073741824
2021-10-07 09:01:53,250 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) CPU STATS after allocation: for host: 2, old used: 0, old reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used:500, reserved:0; requested cpu:500,alloc_from_last:false
2021-10-07 09:01:53,250 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) RAM STATS after allocation: for host: 2, old used: (0 bytes) 0, old reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; new used: (1.00 GB) 1073741824, reserved: (0 bytes) 0; requested mem: (1.00 GB) 1073741824,alloc_from_last:false
2021-10-07 09:01:53,263 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:01:53,263 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:53,273 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:53,273 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) We need to allocate to the last host again, so checking if there is enough reserved capacity
2021-10-07 09:01:53,273 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Reserved CPU: 0 , Requested CPU: 500
2021-10-07 09:01:53,273 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Reserved RAM: (0 bytes) 0 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 09:01:53,273 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) STATS: Failed to alloc resource from host: 2 reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (1.00 GB) 1073741824
2021-10-07 09:01:53,274 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Host does not have enough reserved CPU available, cannot allocate to this host.
2021-10-07 09:01:53,274 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:53,283 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:53,283 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 09:01:53,283 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Free RAM: (61.90 GB) 66469449728 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 09:01:53,283 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 09:01:53,283 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) STATS: Can alloc CPU from host: 2, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:01:53,283 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) STATS: Can alloc MEM from host: 2, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:01:53,388 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Network id=200 is already implemented
2021-10-07 09:01:53,512 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Network id=202 is already implemented
2021-10-07 09:01:54,081 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Reserved NIC for v-1540-VM [ipv4:169.254.134.219 netmask:255.255.0.0 gateway:169.254.0.1]
2021-10-07 09:01:54,154 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Network id=201 is already implemented
2021-10-07 09:01:54,236 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Releasing ip address for instance=5904
2021-10-07 09:01:54,292 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Allocated a nic NicProfile[5904-1540-31c69384-bc59-40af-9e1d-e0fe73a62a59-192.168.204.163-null] for VM[ConsoleProxy|v-1540-VM]
2021-10-07 09:01:54,435 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) template 3 is already in store:5, type:Image
2021-10-07 09:01:54,464 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Not found (templateId:3poolId:2conf:null) in template_spool_ref, persisting it
2021-10-07 09:01:54,516 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) template 3 is already in store:2, type:Primary
2021-10-07 09:01:54,522 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 516
2021-10-07 09:01:54,528 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Acquire lock on VMTemplateStoragePool 516 with timeout 3600 seconds
2021-10-07 09:01:54,571 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) lock is acquired for VMTemplateStoragePool 516
2021-10-07 09:01:54,644 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2021-10-07 09:01:54,685 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand
2021-10-07 09:01:54,694 DEBUG [c.c.a.t.Request] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Seq 2-4126986108531640375: Sending { Cmd , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/1/3/","origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.204.62/cloudstack","_role":"Image"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"https://download.cloudstack.org/systemvm/4.15/systemvmtemplate-4.15.1-kvm.qcow2.bz2","uuid":"8c443ce1-22f9-11ec-a65c-52540037f312","id":"3","format":"QCOW2","accountId":"1","checksum":"0e9f9a7d0957c3e0a2088e41b2da2cec","hvm":"false","displayText":"SystemVM Template (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6db2511f-6992-3120-b49e-fafdfa8ec257","id":"2","poolType":"RBD","host":"192.168.206.63","path":"cloudstack","port":"0","url":"RBD://192.168.206.63/cloudstack/?ROLE=Primary&STOREUUID=6db2511f-6992-3120-b49e-fafdfa8ec257","isManaged":"false"}},"name":"routing-3","hypervisorType":"KVM","bootable":"false","uniqueName":"routing-3","directDownload":"false","deployAsIs":"false"}},"executeInSequence":"true","options":{},"options2":{},"wait":"10800","bypassHostMaintenance":"false"}}] }
2021-10-07 09:01:54,928 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Found existing network configuration for offering [Network Offering [1-Public-System-Public-Network]: Ntwk[200|Public|1]
2021-10-07 09:01:54,928 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 09:01:55,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2021-10-07 09:01:55,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 09:01:55,152 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]: Ntwk[201|Management|2]
2021-10-07 09:01:55,152 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 09:01:55,266 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Found existing network configuration for offering [Network Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
2021-10-07 09:01:55,266 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Releasing lock for Acct[88007017-22fa-11ec-a65c-52540037f312-system]
2021-10-07 09:01:55,427 INFO [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) allocating virtual machine from template:8c443ce1-22f9-11ec-a65c-52540037f312 with hostname:s-1541-VM and 4 networks
2021-10-07 09:01:55,445 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocating entries for VM: VM[SecondaryStorageVm|s-1541-VM]
2021-10-07 09:01:55,451 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocating nics for VM[SecondaryStorageVm|s-1541-VM]
2021-10-07 09:01:55,455 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocating nic for vm VM[SecondaryStorageVm|s-1541-VM] in network Ntwk[200|Public|1] with requested profile NicProfile[0-0-null-null-null]
2021-10-07 09:01:55,519 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocating nic for vm VM[SecondaryStorageVm|s-1541-VM] in network Ntwk[202|Control|3] with requested profile null
2021-10-07 09:01:55,532 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocating nic for vm VM[SecondaryStorageVm|s-1541-VM] in network Ntwk[201|Management|2] with requested profile null
2021-10-07 09:01:55,551 DEBUG [o.a.c.e.o.NetworkOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocating nic for vm VM[SecondaryStorageVm|s-1541-VM] in network Ntwk[203|Storage|4] with requested profile null
2021-10-07 09:01:55,569 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocating disks for VM[SecondaryStorageVm|s-1541-VM]
2021-10-07 09:01:55,569 INFO [o.a.c.e.o.VolumeOrchestrator] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) adding disk object ROOT-1541 to s-1541-VM
2021-10-07 09:01:55,617 DEBUG [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Allocation completed for VM: VM[SecondaryStorageVm|s-1541-VM]
2021-10-07 09:01:55,626 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) received secondary storage vm alert
2021-10-07 09:01:55,629 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) New secondary storage vm created, zone: gd-cs01, secStorageVm: s-1541-VM, public IP: null, private IP: null
2021-10-07 09:01:55,696 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (secstorage-1:ctx-e9b8c67d) (logid:9c5e35ad) Sync job-4742 execution on object VmWorkJobQueue.1541
2021-10-07 09:01:56,439 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-50dbb166) (logid:8761975a) Execute sync-queue item: SyncQueueItemVO {id:4706, queueId: 4706, contentType: AsyncJob, contentId: 4742, lastProcessMsid: 90520741394843, lastprocessNumber: 1, lastProcessTime: Thu Oct 07 09:01:56 EDT 2021, created: Thu Oct 07 09:01:55 EDT 2021}
2021-10-07 09:01:56,445 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-50dbb166) (logid:8761975a) Schedule queued job-4742
2021-10-07 09:01:56,561 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742) (logid:b99fe295) Add job-4742 into job monitoring
2021-10-07 09:01:56,573 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742) (logid:e381c0f1) Executing AsyncJobVO {id:4742, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGBXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 09:01:55 EDT 2021, removed: null}
2021-10-07 09:01:56,574 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742) (logid:e381c0f1) Run VM work job: com.cloud.vm.VmWorkStart for VM 1541, job origin: 3440
2021-10-07 09:01:56,581 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Execute VM work job: com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":1541,"handlerName":"VirtualMachineManagerImpl"}
2021-10-07 09:01:56,603 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2021-10-07 09:01:56,605 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Successfully transitioned to start state for VM[SecondaryStorageVm|s-1541-VM] reservation id = 2666993c-031e-4140-bf6c-2e53b5afb577
2021-10-07 09:01:56,656 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 09:01:56,675 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) DeploymentPlanner allocation algorithm: null
2021-10-07 09:01:56,676 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: (512.00 MB) 536870912
2021-10-07 09:01:56,676 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 09:01:56,681 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Deploy avoids pods: null, clusters: null, hosts: null
2021-10-07 09:01:56,693 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Searching all possible resources under this Zone: 1
2021-10-07 09:01:56,705 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 09:01:56,760 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 09:01:56,764 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 09:01:56,764 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 09:01:56,771 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 09:01:56,793 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 09:01:56,793 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Looking for speed=500Mhz, Ram=512 MB
2021-10-07 09:01:56,822 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:01:56,822 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:56,836 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:56,836 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73100 , Requested CPU: 500
2021-10-07 09:01:56,836 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (60.90 GB) 65395707904 , Requested RAM: (512.00 MB) 536870912
2021-10-07 09:01:56,836 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 09:01:56,837 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:01:56,837 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 2, used: (1.00 GB) 1073741824, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:01:56,837 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 2
2021-10-07 09:01:56,889 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:01:56,889 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:56,906 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:56,906 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Free CPU: 73600 , Requested CPU: 500
2021-10-07 09:01:56,906 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (512.00 MB) 536870912
2021-10-07 09:01:56,906 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 09:01:56,906 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:01:56,906 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:01:56,907 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Found a suitable host, adding to list: 1
2021-10-07 09:01:56,907 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37 FirstFitRoutingAllocator) (logid:e381c0f1) Host Allocator returning 2 suitable hosts
2021-10-07 09:01:56,924 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking suitable pools for volume (Id, Type): (1541,ROOT)
2021-10-07 09:01:56,924 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) We need to allocate new storagepool for this volume
2021-10-07 09:01:56,930 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Calling StoragePoolAllocators to find suitable pools
2021-10-07 09:01:56,947 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) System VMs will use shared storage for zone id=1
2021-10-07 09:01:56,947 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 09:01:56,947 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 09:01:56,947 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 09:01:56,951 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 09:01:56,957 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 09:01:56,964 INFO [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Storage pool null (2) does not supply IOPS capacity, assuming enough capacity
2021-10-07 09:01:56,968 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking pool 2 for storage, totalSize: 30005954347008, usedBytes: 11020345267, usedPct: 3.6727194674609234E-4, disable threshold: 0.85
2021-10-07 09:01:56,968 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Destination pool id: 2
2021-10-07 09:01:57,046 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Pool ID for the volume with ID 1541 is null
2021-10-07 09:01:57,069 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Found storage pool CS_Ceph of type RBD with over-provisioning factor 2
2021-10-07 09:01:57,069 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Total over-provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 09:01:57,069 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Total capacity of the pool CS_Ceph with ID 2 is (54.5805 TB) 60011908694016
2021-10-07 09:01:57,073 DEBUG [c.c.s.StorageManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking pool: 2 for storage allocation , maxSize : (54.5805 TB) 60011908694016, totalAllocatedSize : (0 bytes) 0, askingSize : (0 bytes) 0, allocated disable threshold: 0.85
2021-10-07 09:01:57,073 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) ClusterScopeStoragePoolAllocator returning 1 suitable storage pools
2021-10-07 09:01:57,073 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Trying to find a potenial host and associated storage pools from the suitable host/pool lists for this VM
2021-10-07 09:01:57,075 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking if host: 2 can access any suitable storage pool for volume: ROOT
2021-10-07 09:01:57,078 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Host: 2 can access pool: 2
2021-10-07 09:01:57,081 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Found a potential host id: 2 name: gd-cs01-clst01-compute02 and associated storage pools for this VM
2021-10-07 09:01:57,085 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Returning Deployment Destination: Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1541|ROOT-->Pool(2))]
2021-10-07 09:01:57,085 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Deployment found - P0=VM[SecondaryStorageVm|s-1541-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(2)-Storage(Volume(1541|ROOT-->Pool(2))]
2021-10-07 09:01:57,168 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) VM state transitted from :Starting to Starting with event: OperationRetryvm's original host id: null new host id: 2 host id before state transition: null
2021-10-07 09:01:57,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:57,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) We are allocating VM, increasing the used capacity of this host:2
2021-10-07 09:01:57,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Current Used CPU: 500 , Free CPU:73100 ,Requested CPU: 500
2021-10-07 09:01:57,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Current Used RAM: (1.00 GB) 1073741824 , Free RAM:(60.90 GB) 65395707904 ,Requested RAM: (512.00 MB) 536870912
2021-10-07 09:01:57,205 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) CPU STATS after allocation: for host: 2, old used: 500, old reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used:1000, reserved:0; requested cpu:500,alloc_from_last:false
2021-10-07 09:01:57,206 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) RAM STATS after allocation: for host: 2, old used: (1.00 GB) 1073741824, old reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; new used: (1.50 GB) 1610612736, reserved: (0 bytes) 0; requested mem: (512.00 MB) 536870912,alloc_from_last:false
2021-10-07 09:01:57,214 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Host: 2 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:01:57,214 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:57,222 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:57,222 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) We need to allocate to the last host again, so checking if there is enough reserved capacity
2021-10-07 09:01:57,222 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Reserved CPU: 0 , Requested CPU: 500
2021-10-07 09:01:57,222 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Reserved RAM: (0 bytes) 0 , Requested RAM: (512.00 MB) 536870912
2021-10-07 09:01:57,222 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) STATS: Failed to alloc resource from host: 2 reservedCpu: 0, requested cpu: 500, reservedMem: (0 bytes) 0, requested mem: (512.00 MB) 536870912
2021-10-07 09:01:57,222 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Host does not have enough reserved CPU available, cannot allocate to this host.
2021-10-07 09:01:57,222 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Checking if host: 2 has enough capacity for requested CPU: 500 and requested RAM: (512.00 MB) 536870912 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:01:57,230 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:01:57,230 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Free CPU: 73100 , Requested CPU: 500
2021-10-07 09:01:57,230 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Free RAM: (60.90 GB) 65395707904 , Requested RAM: (512.00 MB) 536870912
2021-10-07 09:01:57,230 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Host has enough CPU and RAM available
2021-10-07 09:01:57,230 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) STATS: Can alloc CPU from host: 2, used: 500, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:01:57,230 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) STATS: Can alloc MEM from host: 2, used: (1.00 GB) 1073741824, reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; requested mem: (512.00 MB) 536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:01:57,307 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Network id=200 is already implemented
2021-10-07 09:01:57,444 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Network id=202 is already implemented
2021-10-07 09:01:57,950 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 09:01:57,952 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 09:01:57,952 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 09:01:57,960 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Run missing VM report. current time: 1633611717960
2021-10-07 09:01:57,967 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1540
2021-10-07 09:01:57,967 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Detected missing VM. host: 2, vm id: 1540(322770c8-c9c2-4b8a-8ee3-0b96fd3226e1), power state: PowerReportMissing, last state update: 2021/10/07T09:01:53.000Z
2021-10-07 09:01:57,967 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) vm id: 1540 - time since last state update(4960ms) has not passed graceful period yet
2021-10-07 09:01:57,971 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1541
2021-10-07 09:01:57,972 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Detected missing VM. host: 2, vm id: 1541(80b1b86e-958a-4be2-96c2-494376f24404), power state: PowerReportMissing, last state update: 2021/10/07T09:01:57.000Z
2021-10-07 09:01:57,972 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) vm id: 1541 - time since last state update(960ms) has not passed graceful period yet
2021-10-07 09:01:57,972 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-13:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 09:01:58,039 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Reserved NIC for s-1541-VM [ipv4:169.254.187.23 netmask:255.255.0.0 gateway:169.254.0.1]
2021-10-07 09:01:58,092 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Network id=201 is already implemented
2021-10-07 09:01:58,173 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Releasing ip address for instance=5907
2021-10-07 09:01:58,249 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Allocated a nic NicProfile[5907-1541-2666993c-031e-4140-bf6c-2e53b5afb577-192.168.204.169-null] for VM[SecondaryStorageVm|s-1541-VM]
2021-10-07 09:01:58,303 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Network id=203 is already implemented
2021-10-07 09:01:58,565 DEBUG [c.c.n.g.StorageNetworkGuru] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Allocated a storage nic NicProfile[5908-1541-2666993c-031e-4140-bf6c-2e53b5afb577-192.168.204.190-null] for VM[SecondaryStorageVm|s-1541-VM]
2021-10-07 09:01:58,632 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) template 3 is already in store:5, type:Image
2021-10-07 09:01:58,652 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) template 3 is already in store:2, type:Primary
2021-10-07 09:01:58,655 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Found template routing-3 in storage pool 2 with VMTemplateStoragePool id: 516
2021-10-07 09:01:58,657 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Acquire lock on VMTemplateStoragePool 516 with timeout 3600 seconds
2021-10-07 09:02:05,355 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-91c1c827) (logid:30907c7f) Found 0 routers to update status.
2021-10-07 09:02:05,359 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-91c1c827) (logid:30907c7f) Found 0 VPC's to update Redundant State.
2021-10-07 09:02:05,364 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-91c1c827) (logid:30907c7f) Found 0 networks to update RvR status.
2021-10-07 09:02:14,178 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 09:02:14,179 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 09:02:14,179 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 09:02:14,184 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-10:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 09:02:17,580 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-86b4e49a) (logid:038a60e3) StorageCollector is running...
2021-10-07 09:02:17,603 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-86b4e49a) (logid:038a60e3) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 09:02:17,609 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-1:ctx-86b4e49a) (logid:038a60e3) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 09:02:17,609 DEBUG [c.c.h.XenServerGuru] (StatsCollector-1:ctx-86b4e49a) (logid:038a60e3) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 09:02:17,887 DEBUG [c.c.a.t.Request] (StatsCollector-1:ctx-86b4e49a) (logid:038a60e3) Seq 1-3253569255798472745: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 09:02:25,166 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-b35a1855) (logid:35a4797f) AutoScaling Monitor is running...
2021-10-07 09:02:35,354 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e3aa6856) (logid:21c6b891) Found 0 routers to update status.
2021-10-07 09:02:35,359 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e3aa6856) (logid:21c6b891) Found 0 VPC's to update Redundant State.
2021-10-07 09:02:35,365 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e3aa6856) (logid:21c6b891) Found 0 networks to update RvR status.
2021-10-07 09:02:39,162 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-9b2e473f) (logid:6a9d2257) HostStatsCollector is running...
2021-10-07 09:02:39,205 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-9b2e473f) (logid:6a9d2257) Seq 1-3253569255798472746: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:02:39,238 DEBUG [c.c.a.t.Request] (StatsCollector-3:ctx-9b2e473f) (logid:6a9d2257) Seq 2-4126986108531640376: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:02:57,947 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 09:02:57,949 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 09:02:57,949 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 09:02:57,955 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Run missing VM report. current time: 1633611777955
2021-10-07 09:02:57,960 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1540
2021-10-07 09:02:57,960 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Detected missing VM. host: 2, vm id: 1540(322770c8-c9c2-4b8a-8ee3-0b96fd3226e1), power state: PowerReportMissing, last state update: 2021/10/07T09:01:53.000Z
2021-10-07 09:02:57,960 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) vm id: 1540 - time since last state update(64955ms) has not passed graceful period yet
2021-10-07 09:02:57,965 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1541
2021-10-07 09:02:57,965 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Detected missing VM. host: 2, vm id: 1541(80b1b86e-958a-4be2-96c2-494376f24404), power state: PowerReportMissing, last state update: 2021/10/07T09:01:57.000Z
2021-10-07 09:02:57,965 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) vm id: 1541 - time since last state update(60955ms) has not passed graceful period yet
2021-10-07 09:02:57,965 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-7:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 09:03:04,597 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:ctx-dd1a3dde) (logid:de3e154e) Backup backup.poll is being called at 2021-10-07 13:03:04 GMT
2021-10-07 09:03:05,124 INFO [c.c.r.ResourceLimitManagerImpl] (ResourceCountChecker-1:ctx-e12ab6e8) (logid:80f7eab0) Started resource counters recalculation periodic task.
2021-10-07 09:03:05,125 DEBUG [c.c.n.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:ctx-a2b46c4d) (logid:9d7c87cc) External devices are not used. Skipping external device usage collection
2021-10-07 09:03:05,320 INFO [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-b42dd8db) (logid:5279cfcb) checking health of usage server
2021-10-07 09:03:05,325 DEBUG [c.c.h.HighAvailabilityManagerImpl] (HA-2:ctx-b42dd8db) (logid:5279cfcb) usage server running? false, heartbeat: null
2021-10-07 09:03:05,329 WARN [c.c.a.AlertManagerImpl] (HA-2:ctx-b42dd8db) (logid:5279cfcb) AlertType:: 13 | dataCenterId:: 0 | podId:: 0 | clusterId:: null | message:: No usage server process running
2021-10-07 09:03:05,340 DEBUG [c.c.a.AlertManagerImpl] (HA-2:ctx-b42dd8db) (logid:5279cfcb) Have already sent: 1 emails for alert type '13' -- skipping send email
2021-10-07 09:03:05,351 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-9e9f94e6) (logid:5568b66c) Found 0 routers to update status.
2021-10-07 09:03:05,352 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-30ec62ec) (logid:0b9f9d45) Found 0 running routers.
2021-10-07 09:03:05,353 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-9e9f94e6) (logid:5568b66c) Found 0 VPC's to update Redundant State.
2021-10-07 09:03:05,355 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-9e9f94e6) (logid:5568b66c) Found 0 networks to update RvR status.
2021-10-07 09:03:05,388 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-b2a9dceb) (logid:3a5f8afd) Found 0 running routers.
2021-10-07 09:03:05,392 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-119fc552) (logid:99910559) Found 0 running routers.
2021-10-07 09:03:05,542 INFO [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-5cd5cbcc) (logid:e5a65863) Found 0 running routers. Fetching, analysing and updating DB for the health checks.
2021-10-07 09:03:05,636 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-4ff77ac3) (logid:56232abc) Snapshot scheduler.poll is being called at 2021-10-07 13:03:05 GMT
2021-10-07 09:03:05,640 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-4ff77ac3) (logid:56232abc) Got 0 snapshots to be executed at 2021-10-07 13:03:05 GMT
2021-10-07 09:03:06,393 INFO [o.a.c.e.o.NetworkOrchestrator] (Network-Scavenger-1:ctx-2276c825) (logid:0c78feee) NetworkGarbageCollector uses '600' seconds for GC interval.
2021-10-07 09:03:09,727 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-41fb7b20) (logid:e4a40a44) Resetting hosts suitable for reconnect
2021-10-07 09:03:09,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-41fb7b20) (logid:e4a40a44) Completed resetting hosts suitable for reconnect
2021-10-07 09:03:09,731 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-41fb7b20) (logid:e4a40a44) Acquiring hosts for clusters already owned by this management server
2021-10-07 09:03:09,732 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-41fb7b20) (logid:e4a40a44) Completed acquiring hosts for clusters already owned by this management server
2021-10-07 09:03:09,732 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-41fb7b20) (logid:e4a40a44) Acquiring hosts for clusters not owned by any management server
2021-10-07 09:03:09,734 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-41fb7b20) (logid:e4a40a44) Completed acquiring hosts for clusters not owned by any management server
2021-10-07 09:03:14,178 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Ping from Routing host 1(gd-cs01-clst01-compute01)
2021-10-07 09:03:14,179 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Process host VM state report from ping process. host: 1
2021-10-07 09:03:14,179 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Process VM state report. host: 1, number of records in report: 0
2021-10-07 09:03:14,183 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-2:null) (logid:) Done with process of VM state report. host: 1
2021-10-07 09:03:15,311 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-e1734433) (logid:29935bbf) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
2021-10-07 09:03:15,339 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-e1734433) (logid:29935bbf) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
2021-10-07 09:03:17,889 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-44ccd438) (logid:a9c82896) StorageCollector is running...
2021-10-07 09:03:17,907 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-44ccd438) (logid:a9c82896) There is no secondary storage VM for secondary storage host CS_Ceph
2021-10-07 09:03:17,912 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StatsCollector-5:ctx-44ccd438) (logid:a9c82896) getCommandHostDelegation: class com.cloud.agent.api.GetStorageStatsCommand
2021-10-07 09:03:17,912 DEBUG [c.c.h.XenServerGuru] (StatsCollector-5:ctx-44ccd438) (logid:a9c82896) We are returning the default host to execute commands because the command is not of Copy type.
2021-10-07 09:03:18,167 DEBUG [c.c.a.t.Request] (StatsCollector-5:ctx-44ccd438) (logid:a9c82896) Seq 1-3253569255798472747: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2021-10-07 09:03:25,171 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-5fda91a1) (logid:9db10228) AutoScaling Monitor is running...
2021-10-07 09:03:34,548 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Running Capacity Checker ...
2021-10-07 09:03:34,548 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) recalculating system capacity
2021-10-07 09:03:34,548 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Executing cpu/ram capacity update
2021-10-07 09:03:34,559 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Found 0 VMs on host 1
2021-10-07 09:03:34,562 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Found 0 VMs are Migrating from host 1
2021-10-07 09:03:34,569 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Found 0 VM, not running on host 1
2021-10-07 09:03:34,575 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) No need to calibrate cpu capacity, host:1 usedCpuCore: 0 reservedCpuCore: 0
2021-10-07 09:03:34,578 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) No need to calibrate cpu capacity, host:1 usedCpu: 0 reservedCpu: 0
2021-10-07 09:03:34,578 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) No need to calibrate memory capacity, host:1 usedMem: (0 bytes) 0 reservedMem: (0 bytes) 0
2021-10-07 09:03:34,619 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Found 2 VMs on host 2
2021-10-07 09:03:34,623 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Found 0 VMs are Migrating from host 2
2021-10-07 09:03:34,637 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Found 0 VM, not running on host 2
2021-10-07 09:03:34,646 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) No need to calibrate cpu capacity, host:2 usedCpuCore: 2 reservedCpuCore: 0
2021-10-07 09:03:34,646 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) No need to calibrate cpu capacity, host:2 usedCpu: 1000 reservedCpu: 0
2021-10-07 09:03:34,646 DEBUG [c.c.c.CapacityManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) No need to calibrate memory capacity, host:2 usedMem: (1.50 GB) 1610612736 reservedMem: (0 bytes) 0
2021-10-07 09:03:34,679 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Done executing cpu/ram capacity update
2021-10-07 09:03:34,680 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Executing storage capacity update
2021-10-07 09:03:34,699 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Found storage pool CS_Ceph of type RBD with overprovisioning factor 2
2021-10-07 09:03:34,699 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Total over provisioned capacity calculated is 2 * (27.2903 TB) 30005954347008
2021-10-07 09:03:34,699 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Total over provisioned capacity of the pool CS_Ceph id: 2 is (54.5805 TB) 60011908694016
2021-10-07 09:03:34,704 DEBUG [c.c.s.StorageManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Successfully set Capacity - (54.5805 TB) 60011908694016 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 2, PodId 1
2021-10-07 09:03:34,704 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Done executing storage capacity update
2021-10-07 09:03:34,704 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Executing capacity updates for public ip and Vlans
2021-10-07 09:03:34,741 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Done capacity updates for public ip and Vlans
2021-10-07 09:03:34,741 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Executing capacity updates for private ip
2021-10-07 09:03:34,754 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Done executing capacity updates for private ip
2021-10-07 09:03:34,754 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Done recalculating system capacity
2021-10-07 09:03:34,826 DEBUG [c.c.a.AlertManagerImpl] (CapacityChecker:ctx-aa0ec391) (logid:913c3639) Done running Capacity Checker ...
2021-10-07 09:03:35,352 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e5f8d536) (logid:0e6aef74) Found 0 routers to update status.
2021-10-07 09:03:35,354 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e5f8d536) (logid:0e6aef74) Found 0 VPC's to update Redundant State.
2021-10-07 09:03:35,356 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-e5f8d536) (logid:0e6aef74) Found 0 networks to update RvR status.
2021-10-07 09:03:35,487 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-433973f0) (logid:823990a7) Checking if any host reservation can be released ...
2021-10-07 09:03:35,507 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-433973f0) (logid:823990a7) Host has no VMs associated, releasing the planner reservation for host 1
2021-10-07 09:03:35,686 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-433973f0) (logid:823990a7) Cannot release reservation, Found 2 VMs Running on host 2
2021-10-07 09:03:35,686 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (HostReservationReleaseChecker:ctx-433973f0) (logid:823990a7) Done running HostReservationReleaseChecker ...
2021-10-07 09:03:39,244 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-561222be) (logid:2c3bddbc) HostStatsCollector is running...
2021-10-07 09:03:39,274 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-561222be) (logid:2c3bddbc) Seq 1-3253569255798472748: Received: { Ans: , MgmtId: 90520741394843, via: 1(gd-cs01-clst01-compute01), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:03:39,295 DEBUG [c.c.a.t.Request] (StatsCollector-4:ctx-561222be) (logid:2c3bddbc) Seq 2-4126986108531640377: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2021-10-07 09:03:48,554 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:ctx-d7ff9905) (logid:5026862f) Task (job-4741) has been pending for 115 seconds
2021-10-07 09:03:48,554 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-0:ctx-d7ff9905) (logid:5026862f) Task (job-4742) has been pending for 111 seconds
2021-10-07 09:03:57,946 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Ping from Routing host 2(gd-cs01-clst01-compute02)
2021-10-07 09:03:57,947 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Process host VM state report from ping process. host: 2
2021-10-07 09:03:57,947 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Process VM state report. host: 2, number of records in report: 0
2021-10-07 09:03:57,957 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Run missing VM report. current time: 1633611837957
2021-10-07 09:03:57,964 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1540
2021-10-07 09:03:57,964 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Detected missing VM. host: 2, vm id: 1540(322770c8-c9c2-4b8a-8ee3-0b96fd3226e1), power state: PowerReportMissing, last state update: 2021/10/07T09:01:53.000Z
2021-10-07 09:03:57,964 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) vm id: 1540 - time since last state update(124957ms) has passed graceful period
2021-10-07 09:03:58,017 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) VM state report is updated. host: 2, vm id: 1540, power state: PowerReportMissing
2021-10-07 09:03:58,030 INFO [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-8:null) (logid:) There is pending job or HA tasks working on the VM. vm id: 1540, postpone power-change report by resetting power-change counters
2021-10-07 09:03:58,083 WARN [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) VM power state update time is null, falling back to update time for vm id: 1541
2021-10-07 09:03:58,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Detected missing VM. host: 2, vm id: 1541(80b1b86e-958a-4be2-96c2-494376f24404), power state: PowerReportMissing, last state update: 2021/10/07T09:01:57.000Z
2021-10-07 09:03:58,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) vm id: 1541 - time since last state update(120957ms) has passed graceful period
2021-10-07 09:03:58,118 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) VM state report is updated. host: 2, vm id: 1541, power state: PowerReportMissing
2021-10-07 09:03:58,125 INFO [c.c.v.VirtualMachineManagerImpl] (AgentManager-Handler-8:null) (logid:) There is pending job or HA tasks working on the VM. vm id: 1541, postpone power-change report by resetting power-change counters
2021-10-07 09:03:58,169 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-8:null) (logid:) Done with process of VM state report. host: 2
2021-10-07 09:04:01,127 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 2-4126986108531640375: Processing: { Ans: , MgmtId: 90520741394843, via: 2, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:04:01,127 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-12:null) (logid:) Seq 2-4126986108531640375: No more commands found
2021-10-07 09:04:01,127 DEBUG [c.c.a.t.Request] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Seq 2-4126986108531640375: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2021-10-07 09:04:01,227 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) releasing lock for VMTemplateStoragePool 516
2021-10-07 09:04:01,383 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Unable to create Vol[1540|vm=1540|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
2021-10-07 09:04:01,384 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:2] is unreachable: Unable to create Vol[1540|vm=1540|ROOT]:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error: Child process (/bin/mount 192.168.204.62:/cloudstack/template/tmpl/1/3 /mnt/ee90a381-7f4b-398d-b91b-cb4c3ddac152 -o nodev,nosuid,noexec) unexpected exit status 32: mount.nfs: Connection timed out
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1491)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1566)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1160)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-07 09:04:01,461 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Cleaning up resources for the vm VM[ConsoleProxy|v-1540-VM] in Starting state
2021-10-07 09:04:01,495 DEBUG [c.c.a.t.Request] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Seq 2-4126986108531640378: Sending { Cmd , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.134.219","forceStop":"false","volumesToDisconnect":[],"vmName":"v-1540-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:04:03,843 INFO [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Unable to acquire lock on VMTemplateStoragePool 516
2021-10-07 09:04:03,878 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Cleaning up resources for the vm VM[SecondaryStorageVm|s-1541-VM] in Starting state
2021-10-07 09:04:03,896 DEBUG [c.c.a.t.Request] (Work-Job-Executor-140:ctx-dbbcfc2a job-3440/job-4742 ctx-d39c6c37) (logid:e381c0f1) Seq 2-4126986108531640379: Sending { Cmd , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.187.23","forceStop":"false","volumesToDisconnect":[],"vmName":"s-1541-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:04:04,770 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) (logid:) Seq 2-4126986108531640378: Processing: { Ans: , MgmtId: 90520741394843, via: 2, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2021-10-07 09:04:04,771 DEBUG [c.c.a.t.Request] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Seq 2-4126986108531640378: Received: { Ans: , MgmtId: 90520741394843, via: 2(gd-cs01-clst01-compute02), Ver: v1, Flags: 10, { StopAnswer } }
2021-10-07 09:04:04,852 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Released nic: NicProfile[5903-1540-null-null-null]
2021-10-07 09:04:04,969 DEBUG [c.c.d.d.DataCenterIpAddressDaoImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Releasing ip address for reservationId=31c69384-bc59-40af-9e1d-e0fe73a62a59, instance=5904
2021-10-07 09:04:04,971 DEBUG [c.c.n.g.PodBasedNetworkGuru] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Released nic: NicProfile[5904-1540-null-null-null]
2021-10-07 09:04:05,023 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Successfully released network resources for the vm VM[ConsoleProxy|v-1540-VM]
2021-10-07 09:04:05,023 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Successfully cleaned up resources for the VM VM[ConsoleProxy|v-1540-VM] in Starting state
2021-10-07 09:04:05,029 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) DeploymentPlanner allocation algorithm: null
2021-10-07 09:04:05,030 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB) 1073741824
2021-10-07 09:04:05,030 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Is ROOT volume READY (pool already allocated)?: No
2021-10-07 09:04:05,033 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Deploy avoids pods: null, clusters: null, hosts: [2]
2021-10-07 09:04:05,041 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Searching all possible resources under this Zone: 1
2021-10-07 09:04:05,046 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 09:04:05,088 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking resources in Cluster: 1 under Pod: 1
2021-10-07 09:04:05,096 INFO [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Guest VM is requested with Custom[UEFI] Boot Type false
2021-10-07 09:04:05,096 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for hosts in dc: 1 pod:1 cluster:1
2021-10-07 09:04:05,109 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) FirstFitAllocator has 2 hosts to check for allocation: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 09:04:05,142 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Found 2 hosts for allocation after prioritization: [Host[-2-Routing], Host[-1-Routing]]
2021-10-07 09:04:05,142 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Looking for speed=500Mhz, Ram=1024 MB
2021-10-07 09:04:05,142 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host name: gd-cs01-clst01-compute02, hostId: 2 is in avoid set, skipping this and trying other available hosts
2021-10-07 09:04:05,165 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host: 1 has cpu capability (cpu:32, speed:2300) to support requested CPU: 1 and requested speed: 500
2021-10-07 09:04:05,165 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Checking if host: 1 has enough capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 , cpuOverprovisioningFactor: 1.0
2021-10-07 09:04:05,175 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:04:05,175 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Free CPU: 73600 , Requested CPU: 500
2021-10-07 09:04:05,175 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Free RAM: (61.89 GB) 66452602880 , Requested RAM: (1.00 GB) 1073741824
2021-10-07 09:04:05,175 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host has enough CPU and RAM available
2021-10-07 09:04:05,175 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc CPU from host: 1, used: 0, reserved: 0, actual total: 73600, total with overprovisioning: 73600; requested cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2021-10-07 09:04:05,175 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (0 bytes) 0, total: (61.89 GB) 66452602880; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?: false , considerReservedCapacity?: true
2021-10-07 09:04:05,175 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Found a suitable host, adding to list: 1
2021-10-07 09:04:05,176 DEBUG [c.c.a.m.a.i.FirstFitAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67 FirstFitRoutingAllocator) (logid:bcef1bbe) Host Allocator returning 1 suitable hosts
2021-10-07 09:04:05,185 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking suitable pools for volume (Id, Type): (1540,ROOT)
2021-10-07 09:04:05,185 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) We need to allocate new storagepool for this volume
2021-10-07 09:04:05,188 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Calling StoragePoolAllocators to find suitable pools
2021-10-07 09:04:05,198 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) System VMs will use shared storage for zone id=1
2021-10-07 09:04:05,198 DEBUG [o.a.c.s.a.LocalStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2021-10-07 09:04:05,198 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator looking for storage pool
2021-10-07 09:04:05,198 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Looking for pools in dc: 1 pod:1 cluster:1. Disabled pools will be ignored.
2021-10-07 09:04:05,201 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Found pools matching tags: [Pool[2|RBD]]
2021-10-07 09:04:05,207 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Checking if storage pool is suitable, name: null ,poolId: 2
2021-10-07 09:04:05,207 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) StoragePool is in avoid set, skipping this pool
2021-10-07 09:04:05,207 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) ClusterScopeStoragePoolAllocator returning 0 suitable storage pools
2021-10-07 09:04:05,207 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) ZoneWideStoragePoolAllocator to find storage pool
2021-10-07 09:04:05,215 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) No suitable pools found for volume: Vol[1540|vm=1540|ROOT] under cluster: 1
2021-10-07 09:04:05,215 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) No suitable pools found
2021-10-07 09:04:05,215 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) No suitable storagePools found under this Cluster: 1
2021-10-07 09:04:05,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2021-10-07 09:04:05,228 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Searching all possible resources under this Zone: 1
2021-10-07 09:04:05,233 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Listing clusters in order of aggregate capacity, that have (at least one host with) enough CPU and RAM capacity under this Zone: 1
2021-10-07 09:04:05,240 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Removing from the clusterId list these clusters from avoid set: [1]
2021-10-07 09:04:05,245 DEBUG [c.c.d.FirstFitPlanner] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) No clusters found after removing disabled clusters and clusters in avoid list, returning.
2021-10-07 09:04:05,347 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: 2
2021-10-07 09:04:05,352 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f423f7b8) (logid:c88c7a6e) Found 0 routers to update status.
2021-10-07 09:04:05,355 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f423f7b8) (logid:c88c7a6e) Found 0 VPC's to update Redundant State.
2021-10-07 09:04:05,357 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f423f7b8) (logid:c88c7a6e) Found 0 networks to update RvR status.
2021-10-07 09:04:05,383 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Hosts's actual total CPU: 73600 and CPU after applying overprovisioning: 73600
2021-10-07 09:04:05,384 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Hosts's actual total RAM: (61.90 GB) 66469449728 and RAM after applying overprovisioning: (61.90 GB) 66469449728
2021-10-07 09:04:05,384 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) release cpu from host: 2, old used: 1000,reserved: 0, actual total: 73600, total with overprovisioning: 73600; new used: 500,reserved:0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 09:04:05,384 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) release mem from host: 2, old used: (1.50 GB) 1610612736,reserved: (0 bytes) 0, total: (61.90 GB) 66469449728; new used: (512.00 MB) 536870912,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse
2021-10-07 09:04:05,449 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1540-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 09:04:05,450 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741 ctx-5c2c1e67) (logid:bcef1bbe) Rethrow exception com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1540-VM]Scope=interface com.cloud.dc.DataCenter; id=1
2021-10-07 09:04:05,450 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741) (logid:bcef1bbe) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 1540, job origin: 3439
2021-10-07 09:04:05,450 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-138:ctx-5acfce4c job-3439/job-4741) (logid:bcef1bbe) Unable to complete AsyncJobVO {id:4741, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGBHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90520741394843, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Oct 07 09:01:50 EDT 2021, removed: null}, job origin:3439
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-1540-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1119)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5502)
at jdk.internal.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5669)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment