Skip to content

Instantly share code, notes, and snippets.

@rhb2
rhb2 / errors
Created February 24, 2021 00:06
TASK [common_linux_packages : install yum-plugin-priorities] *****************************************************************
fatal: [10.88.88.133]: FAILED! => {"ansible_facts": {"pkg_mgr": "yum"}, "changed": false, "msg": "https://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: [Errno 12] Timeout on https://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')\nTrying other mirror.\nhttps://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: [Errno 12] Timeout on https://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: (28, 'Connection time

smaug-ansible with vmware

Testing smaug-ansible changes can be done on a laptop. This is a guide to do so using vmware.

Prerequisites

The following must be installed on the host machine running the VMs:

  • Python 3.x
  • pip (Python package intaller)
32edcc12-778e-46e9-a864-e570f9d723f0=# select count(*) from evacuateobjects where skipped_reason is not null;
count
-------
3855
(1 row)
3855 total failures
Of those failures, there were sustained on the following agents:
32edcc12-778e-46e9-a864-e570f9d723f0=# select distinct assignment_id,dest_shark,skipped_reason from (select assignment_id,dest_shark,skipped_reason from evacuateobjects where skipped_reason is NOT NULL) as foo order by dest_shark;
assignment_id | dest_shark | skipped_reason
--------------------------------------+----------------------------------+-------------------------
d2c91df2-f6b1-4fe2-a007-5dc34cc9245d | 1023.stor.eu-central.scloud.host | destination_unreachable
c5291fbf-5218-41e5-b8bf-991e538a52a6 | 1032.stor.eu-central.scloud.host | source_other_error
db6e4ab4-9892-4823-8b48-085544083e9f | 1118.stor.eu-central.scloud.host | source_other_error
eef43f7c-86d8-44db-8ac1-334948d37c99 | 1136.stor.eu-central.scloud.host | agent_fs_error
20d15f1a-7b55-466a-845d-c62f7ade0537 | 1138.stor.eu-central.scloud.host | {http_status_code:404}
82ae90d7-06cd-4890-874c-93bb324621a4 | 1231.stor.eu-central.scloud.host | agent_fs_error
8de5f6cf-339e-482a-8c16-ea3
dest_shark | assignment_id | source
-------------------------------+--------------------------------------+------------------------------
1364.stor.us-east.scloud.host | efe01c99-7076-4ab1-b3b0-37677e5785c9 | 55.stor.us-east.scloud.host
1405.stor.us-east.scloud.host | 8dd433bd-2d1a-4694-b00d-c241667e58dc | 205.stor.us-east.scloud.host
1580.stor.us-east.scloud.host | 659a2036-9891-496a-b9f0-deafbfbb193e | 145.stor.us-east.scloud.host
1206.stor.us-east.scloud.host | 03a33f61-afb7-4bfa-b78e-d76e7a0907e2 | 91.stor.us-east.scloud.host
1409.stor.us-east.scloud.host | 340ac2d5-e297-4c24-9ded-901f90079bfd | 74.stor.us-east.scloud.host
1465.stor.us-east.scloud.host | 97c6c792-a17d-4015-99fb-c38852149b8f | 205.stor.us-east.scloud.host
1440.stor.us-east.scloud.host | 7f304a74-4fc3-44bd-8ee3-61dfecf2e286 | 205.stor.us-east.scloud.host
1002.stor.us-east.scloud.host | 0f572419-26c3-48df-a138-f353d2153086 | 205.stor.us-east.scloud.host
@rhb2
rhb2 / ScavengerHunt.md
Last active January 5, 2021 23:58
Scavenger Hunt

From the headnode, login to the rebalancer zone

[rbogart@headnode (ap-northeast-1a) ~]$ manta-login rebalancer
0:   rebalancer        1 90ed0873-1c3a-4a25-84fe-53cb0a5e5443 10.92.72.42     
1:   rebalancer        1 d76fdeca-75ed-4451-8046-eda32382789c 10.92.64.42     
Choose a number: 0
[Connected to zone '90ed0873-1c3a-4a25-84fe-53cb0a5e5443' pts/8]
Last login: Tue Jan  5 19:52:23 on pts/6
 =  J O Y E N T  =
Component File Reviewer Comments
admin admin.go
admin operation-mode.go
admin operation-recovery.go
admin operation-status.go
chunk chunk.go
chunk operation-delete.go
chunk operation-delete_test.go
chunk operation-get.go
sequenceDiagram
    participant User
    participant Gage
    participant Object API
    participant Chunk API
    participant FoundationDB
    User-->>Gage: S3 GET
    Gage->>Object API: SWebDAV GET
 Object API->>FoundationDB: Get "key" = "value"
[root@headnode (rbogart-dc) ~]# manta-hotpatch-rebalancer-agent deploy -a 4518269c-070a-454b-b597-8033c6798471
This will do the following:
- Import rebalancer-agent image 4518269c-070a-454b-b597-8033c6798471
(master-20200616T183110Z-g82b8008) from updates.joyent.com.
- Hotpatch rebalancer-agent image 4518269c-070a-454b-b597-8033c6798471
(master-20200616T183110Z-g82b8008) on all 4 storage instances in this DC
Would you like to hotpatch? [y/N] y
Trace logging to "/var/tmp/manta-hotpatch-rebalancer-agent.20200617T223409Z.deploy.log"
Importing image 4518269c-070a-454b-b597-8033c6798471 from updates.joyent.com
#!/usr/sbin/dtrace -s
#pragma D option quiet
pid$target::malloc:entry
{
self->size = arg0;
}
pid$target::malloc:return
/self->size/