Skip to content

Instantly share code, notes, and snippets.

View tanabarr's full-sized avatar
💭
working on daos.io

Tom Nabarro tanabarr

💭
working on daos.io
10 followers · 5 following
View GitHub Profile
@tanabarr
tanabarr / prepare_scm
Last active September 12, 2019 14:00
Using daos_server to prepare local SCM/DCPM modules
[root@wolf-72 daos_m]# daos_server storage prepare --debug
bash: daos_server: command not found
[root@wolf-72 daos_m]# source scons_local/utils/setup_local.sh
/home/tanabarr/projects/daos_m
Build vars file found: ./.build_vars.sh
OLD_PATH is /usr/lib64/qt-3.3/bin /usr/local/bin /usr/bin /usr/local/sbin /usr/sbin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin /home/tanabarr/bin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin
[root@wolf-72 daos_m]# daos_server storage prepare --debug
DEBUG 13:54:28.030669 ctl_storage.go:98: spdk setup with _NRHUGE=1024
DEBUG 13:54:28.033110 ctl_storage.go:98: spdk setup with _TARGET_USER=root
Preparing locally-attached SCM...
@tanabarr
tanabarr / basic_daos_deployment_workflow
Last active August 28, 2019 13:09
Deployment workflow for DAOS using the control plane tools
## Prepare devices for use
[root@<hostname> tanabarr]# cd projects/daos_m/
[root@<hostname> daos_m]# source scons_local/utils/setup_local.sh
/home/tanabarr/projects/daos_m
Build vars file found: ./.build_vars.sh
OLD_PATH is /usr/lib64/qt-3.3/bin /usr/local/bin /usr/bin /usr/local/sbin /usr/sbin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin /home/tanabarr/bin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin
[root@<hostname> daos_m]# daos_server storage prep-scm
Memory allocation goals for SCM will be changed and namespaces modified, this will be a destructive operation. ensure namespaces are unmounted and SCM is otherwise unused.
Are you sure you want to continue? (yes/no)
@tanabarr
tanabarr / cp_pool_create_then_format
Last active July 23, 2019 13:18
Creating DAOS pools from the Control Plane, then formatting to wipe NVMe
## Prepare devices for use
[root@boro-45 daos_m]# daos_server -i storage prep-nvme
2019/07/04 06:43:34 storage_nvme.go:97: debug: spdk setup with _NRHUGE=1024
2019/07/04 06:43:34 storage_nvme.go:101: debug: spdk setup with _TARGET_USER=root
## Apply configuration file changes (pciaddress already in server config example)
--- a/utils/config/daos.yml
+++ b/utils/config/daos.yml
@tanabarr
tanabarr / verify_blob_removed
Created July 4, 2019 15:37
Verify SPDK blobs are removed through Control Plane "storage format" command
-model: 'INTEL SSDPEDMD016T4 '
serial: 'CVFT6010002F1P6DGN '
pciaddr: 0000:81:00.0
fwrev: 8DV10171
namespaces: id: 1
capacity: 1600
0000:81:00.0 == Nvme0n1 (note names do not necessarily correlate with those reported by nvme-cli tool)
dev_list: ["0000:81:00.0"] # generate regular nvme.conf
@tanabarr
tanabarr / pool_nvme_format
Last active July 4, 2019 14:43
Creating DAOS pools then formatting to wipe NVMe
## Verify devices are working correctly
[root@wolf-72 daos_m]# readlink /sys/block/nvme0n1/device/device
../../../0000:87:00.0
[root@wolf-72 daos_m]# readlink /sys/block/nvme1n1/device/device
../../../0000:da:00.0
[root@wolf-72 daos_m]# readlink /sys/block/nvme2n1/device/device
../../../0000:81:00.0
[root@wolf-72 daos_m]# nvme format /dev/nvme0
^[[ASuccess formatting namespace:ffffffff
@tanabarr
tanabarr / max_pool_size
Last active July 4, 2019 13:39
daos running on a single server with multiple nvme devices can't create a pool with the cumulative available size
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg create --size=50G --nvme=1900G
failed to create pool: -1007
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg create --size=50G --nvme=1500G
failed to create pool: -1007
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg create --size=50G --nvme=300G
b697a5ed-c0ee-4214-ae69-e336ee2bca92 0
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg query --pool=b697a5ed-c0ee-4214-ae69-e336ee2bca92 --svc=0
Pool b697a5ed-c0ee-4214-ae69-e336ee2bca92, ntarget=1, disabled=0
Pool space info:
- Target(VOS) count:1
@tanabarr
tanabarr / nvme_devs
Created July 4, 2019 11:11
when white list is applied to spdk/scripts/setup.sh, unexpected nvme* entries removed from /dev/
[root@wolf-72 daos_m]# daos_server storage prep-nvme --reset
[root@wolf-72 daos_m]# nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 PHKS7335008X375AGN INTEL SSDPED1K375GA 1 375.08 GB / 375.08 GB 512 B + 0 B E2010324
/dev/nvme1n1 CVFT5392000G1P6DGN INTEL SSDPEDMD016T4 1 1.60 TB / 1.60 TB 512 B + 0 B 8DV10171
/dev/nvme2n1 CVFT6010002F1P6DGN INTEL SSDPEDMD016T4 1 1.60 TB / 1.60 TB 512 B + 0 B 8DV10171
[root@wolf-72 daos_m]# ls -lah /dev/nvm*
crw------- 1 root root 243, 0 Jul 3 17:54 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Jul 3 17:59 /dev/nvme0n1
@tanabarr
tanabarr / daos_shell_pool_create_destroy
Created July 2, 2019 15:38
creating and destroying DAOS pool through the control plane management tool
CLIENT:
[user@wolf-45 daos_tom]$ daos_shell -o $(pwd)/utils/config/daos.yml pool create --help
Usage:
daos_shell [OPTIONS] pool create [create-OPTIONS]
Application Options:
-l, --host-list= comma separated list of addresses <ipv4addr/hostname:port>
-f, --host-file= path of hostfile specifying list of addresses <ipv4addr/hostname:port>, if specified
takes preference over HostList
@tanabarr
tanabarr / pool_create_destroy
Last active June 27, 2019 22:58
from the control plane
06/27-22:51:04.99 boro-45 DAOS[184849/184849] fi INFO src/gurt/fault_inject.c:496 d_fault_inject_init() No config file, fault injection is OFF.
06/27-22:51:04.99 boro-45 DAOS[184849/184849] fi DBUG src/gurt/fault_inject.c:165 fault_attr_set() new fault id: 1 added.
06/27-22:51:05.06 boro-45 DAOS[184849/184849] server INFO src/iosrv/init.c:295 set_abt_max_num_xstreams() Setting ABT_MAX_NUM_XSTREAMS to 5
06/27-22:51:05.06 boro-45 DAOS[184849/184849] server INFO src/iosrv/init.c:368 server_init() Module interface successfully initialized
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:58 data_init() initializing crt_gdata...
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:94 data_init() set the global timeout value as 30 second.
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:114 data_init() CRT_CREDIT_EP_CTX set as 32 for flow control.
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:136 data_init(
@tanabarr
tanabarr / create_pool
Created June 10, 2019 16:15
06/10-15:55:44.58 boro-45 DAOS[132901/132907] mgmt DBUG src/mgmt/srv.c:302 process_create_pool_request() Received request to create pool
06/10-15:55:44.58 boro-45 DAOS[132901/132907] rpc DBUG src/cart/crt_register.c:330 crt_opc_lookup() looking up opcode: 0xff000000
06/10-15:55:44.58 boro-45 DAOS[132901/132907] rpc DBUG src/cart/crt_register.c:330 crt_opc_lookup() looking up opcode: 0x1010005
06/10-15:55:44.59 boro-45 DAOS[132901/132938] mgmt DBUG src/mgmt/srv_target.c:322 tgt_vos_preallocate() 606f1b7a: creating vos file /mnt/daos/NEWBORNS/606f1b7a-557f-0000-a01b-0a50557f0000/vos-0
06/10-15:55:44.68 boro-45 DAOS[132901/132908] vos DBUG src/vos/vos_pool.c:230 vos_pool_create() Pool Path: /mnt/daos/NEWBORNS/606f1b7a-557f-0000-a01b-0a50557f0000/vos-0, size: 0:1073741824, UUID: 606f1b7a
06/10-15:55:44.70 boro-45 DAOS[132901/132908] vos DBUG src/vos/vos_pool.c:312 vos_pool_create() Creating blob for xs:0x7f55480cd750 pool:606f1b7a
06/10-15:55:44.70 boro-45 DAOS[132901/132908] bio DBUG src/bio/smd/smd_pool.