Skip to content

Instantly share code, notes, and snippets.

View tanabarr's full-sized avatar
💭
working on daos.io

Tom Nabarro tanabarr

💭
working on daos.io
View GitHub Profile
[root@boro-45 daos_m]#]umount /mnt/daos; rm -rf /mnt/daos
[root@boro-45 daos_m]# opt/ompi/bin/orterun -np 1 -H localhost --allow-run-as-root install/bin/daos_server -c 1 -o /home/tanabarr/projects/daos_m/utils/config/examples/daos_server_sockets.yml
2019/03/30 19:45:27 config.go:84: debug: DAOS config read from /home/tanabarr/projects/daos_m/utils/config/examples/daos_server_sockets.yml
2019/03/30 19:45:27 main.go:76: debug: Switching control log level to DEBUG
boro-45.boro.hpdd.intel.com 2019/03/30 19:45:27 config.go:111: debug: Warning: active config could not be saved (open /home/tanabarr/projects/daos_m/utils/config/examples/.daos_server.active.yml: permission denied)
boro-45.boro.hpdd.intel.com 2019/03/30 19:45:27 config.go:120: debug: Active config saved to /tmp/.daos_server.active.yml (read-only)
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: spdk -c 0x1 --file-prefix=spdk1738391849 --base-virtaddr=0x200000000000 --proc-type=auto ]
EAL: Detected 72 lcore(s)
EAL: Auto-d
@tanabarr
tanabarr / go-spdk_unit_tests
Created April 8, 2019 23:15
running go-spdk
running go-spdk unit tests
sudo ~/projects/daos_m/install/bin/daos_server storage prep-nvme
cd /usr/lib
ln -sn /home/tanabarr/projects/daos_m/opt/spdk/lib/libspdk.so .
go get -u -v github.com/go-task/task/
go get -u -v github.com/go-task/task/cmd/task
cd ~/go/src/github.com/daos-stack/go-spdk
task main-task
cd spdk
LD_LIBRARY_PATH=".;/home/tanabarr/projects/daos_m/opt/spdk/lib" go test -v
[root@boro-45 cli]# ./blobcli -b Nvme0n1 -l bdevs
Starting SPDK v18.07-pre / DPDK 18.02.0 initialization...
[ DPDK EAL parameters: blobcli -c 0x1 --file-prefix=spdk_pid10818 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Multi-process socket /var/run/.spdk_pid10818_unix
EAL: Probing VFIO support...
app.c: 530:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 718:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
reactor.c: 492:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
06/10-15:55:44.58 boro-45 DAOS[132901/132907] mgmt DBUG src/mgmt/srv.c:302 process_create_pool_request() Received request to create pool
06/10-15:55:44.58 boro-45 DAOS[132901/132907] rpc DBUG src/cart/crt_register.c:330 crt_opc_lookup() looking up opcode: 0xff000000
06/10-15:55:44.58 boro-45 DAOS[132901/132907] rpc DBUG src/cart/crt_register.c:330 crt_opc_lookup() looking up opcode: 0x1010005
06/10-15:55:44.59 boro-45 DAOS[132901/132938] mgmt DBUG src/mgmt/srv_target.c:322 tgt_vos_preallocate() 606f1b7a: creating vos file /mnt/daos/NEWBORNS/606f1b7a-557f-0000-a01b-0a50557f0000/vos-0
06/10-15:55:44.68 boro-45 DAOS[132901/132908] vos DBUG src/vos/vos_pool.c:230 vos_pool_create() Pool Path: /mnt/daos/NEWBORNS/606f1b7a-557f-0000-a01b-0a50557f0000/vos-0, size: 0:1073741824, UUID: 606f1b7a
06/10-15:55:44.70 boro-45 DAOS[132901/132908] vos DBUG src/vos/vos_pool.c:312 vos_pool_create() Creating blob for xs:0x7f55480cd750 pool:606f1b7a
06/10-15:55:44.70 boro-45 DAOS[132901/132908] bio DBUG src/bio/smd/smd_pool.
@tanabarr
tanabarr / pool_create_destroy
Last active June 27, 2019 22:58
from the control plane
06/27-22:51:04.99 boro-45 DAOS[184849/184849] fi INFO src/gurt/fault_inject.c:496 d_fault_inject_init() No config file, fault injection is OFF.
06/27-22:51:04.99 boro-45 DAOS[184849/184849] fi DBUG src/gurt/fault_inject.c:165 fault_attr_set() new fault id: 1 added.
06/27-22:51:05.06 boro-45 DAOS[184849/184849] server INFO src/iosrv/init.c:295 set_abt_max_num_xstreams() Setting ABT_MAX_NUM_XSTREAMS to 5
06/27-22:51:05.06 boro-45 DAOS[184849/184849] server INFO src/iosrv/init.c:368 server_init() Module interface successfully initialized
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:58 data_init() initializing crt_gdata...
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:94 data_init() set the global timeout value as 30 second.
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:114 data_init() CRT_CREDIT_EP_CTX set as 32 for flow control.
06/27-22:51:05.06 boro-45 DAOS[184849/184849] crt DBUG src/cart/crt_init.c:136 data_init(
@tanabarr
tanabarr / daos_shell_pool_create_destroy
Created July 2, 2019 15:38
creating and destroying DAOS pool through the control plane management tool
CLIENT:
[user@wolf-45 daos_tom]$ daos_shell -o $(pwd)/utils/config/daos.yml pool create --help
Usage:
daos_shell [OPTIONS] pool create [create-OPTIONS]
Application Options:
-l, --host-list= comma separated list of addresses <ipv4addr/hostname:port>
-f, --host-file= path of hostfile specifying list of addresses <ipv4addr/hostname:port>, if specified
takes preference over HostList
@tanabarr
tanabarr / nvme_devs
Created July 4, 2019 11:11
when white list is applied to spdk/scripts/setup.sh, unexpected nvme* entries removed from /dev/
[root@wolf-72 daos_m]# daos_server storage prep-nvme --reset
[root@wolf-72 daos_m]# nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 PHKS7335008X375AGN INTEL SSDPED1K375GA 1 375.08 GB / 375.08 GB 512 B + 0 B E2010324
/dev/nvme1n1 CVFT5392000G1P6DGN INTEL SSDPEDMD016T4 1 1.60 TB / 1.60 TB 512 B + 0 B 8DV10171
/dev/nvme2n1 CVFT6010002F1P6DGN INTEL SSDPEDMD016T4 1 1.60 TB / 1.60 TB 512 B + 0 B 8DV10171
[root@wolf-72 daos_m]# ls -lah /dev/nvm*
crw------- 1 root root 243, 0 Jul 3 17:54 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Jul 3 17:59 /dev/nvme0n1
@tanabarr
tanabarr / max_pool_size
Last active July 4, 2019 13:39
daos running on a single server with multiple nvme devices can't create a pool with the cumulative available size
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg create --size=50G --nvme=1900G
failed to create pool: -1007
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg create --size=50G --nvme=1500G
failed to create pool: -1007
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg create --size=50G --nvme=300G
b697a5ed-c0ee-4214-ae69-e336ee2bca92 0
[root@wolf-72 daos_m]# /home/tanabarr/projects/daos_m/install/bin/orterun -np 1 --ompi-server file:/tmp/urifile --enable-recovery --allow-run-as-root install/bin/dmg query --pool=b697a5ed-c0ee-4214-ae69-e336ee2bca92 --svc=0
Pool b697a5ed-c0ee-4214-ae69-e336ee2bca92, ntarget=1, disabled=0
Pool space info:
- Target(VOS) count:1
@tanabarr
tanabarr / pool_nvme_format
Last active July 4, 2019 14:43
Creating DAOS pools then formatting to wipe NVMe
## Verify devices are working correctly
[root@wolf-72 daos_m]# readlink /sys/block/nvme0n1/device/device
../../../0000:87:00.0
[root@wolf-72 daos_m]# readlink /sys/block/nvme1n1/device/device
../../../0000:da:00.0
[root@wolf-72 daos_m]# readlink /sys/block/nvme2n1/device/device
../../../0000:81:00.0
[root@wolf-72 daos_m]# nvme format /dev/nvme0
^[[ASuccess formatting namespace:ffffffff
@tanabarr
tanabarr / verify_blob_removed
Created July 4, 2019 15:37
Verify SPDK blobs are removed through Control Plane "storage format" command
-model: 'INTEL SSDPEDMD016T4 '
serial: 'CVFT6010002F1P6DGN '
pciaddr: 0000:81:00.0
fwrev: 8DV10171
namespaces: id: 1
capacity: 1600
0000:81:00.0 == Nvme0n1 (note names do not necessarily correlate with those reported by nvme-cli tool)
dev_list: ["0000:81:00.0"] # generate regular nvme.conf