Skip to content

Instantly share code, notes, and snippets.

View tanabarr's full-sized avatar
💭
working on daos.io

Tom Nabarro tanabarr

💭
working on daos.io
View GitHub Profile
@tanabarr
tanabarr / isa-l undefined
Created February 1, 2021 09:29
dpdk build error
Found ninja-1.7.2 at /usr/bin/ninja
+ ninja -C build
ninja: Entering directory `build'
[1415/1451] Linking target drivers/librte_compress_isal.so.21.0.
FAILED: drivers/librte_compress_isal.so.21.0
cc -o drivers/librte_compress_isal.so.21.0 'drivers/drivers@@rte_compress_isal@sha/meson-generated_.._rte_compress_isal.pmd.c.o' 'drivers/drivers@@tmp_rte_compress_isal@sta/compress_isal_isal_compress_pmd.c.o' 'drivers/drivers@@tmp_rte_compress_isal@sta/compress_isal_isal_compress_pmd_ops.c.o' -Wl,--no-undefined -Wl,--as-needed -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,librte_compress_isal.so.21 -Wl,--no-as-needed -pthread -lm -ldl -lnuma lib/librte_compressdev.so.21.0 lib/librte_eal.so.21.0 lib/librte_kvargs.so.21.0 lib/librte_telemetry.so.21.0 lib/librte_mbuf.so.21.0 lib/librte_mempool.so.21.0 lib/librte_ring.so.21.0 drivers/librte_bus_vdev.so.21.0 -Wl,--end-group -Wl,--version-script=/home/tanabarr/projects/daos-dpdk-20_11/_topdir/BUILD/dpdk-20.11/drivers/compress/isal/version.map '-Wl,-rpath,$ORIGIN/.
@tanabarr
tanabarr / create_pool_with_acls
Created September 26, 2019 14:03
Create a DAOS pool with permissions read from a supplied ACL file.
[root@wolf-71 daos_m]# daos_shell pool create -a /tmp/acls -s 1G -n 10G -i
Active connections: [localhost:10001]
Creating DAOS pool with 1GB SCM and 10GB NvMe storage (0.100 ratio)
Pool-create command SUCCEEDED: UUID: 58191829-6d76-4a37-b7cd-ded5604bfbde, Service replicas: 0
[root@wolf-71 daos_m]# cat /tmp/acls
A::root@:rw
A::OWNER@:rw
A:g:readers@:r
L:f:baduser@:rw
U:f:EVERYONE@:rw
@tanabarr
tanabarr / daos_test_wolf_psm2
Last active September 26, 2019 13:05
Commands and config file needed to run daos_test to validate installation using psm2 provider.
Build from commit 2e4fcb939e83f8d65b1cc1667f2a195f8cf1131c
Verify provider is available:
daos_m/install/bin/fi_info | grep psm
(should show some output, if not then rebuild daos_m/_build.external/ofi)
need to run client and server on different hosts to be able to reconnect without restarting server (required in daos_test)
CONFIG:
@tanabarr
tanabarr / daos_test_wolf_sockets
Last active October 9, 2019 10:01
Commands and config file needed to run daos_test to validate installation using sockets provider.
Build from commit 2e4fcb939e83f8d65b1cc1667f2a195f8cf1131c
CONFIG:
--- a/utils/config/examples/daos_server_sockets.yml
+++ b/utils/config/examples/daos_server_sockets.yml
@@ -3,13 +3,15 @@
name: daos_server # map to -g daos_server
port: 10001 # mgmt port
provider: ofi+sockets # map to CRT_PHY_ADDR_STR=ofi+sockets
@tanabarr
tanabarr / daos_test_psm2
Last active October 31, 2019 09:26
Run DAOS pool test suite with OFI PSM2 provider, DCPM modules and NVMe SSDs.
CONFIG:
name: daos_server # map to -g daos_server
port: 10001 # mgmt port
provider: ofi+psm2 # map to CRT_PHY_ADDR_STR=ofi+psm2
nr_hugepages: 4096
control_log_mask: DEBUG
control_log_file: /tmp/daos_control.log
transport_config:
@tanabarr
tanabarr / run_daos_docker_troubleshooting
Created September 24, 2019 15:49
Probe UIO and map hugepages when running DAOS in docker container
daos@wolf-130:~/docker/daos$ ls /sys/bus/pci/drivers/uio_pci_generic
ls: cannot access /sys/bus/pci/drivers/uio_pci_generic: No such file or director
daos@wolf-130:~/docker/daos$ sudo modprobe uio_pci_generic
daos@wolf-130:~/docker/daos $ ls /sys/bus/pci/drivers/uio_pci_generic
0000:00:04.0 0000:00:04.3 0000:00:04.6 0000:5f:00.0 0000:80:04.2 0000:80:04.5 0000:81:00.0 module uevent
0000:00:04.1 0000:00:04.4 0000:00:04.7 0000:80:04.0 0000:80:04.3 0000:80:04.6 0000:da:00.0 new_id unbind
0000:00:04.2 0000:00:04.5 0000:5e:00.0 0000:80:04.1 0000:80:04.4 0000:80:04.7 bind remove_id
@tanabarr
tanabarr / pool_create_no_client_config
Created September 23, 2019 15:12
Creating Pool without specifying a client side config file in command
SERVER:
--- a/utils/config/examples/daos_server_sockets.yml
+++ b/utils/config/examples/daos_server_sockets.yml
@@@ -6,10 -6,8 +6,12 @@@ provider: ofi+sockets # map to CR
socket_dir: /tmp/daos_sockets
nr_hugepages: 4096
control_log_mask: DEBUG
- control_log_file: /tmp/daos_control.log
++#control_log_file: /tmp/daos_control.log
@tanabarr
tanabarr / prepare_scm
Last active September 12, 2019 14:00
Using daos_server to prepare local SCM/DCPM modules
[root@wolf-72 daos_m]# daos_server storage prepare --debug
bash: daos_server: command not found
[root@wolf-72 daos_m]# source scons_local/utils/setup_local.sh
/home/tanabarr/projects/daos_m
Build vars file found: ./.build_vars.sh
OLD_PATH is /usr/lib64/qt-3.3/bin /usr/local/bin /usr/bin /usr/local/sbin /usr/sbin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin /home/tanabarr/bin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin
[root@wolf-72 daos_m]# daos_server storage prepare --debug
DEBUG 13:54:28.030669 ctl_storage.go:98: spdk setup with _NRHUGE=1024
DEBUG 13:54:28.033110 ctl_storage.go:98: spdk setup with _TARGET_USER=root
Preparing locally-attached SCM...
@tanabarr
tanabarr / create_container
Last active October 31, 2019 00:37
Create DAOS container after creating DAOS pool through the control plane (dmg)
These commands were issued on binaries built from f6a34514a45eb3fb38dc43771d11c9f61e510516
SERVER CONFIG (NVMe is disabled to simplify this example only, should work fine with it enabled):
--- a/utils/config/examples/daos_server_sockets.yml
+++ b/utils/config/examples/daos_server_sockets.yml
@@ -6,10 +6,11 @@ provider: ofi+sockets # map to CRT_PHY_ADDR_STR=ofi+sockets
socket_dir: /tmp/daos_sockets
nr_hugepages: 4096
control_log_mask: DEBUG
@tanabarr
tanabarr / basic_daos_deployment_workflow
Last active August 28, 2019 13:09
Deployment workflow for DAOS using the control plane tools
## Prepare devices for use
[root@<hostname> tanabarr]# cd projects/daos_m/
[root@<hostname> daos_m]# source scons_local/utils/setup_local.sh
/home/tanabarr/projects/daos_m
Build vars file found: ./.build_vars.sh
OLD_PATH is /usr/lib64/qt-3.3/bin /usr/local/bin /usr/bin /usr/local/sbin /usr/sbin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin /home/tanabarr/bin /usr/lib/go-1.10/bin /usr/local/go/bin /usr/local/go/bin
[root@<hostname> daos_m]# daos_server storage prep-scm
Memory allocation goals for SCM will be changed and namespaces modified, this will be a destructive operation. ensure namespaces are unmounted and SCM is otherwise unused.
Are you sure you want to continue? (yes/no)