Skip to content

Instantly share code, notes, and snippets.

View tanabarr's full-sized avatar
💭
working on daos.io

Tom Nabarro tanabarr

💭
working on daos.io
10 followers · 4 following
View GitHub Profile
@tanabarr
tanabarr / isa-l undefined
Created February 1, 2021 09:29
dpdk build error
Found ninja-1.7.2 at /usr/bin/ninja
+ ninja -C build
ninja: Entering directory `build'
[1415/1451] Linking target drivers/librte_compress_isal.so.21.0.
FAILED: drivers/librte_compress_isal.so.21.0
cc -o drivers/librte_compress_isal.so.21.0 'drivers/drivers@@rte_compress_isal@sha/meson-generated_.._rte_compress_isal.pmd.c.o' 'drivers/drivers@@tmp_rte_compress_isal@sta/compress_isal_isal_compress_pmd.c.o' 'drivers/drivers@@tmp_rte_compress_isal@sta/compress_isal_isal_compress_pmd_ops.c.o' -Wl,--no-undefined -Wl,--as-needed -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,librte_compress_isal.so.21 -Wl,--no-as-needed -pthread -lm -ldl -lnuma lib/librte_compressdev.so.21.0 lib/librte_eal.so.21.0 lib/librte_kvargs.so.21.0 lib/librte_telemetry.so.21.0 lib/librte_mbuf.so.21.0 lib/librte_mempool.so.21.0 lib/librte_ring.so.21.0 drivers/librte_bus_vdev.so.21.0 -Wl,--end-group -Wl,--version-script=/home/tanabarr/projects/daos-dpdk-20_11/_topdir/BUILD/dpdk-20.11/drivers/compress/isal/version.map '-Wl,-rpath,$ORIGIN/.
@tanabarr
tanabarr / build_with_spdk
Created July 10, 2019 10:49
attempting to build and link to spdk library
[tanabarr@wolf-71 go-spdk]$ export CGO_LDFLAGS="-L/home/tanabarr/projects/daos_m/install/lib"
[tanabarr@wolf-71 go-spdk]$ export CGO_CFLAGS="-I/home/tanabarr/projects/daos_m/install/include"
[tanabarr@wolf-71 go-spdk]$ bin/./task main-task
rm -f *.o *.so
gcc ${CGO_LDFLAGS} ${CGO_CFLAGS} -Werror -g -Wshadow -Wall -Wno-missing-braces -c -fpic -Iinclude src/*.c -lspdk
gcc ${CGO_LDFLAGS} ${CGO_CFLAGS} -shared -o libnvme_control.so *.o
go test -v
# _/home/tanabarr/projects/daos_m/src/control/vendor/github.com/daos-stack/go-spdk/spdk
/tmp/go-build011208827/b046/_x003.o: In function `_cgo_dc3fa52d99f9_Cfunc_spdk_env_init':
spdk.cgo2.c:(.text+0x5e): undefined reference to `spdk_env_init'
@tanabarr
tanabarr / daos_test_psm2
Last active October 31, 2019 09:26
Run DAOS pool test suite with OFI PSM2 provider, DCPM modules and NVMe SSDs.
CONFIG:
name: daos_server # map to -g daos_server
port: 10001 # mgmt port
provider: ofi+psm2 # map to CRT_PHY_ADDR_STR=ofi+psm2
nr_hugepages: 4096
control_log_mask: DEBUG
control_log_file: /tmp/daos_control.log
transport_config:
@tanabarr
tanabarr / create_container
Last active October 31, 2019 00:37
Create DAOS container after creating DAOS pool through the control plane (dmg)
These commands were issued on binaries built from f6a34514a45eb3fb38dc43771d11c9f61e510516
SERVER CONFIG (NVMe is disabled to simplify this example only, should work fine with it enabled):
--- a/utils/config/examples/daos_server_sockets.yml
+++ b/utils/config/examples/daos_server_sockets.yml
@@ -6,10 +6,11 @@ provider: ofi+sockets # map to CRT_PHY_ADDR_STR=ofi+sockets
socket_dir: /tmp/daos_sockets
nr_hugepages: 4096
control_log_mask: DEBUG
@tanabarr
tanabarr / daos_test_wolf_sockets
Last active October 9, 2019 10:01
Commands and config file needed to run daos_test to validate installation using sockets provider.
Build from commit 2e4fcb939e83f8d65b1cc1667f2a195f8cf1131c
CONFIG:
--- a/utils/config/examples/daos_server_sockets.yml
+++ b/utils/config/examples/daos_server_sockets.yml
@@ -3,13 +3,15 @@
name: daos_server # map to -g daos_server
port: 10001 # mgmt port
provider: ofi+sockets # map to CRT_PHY_ADDR_STR=ofi+sockets
@tanabarr
tanabarr / respawn_non_root
Last active October 8, 2019 17:20
respawn daos_server as normal user (specified in server config) after starting as root and performing format from management tool (daos_shell)
---- SERVER: ----
root@wolf-72:/home/tanabarr/projects/daos_m$ umount /mnt/daos; rm -rf /mnt/daos; mkdir /mnt/daos
root@wolf-72:/home/tanabarr/projects/daos_m$ daos_server storage prepare -n -b
Preparing locally-attached NVMe storage...
2019/07/11 20:45:30 storage_nvme.go:98: debug: spdk setup with _NRHUGE=1024
2019/07/11 20:45:30 storage_nvme.go:102: debug: spdk setup with _TARGET_USER=root
+++ b/utils/config/examples/daos_server_sockets.yml
@@ -9,7 +9,7 @@ control_log_mask: DEBUG
@tanabarr
tanabarr / create_pool_with_acls
Created September 26, 2019 14:03
Create a DAOS pool with permissions read from a supplied ACL file.
[root@wolf-71 daos_m]# daos_shell pool create -a /tmp/acls -s 1G -n 10G -i
Active connections: [localhost:10001]
Creating DAOS pool with 1GB SCM and 10GB NvMe storage (0.100 ratio)
Pool-create command SUCCEEDED: UUID: 58191829-6d76-4a37-b7cd-ded5604bfbde, Service replicas: 0
[root@wolf-71 daos_m]# cat /tmp/acls
A::root@:rw
A::OWNER@:rw
A:g:readers@:r
L:f:baduser@:rw
U:f:EVERYONE@:rw
@tanabarr
tanabarr / daos_test_wolf_psm2
Last active September 26, 2019 13:05
Commands and config file needed to run daos_test to validate installation using psm2 provider.
Build from commit 2e4fcb939e83f8d65b1cc1667f2a195f8cf1131c
Verify provider is available:
daos_m/install/bin/fi_info | grep psm
(should show some output, if not then rebuild daos_m/_build.external/ofi)
need to run client and server on different hosts to be able to reconnect without restarting server (required in daos_test)
CONFIG:
@tanabarr
tanabarr / run_daos_docker_troubleshooting
Created September 24, 2019 15:49
Probe UIO and map hugepages when running DAOS in docker container
daos@wolf-130:~/docker/daos$ ls /sys/bus/pci/drivers/uio_pci_generic
ls: cannot access /sys/bus/pci/drivers/uio_pci_generic: No such file or director
daos@wolf-130:~/docker/daos$ sudo modprobe uio_pci_generic
daos@wolf-130:~/docker/daos $ ls /sys/bus/pci/drivers/uio_pci_generic
0000:00:04.0 0000:00:04.3 0000:00:04.6 0000:5f:00.0 0000:80:04.2 0000:80:04.5 0000:81:00.0 module uevent
0000:00:04.1 0000:00:04.4 0000:00:04.7 0000:80:04.0 0000:80:04.3 0000:80:04.6 0000:da:00.0 new_id unbind
0000:00:04.2 0000:00:04.5 0000:5e:00.0 0000:80:04.1 0000:80:04.4 0000:80:04.7 bind remove_id
@tanabarr
tanabarr / pool_create_no_client_config
Created September 23, 2019 15:12
Creating Pool without specifying a client side config file in command
SERVER:
--- a/utils/config/examples/daos_server_sockets.yml
+++ b/utils/config/examples/daos_server_sockets.yml
@@@ -6,10 -6,8 +6,12 @@@ provider: ofi+sockets # map to CR
socket_dir: /tmp/daos_sockets
nr_hugepages: 4096
control_log_mask: DEBUG
- control_log_file: /tmp/daos_control.log
++#control_log_file: /tmp/daos_control.log