peluse

/*
 * This routine returns the disk from which the requested read should
 * be done. There are several exceptions and conditions involved with
 * choosing the best disk for performance while maintaining integrity.
 * 
 * See comments in the code for complete details.
 *
 * The rdev for the device selected will have nr_pending incremented.
 */

peluse

Following the multi-mode directions I'm unable to start the testpod:

peluse@km:~$ kubectl get pods
NAME                         READY   STATUS              RESTARTS   AGE
nginx-web-5bf45d88df-pmd8m   1/1     Running             0          4m44s
spdkcsi-controller-0         3/3     Running             0          10m
spdkcsi-node-q67gh           2/2     Running             0          10m
spdkcsi-node-wf5fk           2/2     Running             0          10m
spdkcsi-test                 0/1     ContainerCreating   0          8m46s

peluse

/*
 *   BSD LICENSE
 *
 *   Copyright (c) Intel Corporation.
 *   All rights reserved.
 *
 *   Redistribution and use in source and binary forms, with or without
 *   modification, are permitted provided that the following conditions
 *   are met:
 *

peluse

Started by upstream project "autotest-nightly" build number 531
originally caused by:
 Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on fedora_vm_2 (SingleDiskVM VM) in workspace /var/jenkins/workspace/RBD_BlobFS_tests/RBD_autotest
Running Prebuild steps
[RBD_autotest] $ /bin/sh -xe /tmp/jenkins7771415223451547306.sh
+ sudo rm -rf /var/jenkins/workspace/RBD_BlobFS_tests/RBD_autotest/build.log /var/jenkins/workspace/RBD_BlobFS_tests/RBD_autotest/copy_or_clone.sh /var/jenkins/workspace/RBD_BlobFS_tests/RBD_autotest/cov_total.info /var/jenkins/workspace/RBD_BlobFS_tests/RBD_autotest/output /var/jenkins/workspace/RBD_BlobFS_tests/RBD_autotest/spdk /var/jenkins/workspace/RBD_BlobFS_tests/RBD_autotest/spdk17d693fa9aa458ae0b7422c5a05451d818c8f337.zip
Success build forhudson.tasks.Shell@1a8e1b70
provisioning config files...

peluse

make -C hello_world S=examples/bdev/hello_world all
echo "  LINK examples/bdev/hello_world/hello_bdev"; cc -o hello_bdev  -Wl,-z,relro,-z,now -Wl,-z,noexecstack -fsanitize=address -pthread hello_bdev.o  -L/home/peluse/spdk/build/lib -Wl,--whole-archive -lspdk_bdev_lvol -lspdk_blobfs -lspdk_blob -lspdk_blob_bdev -lspdk_lvol -lspdk_bdev_malloc -lspdk_bdev_null -lspdk_bdev_nvme -lspdk_nvme -lspdk_bdev_passthru -lspdk_bdev_error -lspdk_bdev_gpt -lspdk_bdev_split -lspdk_bdev_raid -lspdk_bdev_compress -lspdk_reduce -lspdk_bdev_aio -lspdk_bdev_virtio -lspdk_virtio -lspdk_bdev_pmem -lspdk_copy_ioat -lspdk_ioat -lspdk_sock_posix -lspdk_event_bdev -lspdk_event_copy -lspdk_bdev -lspdk_copy -lspdk_event -lspdk_thread -lspdk_util -lspdk_conf -lspdk_trace -lspdk_log -lspdk_jsonrpc -lspdk_json -lspdk_rpc -lspdk_sock -Wl,--no-whole-archive /home/peluse/spdk/build/lib/libspdk_env_dpdk.a -Wl,--whole-archive /home/peluse/spdk/dpdk/build/lib/librte_bus_pci.a /home/peluse/spdk/dpdk/build/lib/librte_bus_vdev.a /home/peluse/spdk/dp

peluse

import com.sonyericsson.hudson.plugins.gerrit.trigger.config.Config
import com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer
import com.sonyericsson.hudson.plugins.gerrit.trigger.PluginImpl
import com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.GerritManualCause
import com.sonyericsson.hudson.plugins.gerrit.trigger.hudsontrigger.actions.manual.ManualTriggerAction
import org.kohsuke.stapler.*;
import org.kohsuke.stapler.StaplerRequest
import org.kohsuke.stapler.StaplerResponse
import javax.servlet.http.HttpServletRequestWrapper
import javax.servlet.http.HttpServletRequest

peluse

(gdb) p *bdev_io->u.bdev.iovs
$24 = {iov_base = 0x7fffe87c3c00, iov_len = 8640}
(gdb) p *bdev_io
$25 = {bdev = 0x612000000ad0, type = 1 '\001', iov = {iov_base = 0x0, iov_len = 0}, u = {bdev = {iovs = 0x7fffec801b90,
      iovcnt = 1, num_blocks = 8, offset_blocks = 16, stored_user_cb = 0x0, split_remaining_num_blocks = 0,
      split_current_offset_blocks = 0}, reset = {ch_ref = 0x7fffec801b90}, nvme_passthru = {cmd = {opc = 144, fuse = 3,
        rsvd1 = 6, psdt = 0, cid = 60544, nsid = 32767, rsvd2 = 1, rsvd3 = 0, mptr = 8, dptr = {prp = {prp1 = 16, prp2 = 0},
          sgl1 = {address = 16, {generic = {reserved = "\000\000\000\000\000\000", subtype = 0 '\000', type = 0 '\000'},
              unkeyed = {length = 0, reserved = "\000\000", subtype = 0 '\000', type = 0 '\000'}, keyed = {length = 0, key = 0,
                subtype = 0, type = 0}}}}, cdw10 = 0, cdw11 = 0, cdw12 = 0, cdw13 = 0, cdw14 = 0, cdw15 = 0}, buf = 0x0,

peluse

Run Summary:    Type  Total    Ran Passed Failed Inactive
              suites      1      1    n/a      0        0
               tests      4      4      4      0        0
             asserts     53     53     53      0      n/a

Elapsed time =    0.001 seconds
+ '[' yes = yes ']'
++ hostname
+ lcov --rc lcov_branch_coverage=1 --rc lcov_function_coverage=1 --rc genhtml_branch_coverage=1 --rc genhtml_function_coverage=1 --rc genhtml_legend=1 --rc geninfo_all_blocks=1 --no-external -q -d . -c -t pels3-64 -o ut_coverage/ut_cov_test.info
geninfo: WARNING: invalid characters removed from testname!

peluse

Some data points:

- doesn't happen w/just reads as workload
- doesn't haapen with a mix of read/write as workload provided that I treat 'write' as pass through.
* conclusion from the above 2, it's in the write path

- doesn't happen if I make both read/write path effectively pass through (no other code changes)
* conclusion from above point, doesn't seem like anything in my channel handlers is busted

- doesn't happen with just 1 vbdev and 1 core

peluse

[peluse@pels3-64 bdevio]$ sudo ./bdevio ../bdev.conf
[sudo] password for peluse:
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: bdevtest -c 0x7 -m 1024 --file-prefix=spdk_pid6626 ]
EAL: Detected 88 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 376:spdk_app_start: *NOTICE*: Total cores available: 3
reactor.c: 654:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
reactor.c: 426:_spdk_reactor_run: *NOTICE*: Reactor started on core 1 on socket 0