Skip to content

Instantly share code, notes, and snippets.

@wphicks
Last active March 8, 2021 15:47
Show Gist options
  • Save wphicks/0182dd17ced81222e7a7beb173f3fe9c to your computer and use it in GitHub Desktop.
Save wphicks/0182dd17ced81222e7a7beb173f3fe9c to your computer and use it in GitHub Desktop.
cuML Python tests compute sanitizer output
This file has been truncated, but you can view the full file.
compute-sanitizer pytest python/cuml/test
========= COMPUTE-SANITIZER
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rapids_pytest_benchmark: 0.0.13
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/whicks/proj_cuml/cuml/python, configfile: pytest.ini
plugins: hypothesis-6.1.1, cov-2.11.1, rapids-pytest-benchmark-0.0.13, benchmark-3.2.3, timeout-1.4.2, repeat-0.8.0, forked-1.3.0, asyncio-0.12.0, xdist-2.2.0
collected 27716 items
python/cuml/test/test_adapters.py ... [ 0%]
python/cuml/test/test_allocator.py ... [ 0%]
python/cuml/test/test_api.py ........................................... [ 0%]
..................................x...........x.....x.........xxx. [ 0%]
python/cuml/test/test_arima.py ........s........s........s........s..... [ 0%]
...s........s........s........s........s........s........s [ 0%]
python/cuml/test/test_array.py ...s..................................s.. [ 0%]
..s....s....s....s..................................s....s....s....s.... [ 1%]
s..................................s....s....s....s....s....s....s....s. [ 1%]
...s....s....s....s....s....s....s....s....s....s....s....s....s....s... [ 1%]
.s....s....s....s....s..................................s....s....s....s [ 1%]
....s..................................s....s....s....s....s............ [ 2%]
......................s....s....s....s....s....s....s....s....s....s.... [ 2%]
s....s....s....s....s....s....s....s....s....s....s....s....s....s....s. [ 2%]
...s.s.................................................................. [ 2%]
........................................................................ [ 3%]
........................................................................ [ 3%]
........................................................................ [ 3%]
........................................................................ [ 4%]
........................................................................ [ 4%]
........................................................................ [ 4%]
........................................................................ [ 4%]
........................................................................ [ 5%]
........................................................................ [ 5%]
........................................................................ [ 5%]
........................................................................ [ 5%]
...............................sssss.s.................................. [ 6%]
........................................................................ [ 6%]
........................................................................ [ 6%]
........................................................................ [ 6%]
........................................................................ [ 7%]
........................................................................ [ 7%]
........................................................................ [ 7%]
........................................................................ [ 7%]
........................................................................ [ 8%]
........................................................................ [ 8%]
........................................................................ [ 8%]
........................................................................ [ 8%]
........................................................................ [ 9%]
........................................................................ [ 9%]
........................................................................ [ 9%]
........................................................................ [ 10%]
........................................................................ [ 10%]
........................................................................ [ 10%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 11%]
........................................................................ [ 12%]
........................................................................ [ 12%]
........................................................................ [ 12%]
........................................................................ [ 12%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 13%]
........................................................................ [ 14%]
........................................................................ [ 14%]
........................................................................ [ 14%]
........................................................................ [ 14%]
........................................................................ [ 15%]
........................................................................ [ 15%]
........................................................................ [ 15%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 16%]
........................................................................ [ 16%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 18%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 20%]
........................................................................ [ 20%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 21%]
........................................................................ [ 21%]
........................................................................ [ 21%]
........................................................................ [ 22%]
........................................................................ [ 22%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 23%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 25%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 26%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 28%]
........................................................................ [ 29%]
........................................................................ [ 29%]
........................................................................ [ 29%]
........................................................................ [ 30%]
........................................................................ [ 30%]
........................................................................ [ 30%]
........................................................................ [ 30%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 33%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 34%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 35%]
........................................................................ [ 36%]
........................................................................ [ 36%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................ [ 38%]
python/cuml/test/test_array_sparse.py .................................. [ 38%]
................... [ 38%]
python/cuml/test/test_auto_arima.py .................................... [ 38%]
............................................. [ 38%]
python/cuml/test/test_base.py .......................................... [ 38%]
................s..............sss..................sss [ 38%]
python/cuml/test/test_batched_lbfgs.py . [ 38%]
python/cuml/test/test_benchmark.py .......................... [ 39%]
python/cuml/test/test_class_enumerator.py .... [ 39%]
python/cuml/test/test_coordinate_descent.py ........ssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssss..ssssssssssssssss........ss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss..ssssssss [ 39%]
ssssssss........ [ 39%]
python/cuml/test/test_cuml_descr_decor.py .s............................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 40%]
........................x...x.xxxx..x...x...x.xxxx..x...x...x.xxxx..x... [ 40%]
x...x.xxxx..x. [ 40%]
python/cuml/test/test_dataset_generator_types.py .................... [ 40%]
python/cuml/test/test_dbscan.py ........ssssssssssssssssssssssssssssssss [ 40%]
ssssssssssssssssssssssssssssssss........ssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssss........ssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssss........ssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 42%]
ssssssssssssssssssssssssssssssss...ssssss...ssssss...ssssss...x......... [ 42%]
........... [ 42%]
python/cuml/test/test_fil.py .ssssssss.ssssssssssssssssssssssssss.ssssss [ 42%]
ss.ssssssssssssssssssssssssss.ssssssss.ssssssssssssssssssssssssss.ssssss [ 42%]
ssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssss [ 43%]
ss...................................................................... [ 43%]
................. [ 43%]
python/cuml/test/test_holtwinters.py .x.x.x.x........................... [ 43%]
... [ 43%]
python/cuml/test/test_incremental_pca.py .ss............ [ 43%]
python/cuml/test/test_input_utils.py ................................... [ 43%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 45%]
........................................................................ [ 45%]
......................................................xxxxxxxxxxxx...... [ 45%]
....x..........x..........x..........x..........x..........x..........x. [ 46%]
.........x..........x..........xxxxxxxxxxxx..........x..........x....... [ 46%]
...x..........x..........x..........x..........x..........x..........x.. [ 46%]
........xxxxxxxxxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xx [ 46%]
xxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxxxxxxxxxxxxx... [ 47%]
...xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxx [ 47%]
xx......xxxxx......xxxxx......xxxxxxxxxxxxxxxx..........x..........x.... [ 47%]
......x..........x..........x..........x..........x..........x.......... [ 48%]
x....................................................................... [ 48%]
............................................................xxxxxxxxxxxx [ 48%]
..........x..........x..........x..........x..........x..........x...... [ 48%]
....x..........x..........x..........xxxxxxxxxxxx..........x..........x. [ 49%]
.........x..........x..........x..........x..........x..........x....... [ 49%]
...x..........xxxxxxxxxxxx......xxxxx......xxxxx......xxxxx......xxxxx.. [ 49%]
....xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxxxxxxxxxx [ 49%]
xxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxx......xxxxx... [ 50%]
...xxxxx......xxxxx......xxxxx......xxxxxxxxxxxxxxxx..........x......... [ 50%]
.x..........x..........x..........x..........x..........x..........x.... [ 50%]
......x................................................................. [ 50%]
.................................xxxxxxxx............................... [ 51%]
.................................................xxxxxxxx............... [ 51%]
.................................................................xxxxxxx [ 51%]
x................................................xxxxxxxxxxxxxxxxxxxxxxx [ 51%]
xxxxxxxxxxxxxxxxx....................................................... [ 52%]
.........................xxxxxxxx....................................... [ 52%]
........................................................................ [ 52%]
.........................................................xxxxxxxx....... [ 52%]
........................................................................ [ 53%]
.xxxxxxxx............................................................... [ 53%]
.................xxxxxxxx............................................... [ 53%]
.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx............................... [ 53%]
.................................................xxxxxxxx............... [ 54%]
........................................................................ [ 54%]
.......................... [ 54%]
python/cuml/test/test_internals_api.py ... [ 54%]
python/cuml/test/test_kmeans.py x.........................sssss.....ssss [ 54%]
ssssss..........sssss.ss.ssssssssssssss [ 54%]
python/cuml/test/test_kneighbors_classifier.py ......................... [ 54%]
........................................................................ [ 55%]
........................................................................ [ 55%]
........................................................................ [ 55%]
.............. [ 55%]
python/cuml/test/test_kneighbors_regressor.py .......................... [ 55%]
........................................................... [ 56%]
python/cuml/test/test_label_binarizer.py ........ [ 56%]
python/cuml/test/test_label_encoder.py ............................ [ 56%]
python/cuml/test/test_lars.py ..ssss..ssssssssssssssss..ssss..ssssssssss [ 56%]
ssssss..ssss..ssssssssssssssss..ssss..ssssssssssssssss..ssss..ssssssssss [ 56%]
ssssss..ssss..ssssssssssssssss..ssssssssssssssss..ssssssssssssssss...... [ 56%]
...... [ 56%]
python/cuml/test/test_linear_model.py ....ssssssssssssssssssssssssssssss [ 57%]
ss.........ssssssssssssssssssssssssssssssss............................. [ 57%]
........... [ 57%]
python/cuml/test/test_logger.py ... [ 57%]
python/cuml/test/test_make_arima.py .................................... [ 57%]
............ [ 57%]
python/cuml/test/test_make_blobs.py .................................... [ 57%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 58%]
........................................................................ [ 59%]
........................................................................ [ 59%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 60%]
........................................................................ [ 61%]
........................................................................ [ 61%]
........................................................................ [ 61%]
.................................... [ 61%]
python/cuml/test/test_make_classification.py ........................... [ 61%]
........................................................................ [ 62%]
........................................................................ [ 62%]
........................................................................ [ 62%]
.............. [ 62%]
python/cuml/test/test_make_regression.py ............................... [ 62%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 63%]
......... [ 63%]
python/cuml/test/test_mbsgd_classifier.py ..................ssssssssssss [ 63%]
ssssssssssssssssssssssss [ 63%]
python/cuml/test/test_mbsgd_regressor.py ..................sssssssssssss [ 63%]
sssssssssssssssssssssss [ 63%]
python/cuml/test/test_metrics.py ......ssssssssssssssssssssssssss...XXss [ 64%]
ssssssss................................................................ [ 64%]
..................................................................s.s.s. [ 64%]
s.s.s.s.s.s.s.s.s...............s.s.s.s.s.s.s.s....s..s..s..s..........s [ 64%]
..s..s..s............................................................... [ 65%]
..............................ssssssssssssss............................ [ 65%]
.ss.ss========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,28,0) in block (0,0,0)
========= Address 0x7f18f78f3ae0 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,29,0) in block (0,0,0)
========= Address 0x7f18f78f3ae8 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,30,0) in block (0,0,0)
========= Address 0x7f18f78f3af0 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,31,0) in block (0,0,0)
========= Address 0x7f18f78f3af8 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,24,0) in block (0,0,0)
========= Address 0x7f18f78f3ac0 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,25,0) in block (0,0,0)
========= Address 0x7f18f78f3ac8 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,26,0) in block (0,0,0)
========= Address 0x7f18f78f3ad0 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Invalid __global__ read of size 8 bytes
========= at 0x140 in void ML::GLM::logSoftmaxKernel<double,int=8,int=32>(double*,double,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const *,ML::GLM::logSoftmaxKernel<double,int=8,int=32> const ,int,int,bool)
========= by thread (0,27,0) in block (0,0,0)
========= Address 0x7f18f78f3ad8 is out of bounds
========= Saved host backtrace up to driver entry point at kernel launch time
========= Host Frame:cuLaunchKernel [0x7f1e83504dd8]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame: [0x7f1e1fa1262b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:cudaLaunchKernel [0x7f1e1fa525b1]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void ML::GLM::launchLogsoftmax<double>(double*, double, ML::GLM::launchLogsoftmax<double> const *, ML::GLM::launchLogsoftmax<double> const , int, int, CUstream_st*) [0x7f1d1ee75688]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee760e3]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaMemcpyAsync.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaMemcpyAsync [0x7f1e1fa4fdc9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void raft::copy<double>(double*, raft::copy<double> const *, unsigned long, CUstream_st*) [0x7f1d1ecbe7e5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee761cb]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:void raft::copy<double>(double*, raft::copy<double> const *, unsigned long, CUstream_st*) [0x7f1d1ecbe81d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::GLMWithData<double, ML::SimpleVec<double> const & ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>(ML::SimpleVec<double>&, double*, CUstream_st*)>::__operator_()__ [0x7f1d1ee761cb]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::OPT_RETCODE ML::GLM::min_lbfgs<double, ML::GLM::GLMWithData<double, ML::GLM::RegularizedGLM<double, ML::GLM::Softmax<double>, ML::GLM::Tikhonov<double>>>>(ML::GLM::LBFGSParam<double> const &, double&, ML::SimpleVec<ML::GLM::LBFGSParam>&, ML::GLM::LBFGSParam&, int*, ML::SimpleVec<ML::GLM::LBFGSParam>, CUstream_st*, int) [0x7f1d1ee768de]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c0ea]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:rmm::device_buffer::deallocate(void) [0x7f1d1ee3bd18]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:int ML::GLM::qn_fit<double, ML::GLM::Softmax<double>>(raft::handle_t const &, double&, double*, double, double, int, double&, double&, int, double&, int, int, int, double, double, int*, ML::STORAGE_ORDER, CUstream_st*) [0x7f1d1ee8c48e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8c9b9]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame:rmm::device_buffer::deallocate(void) [0x7f1d1ee3bd18]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:void ML::GLM::qnFit<double>(raft::handle_t const &, double*, double, int, int, int, bool, raft::handle_t const &, raft::handle_t const &, int, raft::handle_t const &, int, int, int, double, double, int*, bool, int, CUstream_st*) [0x7f1d1ee8cd0f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame:ML::GLM::qnFit(raft::handle_t const &, double*, double*, int, int, int, bool, double, double, int, double, int, int, int, double*, double*, int*, bool, int) [0x7f1d1ee265e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/libcuml++.so
========= Host Frame: [0x7f1cf4d019c3]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1dbb6df451]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/rmm/_lib/device_buffer.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d79136]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6bfc]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6bfc]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1de3d79136]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf4cff720]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1dbb6df451]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/rmm/_lib/device_buffer.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d79136]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6bfc]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6bfc]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1de3d79136]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf4cff5db]
========= in /home/whicks/proj_cuml/cuml/python/cuml/solvers/qn.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1cf47bcaee]
========= in /home/whicks/proj_cuml/cuml/python/cuml/linear_model/logistic_regression.cpython-38-x86_64-linux-gnu.so
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0c9d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
Exception ignored in: 'cupy.cuda.memory.Memory.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/runtime.pyx", line 607, in cupy_backends.cuda.api.runtime.free
File "cupy_backends/cuda/api/runtime.pyx", line 247, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorLaunchFailure: unspecified launch failure
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaFree [0x7f1e1fa41150]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9fa4b]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8f35b6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCodeEx [0x563deb87b149]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyEval_EvalCode [0x563deb921a3b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb921ad3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb9397a3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_FileExFlags [0x563deb93e6c7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyRun_SimpleFileExFlags [0x563deb93e89c]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_RunMain [0x563deb93ee1d]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_BytesMain [0x563deb93efd9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:__libc_start_main [0x7f1e84a070b3]
========= in /lib/x86_64-linux-gnu/libc.so.6
========= Host Frame: [0x563deb8b2b69]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
=========
========= Program hit unspecified launch failure (error 719) on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x7f1e8365c453]
========= in /lib/x86_64-linux-gnu/libcuda.so.1
========= Host Frame:cudaGetLastError [0x7f1e1fa3a73f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/../../../../../libcudart.so.11.0
========= Host Frame: [0x7f1e1fc9c7d0]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1fc9fa9f]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy_backends/cuda/api/runtime.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7f62d]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1de3d7933e]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/cuda/memory.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x7f1e1feaeab6]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cupy/core/core.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7c6be3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8566b5]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/column.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb90a03f]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f435]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb85f415]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c4ebf]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7c6a1b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x7f1d2e8e7794]
========= in /home/whicks/anaconda3/envs/cuml_dev/lib/python3.8/site-packages/cudf/_lib/table.cpython-38-x86_64-linux-gnu.so
========= Host Frame: [0x563deb7ee874]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8cab89]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_GC_New [0x563deb7ffc64]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyList_New [0x563deb7ffef4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b80]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb86aca0]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91bdce]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb91c0e4]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb8d3b99]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyAST_mod2obj [0x563deb91e532]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:Py_CompileStringObject [0x563deb93d9a5]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb93dc11]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb7f3a52]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87bbf7]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899e0b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a7be]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb87ba76]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89e053]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_FastCallDict [0x563deb7e83bd]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_Call_Prepend [0x563deb7f5483]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame: [0x563deb83f6f9]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyObject_MakeTpCall [0x563deb7efb8e]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ea2b]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb899cef]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563deb87a0c3]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b4c8]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89ab41]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyFunction_Vectorcall [0x563deb87b2f6]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:PyObject_Call [0x563deb7f0aae]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalFrameDefault [0x563deb89b571]
========= in /home/whicks/anaconda3/envs/cuml_dev/bin/python
========= Host Frame:_PyEval_EvalCodeWithName [0x563d
View raw

(Sorry about that, but we can’t show files that are this big right now.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment