Skip to content

Instantly share code, notes, and snippets.

@njtierney

njtierney/test-results.md

Created January 17, 2024 01:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save njtierney/f233a6165990a8b983d178453648ce7a to your computer and use it in GitHub Desktop.
Save njtierney/f233a6165990a8b983d178453648ce7a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
test-results.md 
==> Testing R file using 'testthat'

ℹ Loading greta
ℹ Initialising python and checking dependencies, this may take a moment.

✔ Initialising python and checking dependencies ... done!


══ Testing test_posteriors.R ═══════════════════════════════════════════════════

[ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ]
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 2 ]
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 3 ]
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 3 ]
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 4 ]
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 5 ]
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 6 ]
[ FAIL 2 | WARN 0 | SKIP 0 | PASS 6 ]
[ FAIL 3 | WARN 0 | SKIP 0 | PASS 6 ]
[ FAIL 4 | WARN 0 | SKIP 0 | PASS 6 ]
[ FAIL 5 | WARN 0 | SKIP 0 | PASS 6 ]

── Failure (test_posteriors.R:44:3): samplers are unbiased for bivariate normals ──
max(errors) is not less than stats::qnorm(0.99). Difference: 0.432
Backtrace:
 1. check_mvn_samples(slice())
      at test_posteriors.R:44:2
 2. testthat::expect_lte(max(errors), stats::qnorm(0.99))
      at tests/testthat/helpers.R:799:2

── Failure (test_posteriors.R:98:3): samplers are unbiased for Wishart ─────────
stat$p.value is not more than 0.01. Difference: -0.01
Backtrace:
 1. check_samples(x, iid, one_by_one = TRUE)
      at test_posteriors.R:98:2
 2. testthat::expect_gte(stat$p.value, 0.01)
      at tests/testthat/helpers.R:836:2

── Failure (test_posteriors.R:135:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
 1. check_geweke(...)
      at test_posteriors.R:135:2
 2. testthat::expect_gte(stat$p.value, 0.005)
      at tests/testthat/helpers.R:620:2

── Failure (test_posteriors.R:144:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
 1. check_geweke(...)
      at test_posteriors.R:144:2
 2. testthat::expect_gte(stat$p.value, 0.005)
      at tests/testthat/helpers.R:620:2

── Failure (test_posteriors.R:154:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
 1. check_geweke(...)
      at test_posteriors.R:154:2
 2. testthat::expect_gte(stat$p.value, 0.005)
      at tests/testthat/helpers.R:620:2

[ FAIL 5 | WARN 0 | SKIP 0 | PASS 6 ]

Test complete
@njtierney
Copy link
Author

And changing back to commit 671a59aeda7471d1fdb56d2d3121cbededb2f9d5 - which is before I removed the helper.R files.

── Failure (test_posteriors.R:98:3): samplers are unbiased for Wishart ─────────
stat$p.value is not more than 0.01. Difference: -0.01
Backtrace:
 1. check_samples(x, iid, one_by_one = TRUE)
      at test_posteriors.R:98:2
 2. testthat::expect_gte(stat$p.value, 0.01)
      at tests/testthat/helpers.R:982:2

── Failure (test_posteriors.R:135:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
 1. check_geweke(...)
      at test_posteriors.R:135:2
 2. testthat::expect_gte(stat$p.value, 0.005)
      at tests/testthat/helpers.R:766:2

── Failure (test_posteriors.R:144:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
 1. check_geweke(...)
      at test_posteriors.R:144:2
 2. testthat::expect_gte(stat$p.value, 0.005)
      at tests/testthat/helpers.R:766:2

── Failure (test_posteriors.R:154:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
 1. check_geweke(...)
      at test_posteriors.R:154:2
 2. testthat::expect_gte(stat$p.value, 0.005)
      at tests/testthat/helpers.R:766:2


[ FAIL 4 | WARN 0 | SKIP 0 | PASS 7 ]

@njtierney
Copy link
Author

And going back to master

We get the same errors once we add the source("helpers.R") code

── Failure (test_posteriors.R:98:3): samplers are unbiased for Wishart ─────────
stat$p.value is not more than 0.01. Difference: -0.01
Backtrace:
    ▆
 1. └─check_samples(x, iid, one_by_one = TRUE) at test_posteriors.R:98:3
 2.   └─testthat::expect_gte(stat$p.value, 0.01) at tests/testthat/helpers.R:836:3

── Failure (test_posteriors.R:135:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
    ▆
 1. └─check_geweke(...) at test_posteriors.R:135:3
 2.   └─testthat::expect_gte(stat$p.value, 0.005) at tests/testthat/helpers.R:620:3

── Failure (test_posteriors.R:144:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
    ▆
 1. └─check_geweke(...) at test_posteriors.R:144:3
 2.   └─testthat::expect_gte(stat$p.value, 0.005) at tests/testthat/helpers.R:620:3

── Failure (test_posteriors.R:154:3): samplers pass geweke tests ───────────────
stat$p.value is not more than 0.005. Difference: -0.005
Backtrace:
    ▆
 1. └─check_geweke(...) at test_posteriors.R:154:3
 2.   └─testthat::expect_gte(stat$p.value, 0.005) at tests/testthat/helpers.R:620:3
[ FAIL 4 | WARN 0 | SKIP 0 | PASS 7 ]

Test complete

@njtierney
Copy link
Author

as of commit 3da6a0c057202d3b7008333ed667542ed045124a

We get these warnings, but we don't get the geweke failures, so I'm not sure if this is particularly problematic

── Warning (test_posteriors.R:75:3): samplers are unbiased for standard uniform ──
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid) at test_posteriors.R:75:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)

── Warning (test_posteriors.R:75:3): samplers are unbiased for standard uniform ──
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid) at test_posteriors.R:75:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)

── Warning (test_posteriors.R:75:3): samplers are unbiased for standard uniform ──
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid) at test_posteriors.R:75:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)

── Warning (test_posteriors.R:75:3): samplers are unbiased for standard uniform ──
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid) at test_posteriors.R:75:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)

── Warning (test_posteriors.R:106:3): samplers are unbiased for Wishart ────────
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid, one_by_one = TRUE) at test_posteriors.R:106:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)

── Warning (test_posteriors.R:106:3): samplers are unbiased for Wishart ────────
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid, one_by_one = TRUE) at test_posteriors.R:106:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)

── Warning (test_posteriors.R:106:3): samplers are unbiased for Wishart ────────
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid, one_by_one = TRUE) at test_posteriors.R:106:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)

── Warning (test_posteriors.R:106:3): samplers are unbiased for Wishart ────────
start value not changed
Backtrace:
     ▆
  1. └─check_samples(x, iid, one_by_one = TRUE) at test_posteriors.R:106:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:718:3
  3.     └─not_finished(draws, n_effective) at tests/testthat/helpers.R:627:3
  4.       └─coda::gelman.diag(draws, multivariate = FALSE) at tests/testthat/helpers.R:587:3
  5.         ├─stats::window(x, start = end(x)/2 + 1)
  6.         └─greta:::window.greta_mcmc_list(x, start = end(x)/2 + 1)
  7.           ├─stats::window(model_info$raw_draws, start, end, thin, ...) at greta/R/greta_mcmc_list.R:22:3
  8.           └─coda:::window.mcmc.list(...)
  9.             ├─base::structure(lapply(x, window.mcmc, ...), class = "mcmc.list")
 10.             └─base::lapply(x, window.mcmc, ...)
 11.               └─coda (local) FUN(X[[i]], ...)
[ FAIL 0 | WARN 8 | SKIP 0 | PASS 11 ]

@njtierney
Copy link
Author

This is at commit 91fb989545a042fd2b51767ab1ffb09cf0cb414e

And is used R package tensorflow 1.13.1 instead of 1.14.0

geweke tests seem to pass?

==> Testing R file using 'testthat'

ℹ Loading greta
[ FAIL 2 | WARN 0 | SKIP 0 | PASS 9 ]

── Error (test_posteriors.R:89:3): samplers are unbiased for LKJ ───────────────
Error: greta hit a tensorflow error:

Input to reshape is a tensor with 1 values, but the requested shape requires a multiple of 4
	 [[node Reshape (defined at /ops/gen_array_ops.py:7715) ]]

Errors may have originated from an input operation.
Input Source operations connected to node Reshape:
 Placeholder (defined at /ops/array_ops.py:2143)

Original stack trace for 'Reshape':
  File "/ops/gen_array_ops.py", line 7715, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Backtrace:
     ▆
  1. └─check_samples(x, iid, hmc(), one_by_one = TRUE) at test_posteriors.R:89:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:733:3
  3.     └─greta::mcmc(model, sampler = sampler, verbose = verbose, one_by_one = one_by_one) at tests/testthat/helpers.R:627:3
  4.       └─base::lapply(initial_values_split, build_sampler, sampler, model) at greta/R/inference.R:240:3
  5.         └─greta (local) FUN(X[[i]], ...)
  6.           └─sampler$class$new(...) at greta/R/utils.R:977:3
  7.             └─greta (local) initialize(...)
  8.               └─super$initialize(...) at greta/R/inference_class.R:273:7
  9.                 └─self$set_initial_values(initial_values) at greta/R/inference_class.R:47:7
 10.                   └─base::lapply(init_list, self$check_initial_values) at greta/R/inference_class.R:138:7
 11.                     └─greta (local) FUN(X[[i]], ...)
 12.                       └─self$valid_parameters(inits) at greta/R/inference_class.R:104:11
 13.                         └─self$model$dag$log_density() at greta/R/inference_class.R:159:7
 14.                           └─greta:::cleanly(self$tf_sess_run(joint_density_adj)) at greta/R/dag_class.R:460:7

── Error (test_posteriors.R:106:3): samplers are unbiased for Wishart ──────────
Error: greta hit a tensorflow error:

Input to reshape is a tensor with 3 values, but the requested shape requires a multiple of 4
	 [[node Reshape (defined at /ops/gen_array_ops.py:7715) ]]

Errors may have originated from an input operation.
Input Source operations connected to node Reshape:
 Placeholder (defined at /ops/array_ops.py:2143)

Original stack trace for 'Reshape':
  File "/ops/gen_array_ops.py", line 7715, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Backtrace:
     ▆
  1. └─check_samples(x, iid, one_by_one = TRUE) at test_posteriors.R:106:3
  2.   └─get_enough_draws(...) at tests/testthat/helpers.R:733:3
  3.     └─greta::mcmc(model, sampler = sampler, verbose = verbose, one_by_one = one_by_one) at tests/testthat/helpers.R:627:3
  4.       └─base::lapply(initial_values_split, build_sampler, sampler, model) at greta/R/inference.R:240:3
  5.         └─greta (local) FUN(X[[i]], ...)
  6.           └─sampler$class$new(...) at greta/R/utils.R:977:3
  7.             └─greta (local) initialize(...)
  8.               └─super$initialize(...) at greta/R/inference_class.R:273:7
  9.                 └─self$set_initial_values(initial_values) at greta/R/inference_class.R:47:7
 10.                   └─base::lapply(init_list, self$check_initial_values) at greta/R/inference_class.R:138:7
 11.                     └─greta (local) FUN(X[[i]], ...)
 12.                       └─self$valid_parameters(inits) at greta/R/inference_class.R:104:11
 13.                         └─self$model$dag$log_density() at greta/R/inference_class.R:159:7
 14.                           └─greta:::cleanly(self$tf_sess_run(joint_density_adj)) at greta/R/dag_class.R:460:7
[ FAIL 2 | WARN 0 | SKIP 0 | PASS 9 ]

Test complete

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment