This report documents the results of a simulation based calibration (SBC) run for RBesT
. The calibration data will be generated whenever relevant changes to the gMAP
function were made. The calibration runs are performed for typical use cases of gMAP
. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.
The calibration data presented here has been generated at and with the RBesT
git version as:
## Created: 2020-02-12 15:40:56 UTC
## git hash: e3057f62faa0666986ce1dbd8b9c6bde3c4ad788
## MD5: 7c79f49364765d14ce3d0d5ed04b49f5
The MD5 hash of the calibration data file presented here must match the above listed MD5:
## calibration.rds
## "7c79f49364765d14ce3d0d5ed04b49f5"
Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).
Self-consistency of any Bayesian analysis with a proper prior:
\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]
SBC procedure:
Repeat \(s=1, ..., S\) times:
Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]
The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]
Likelihood:
Hierarchical prior:
\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]
\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]
The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the sbc_tools.R
and make_reference_rankhist.R
R programs for the implementation details.
The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.
##
## Attaching package: 'purrr'
## The following object is masked from 'package:testthat':
##
## is_null
problem | likelihood | sd_tau | parameter | statistic | df | p.value | |
---|---|---|---|---|---|---|---|
1 | dense | binomial | 0.5 | mu | 37.747 | 63 | 0.995 |
13 | dense | binomial | 1 | mu | 64.026 | 63 | 0.440 |
25 | dense | gaussian | 0.5 | mu | 65.011 | 63 | 0.406 |
37 | dense | gaussian | 1 | mu | 46.362 | 63 | 0.943 |
49 | dense | poisson | 0.5 | mu | 62.490 | 63 | 0.494 |
61 | dense | poisson | 1 | mu | 59.021 | 63 | 0.619 |
73 | sparse | binomial | 0.5 | mu | 63.424 | 63 | 0.461 |
77 | sparse | binomial | 1 | mu | 69.466 | 63 | 0.269 |
81 | sparse | gaussian | 0.5 | mu | 69.670 | 63 | 0.263 |
85 | sparse | gaussian | 1 | mu | 56.922 | 63 | 0.691 |
89 | sparse | poisson | 0.5 | mu | 51.725 | 63 | 0.844 |
93 | sparse | poisson | 1 | mu | 70.669 | 63 | 0.237 |
problem | likelihood | sd_tau | parameter | statistic | df | p.value | |
---|---|---|---|---|---|---|---|
2 | dense | binomial | 0.5 | tau | 58.624 | 63 | 0.633 |
14 | dense | binomial | 1 | tau | 84.032 | 63 | 0.040 |
26 | dense | gaussian | 0.5 | tau | 69.683 | 63 | 0.263 |
38 | dense | gaussian | 1 | tau | 53.555 | 63 | 0.796 |
50 | dense | poisson | 0.5 | tau | 67.430 | 63 | 0.328 |
62 | dense | poisson | 1 | tau | 79.309 | 63 | 0.080 |
74 | sparse | binomial | 0.5 | tau | 74.202 | 63 | 0.158 |
78 | sparse | binomial | 1 | tau | 71.168 | 63 | 0.224 |
82 | sparse | gaussian | 0.5 | tau | 64.205 | 63 | 0.434 |
86 | sparse | gaussian | 1 | tau | 51.840 | 63 | 0.841 |
90 | sparse | poisson | 0.5 | tau | 67.136 | 63 | 0.337 |
94 | sparse | poisson | 1 | tau | 59.494 | 63 | 0.602 |
problem | likelihood | sd_tau | parameter | statistic | df | p.value | |
---|---|---|---|---|---|---|---|
3 | dense | binomial | 0.5 | theta[10] | 62.925 | 63 | 0.479 |
4 | dense | binomial | 0.5 | theta[1] | 69.709 | 63 | 0.262 |
5 | dense | binomial | 0.5 | theta[2] | 49.830 | 63 | 0.886 |
6 | dense | binomial | 0.5 | theta[3] | 56.115 | 63 | 0.718 |
7 | dense | binomial | 0.5 | theta[4] | 59.789 | 63 | 0.591 |
8 | dense | binomial | 0.5 | theta[5] | 65.203 | 63 | 0.400 |
9 | dense | binomial | 0.5 | theta[6] | 73.779 | 63 | 0.166 |
10 | dense | binomial | 0.5 | theta[7] | 75.750 | 63 | 0.130 |
11 | dense | binomial | 0.5 | theta[8] | 59.763 | 63 | 0.592 |
12 | dense | binomial | 0.5 | theta[9] | 71.859 | 63 | 0.208 |
15 | dense | binomial | 1 | theta[10] | 79.834 | 63 | 0.075 |
16 | dense | binomial | 1 | theta[1] | 59.648 | 63 | 0.597 |
17 | dense | binomial | 1 | theta[2] | 62.016 | 63 | 0.511 |
18 | dense | binomial | 1 | theta[3] | 48.115 | 63 | 0.917 |
19 | dense | binomial | 1 | theta[4] | 52.698 | 63 | 0.819 |
20 | dense | binomial | 1 | theta[5] | 57.254 | 63 | 0.680 |
21 | dense | binomial | 1 | theta[6] | 66.970 | 63 | 0.343 |
22 | dense | binomial | 1 | theta[7] | 88.947 | 63 | 0.017 |
23 | dense | binomial | 1 | theta[8] | 53.261 | 63 | 0.804 |
24 | dense | binomial | 1 | theta[9] | 71.706 | 63 | 0.211 |
27 | dense | gaussian | 0.5 | theta[10] | 66.317 | 63 | 0.363 |
28 | dense | gaussian | 0.5 | theta[1] | 54.208 | 63 | 0.777 |
29 | dense | gaussian | 0.5 | theta[2] | 75.597 | 63 | 0.133 |
30 | dense | gaussian | 0.5 | theta[3] | 44.134 | 63 | 0.966 |
31 | dense | gaussian | 0.5 | theta[4] | 54.784 | 63 | 0.760 |
32 | dense | gaussian | 0.5 | theta[5] | 66.854 | 63 | 0.346 |
33 | dense | gaussian | 0.5 | theta[6] | 63.027 | 63 | 0.475 |
34 | dense | gaussian | 0.5 | theta[7] | 51.046 | 63 | 0.860 |
35 | dense | gaussian | 0.5 | theta[8] | 64.166 | 63 | 0.435 |
36 | dense | gaussian | 0.5 | theta[9] | 50.829 | 63 | 0.865 |
39 | dense | gaussian | 1 | theta[10] | 79.616 | 63 | 0.077 |
40 | dense | gaussian | 1 | theta[1] | 51.917 | 63 | 0.839 |
41 | dense | gaussian | 1 | theta[2] | 68.928 | 63 | 0.284 |
42 | dense | gaussian | 1 | theta[3] | 69.274 | 63 | 0.274 |
43 | dense | gaussian | 1 | theta[4] | 64.243 | 63 | 0.433 |
44 | dense | gaussian | 1 | theta[5] | 44.442 | 63 | 0.963 |
45 | dense | gaussian | 1 | theta[6] | 80.691 | 63 | 0.066 |
46 | dense | gaussian | 1 | theta[7] | 52.326 | 63 | 0.829 |
47 | dense | gaussian | 1 | theta[8] | 59.123 | 63 | 0.615 |
48 | dense | gaussian | 1 | theta[9] | 71.437 | 63 | 0.218 |
51 | dense | poisson | 0.5 | theta[10] | 66.995 | 63 | 0.342 |
52 | dense | poisson | 0.5 | theta[1] | 61.760 | 63 | 0.521 |
53 | dense | poisson | 0.5 | theta[2] | 63.642 | 63 | 0.454 |
54 | dense | poisson | 0.5 | theta[3] | 65.382 | 63 | 0.394 |
55 | dense | poisson | 0.5 | theta[4] | 72.870 | 63 | 0.185 |
56 | dense | poisson | 0.5 | theta[5] | 69.760 | 63 | 0.261 |
57 | dense | poisson | 0.5 | theta[6] | 63.898 | 63 | 0.445 |
58 | dense | poisson | 0.5 | theta[7] | 61.120 | 63 | 0.544 |
59 | dense | poisson | 0.5 | theta[8] | 81.203 | 63 | 0.061 |
60 | dense | poisson | 0.5 | theta[9] | 62.797 | 63 | 0.484 |
63 | dense | poisson | 1 | theta[10] | 57.626 | 63 | 0.668 |
64 | dense | poisson | 1 | theta[1] | 54.938 | 63 | 0.755 |
65 | dense | poisson | 1 | theta[2] | 57.114 | 63 | 0.685 |
66 | dense | poisson | 1 | theta[3] | 81.946 | 63 | 0.055 |
67 | dense | poisson | 1 | theta[4] | 68.224 | 63 | 0.304 |
68 | dense | poisson | 1 | theta[5] | 82.010 | 63 | 0.054 |
69 | dense | poisson | 1 | theta[6] | 53.222 | 63 | 0.805 |
70 | dense | poisson | 1 | theta[7] | 87.322 | 63 | 0.023 |
71 | dense | poisson | 1 | theta[8] | 59.776 | 63 | 0.592 |
72 | dense | poisson | 1 | theta[9] | 80.909 | 63 | 0.064 |
75 | sparse | binomial | 0.5 | theta[1] | 62.963 | 63 | 0.478 |
76 | sparse | binomial | 0.5 | theta[2] | 62.874 | 63 | 0.481 |
79 | sparse | binomial | 1 | theta[1] | 67.456 | 63 | 0.327 |
80 | sparse | binomial | 1 | theta[2] | 74.650 | 63 | 0.150 |
83 | sparse | gaussian | 0.5 | theta[1] | 69.837 | 63 | 0.259 |
84 | sparse | gaussian | 0.5 | theta[2] | 74.342 | 63 | 0.155 |
87 | sparse | gaussian | 1 | theta[1] | 64.230 | 63 | 0.433 |
88 | sparse | gaussian | 1 | theta[2] | 67.814 | 63 | 0.317 |
91 | sparse | poisson | 0.5 | theta[1] | 65.434 | 63 | 0.392 |
92 | sparse | poisson | 0.5 | theta[2] | 30.374 | 63 | 1.000 |
95 | sparse | poisson | 1 | theta[1] | 73.664 | 63 | 0.169 |
96 | sparse | poisson | 1 | theta[2] | 55.002 | 63 | 0.753 |
## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.6 LTS
##
## Matrix products: default
## BLAS: /usr/lib/libblas/libblas.so.3.6.0
## LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
##
## locale:
## [1] C
##
## attached base packages:
## [1] tools stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] purrr_0.3.3 rstan_2.19.2 StanHeaders_2.19.0
## [4] RBesT_1.6-0 testthat_2.2.1 Rcpp_1.0.2
## [7] devtools_2.2.1 usethis_1.5.1 ggplot2_3.2.1
## [10] broom_0.5.2 tidyr_1.0.0 dplyr_0.8.3
## [13] assertthat_0.2.1 knitr_1.25 rmarkdown_1.16
##
## loaded via a namespace (and not attached):
## [1] mvtnorm_1.0-11 lattice_0.20-38 prettyunits_1.0.2
## [4] ps_1.3.0 zeallot_0.1.0 rprojroot_1.3-2
## [7] digest_0.6.21 plyr_1.8.4 R6_2.4.0
## [10] ggridges_0.5.1 backports_1.1.5 stats4_3.6.1
## [13] evaluate_0.14 highr_0.8 pillar_1.4.2
## [16] rlang_0.4.0 lazyeval_0.2.2 rstudioapi_0.10
## [19] callr_3.3.2 checkmate_1.9.4 labeling_0.3
## [22] desc_1.2.0 stringr_1.4.0 loo_2.1.0
## [25] munsell_0.5.0 compiler_3.6.1 xfun_0.10
## [28] pkgconfig_2.0.3 pkgbuild_1.0.6 htmltools_0.4.0
## [31] tidyselect_0.2.5 tibble_2.1.3 gridExtra_2.3
## [34] codetools_0.2-16 matrixStats_0.55.0 crayon_1.3.4
## [37] withr_2.1.2 grid_3.6.1 nlme_3.1-141
## [40] gtable_0.3.0 lifecycle_0.1.0 magrittr_1.5
## [43] scales_1.0.0 cli_1.1.0 stringi_1.4.3
## [46] reshape2_1.4.3 fs_1.3.1 remotes_2.1.0
## [49] ellipsis_0.3.0 generics_0.0.2 vctrs_0.2.0
## [52] Formula_1.2-3 glue_1.3.1 processx_3.4.1
## [55] pkgload_1.0.2 parallel_3.6.1 yaml_2.2.0
## [58] inline_0.3.15 colorspace_1.4-1 sessioninfo_1.1.1
## [61] bayesplot_1.7.0 memoise_1.1.0