loo_moment_match integer overflow

[trinhdhk]

Problem: Integer Overflow when using loo_moment_match()

What I did:

• loo_moment_match for a stanfit model, following the vignette here, with all default parameters and functions.
• Number of cores for parallelisation: 10

Expected behaviour: return the value with no exceptions

Actual behaviour: return an error message

Error in while (t < nrow(acov) - 5 && !is.nan(rho_hat_even + rho_hat_odd) &&  : 
  missing value where TRUE/FALSE needed
In addition: Warning message:
In N * N : NAs produced by integer overflow

Is the problem random? No

What I tried:

• Reduce the number of cores didn't work
• Remove the high-dimensional random effect parameter didn't work, the parameter is required

Session Info

R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
 [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] bayesplot_1.7.2      rstan_2.19.3         ggplot2_3.3.2        StanHeaders_2.21.0-5
[5] magrittr_1.5         data.table_1.12.8   

loaded via a namespace (and not attached):
 [1] tinytex_0.24       tidyselect_1.1.0   xfun_0.15          purrr_0.3.4        colorspace_1.4-1  
 [6] vctrs_0.3.1        generics_0.0.2     stats4_4.0.2       loo_2.3.1          rlang_0.4.7       
[11] pkgbuild_1.1.0     pillar_1.4.6       glue_1.4.1         withr_2.2.0        RColorBrewer_1.1-2
[16] readxl_1.3.1       matrixStats_0.56.0 lifecycle_0.2.0    plyr_1.8.6         munsell_0.5.0     
[21] gtable_0.3.0       cellranger_1.1.0   codetools_0.2-16   inline_0.3.15      GGally_2.0.0      
[26] callr_3.4.3        ps_1.3.3           parallel_4.0.2     fansi_0.4.1        Rcpp_1.0.4.6      
[31] backports_1.1.8    checkmate_2.0.0    scales_1.1.1       ggmcmc_1.4.1       RcppParallel_5.0.1
[36] gridExtra_2.3      processx_3.4.3     dplyr_1.0.0        grid_4.0.2         cli_2.0.2         
[41] tools_4.0.2        tibble_3.0.3       crayon_1.3.4       tidyr_1.1.0        pkgconfig_2.0.3   
[46] ellipsis_0.3.1     prettyunits_1.1.1  ggridges_0.5.2     lubridate_1.7.9    assertthat_0.2.1  
[51] reshape_0.8.8      rstudioapi_0.11    R6_2.4.1           compiler_4.0.2    

[paul-buerkner]

Hi, thank you for posting this issue. It is holiday season so it will take some time until we will check out this issue in detail. For this purpose, it would be important to have a minimally fully reproducible example, that is, code we can just copy and it returns the error.

[avehtari]

This is known problem which was fixed in rhat paper code and the fixed version is also in posterior package. The problem is that when I fixed that, I didn't realize I should search all our repos for copy-pasted code. The problematic code is in autocovariance function in effective_sample_sizes.R. The code has

  N <- length(y)
...
  ac <- Re(ac)[1:N] / (N * N * 2)

length returns integer and N*N returns also an integer, and thus N * N overflows when N>46340, where 46340 is floor(sqrt(.Machine$integer.max)). The simple fix is to change N * N to N^2 as ^ returns numeric type instead of integer.

So this can be fixed by changing the line
https://github.com/stan-dev/loo/blob/master/R/effective_sample_sizes.R#L289
to be the same as
https://github.com/stan-dev/posterior/blob/master/R/convergence.R#L359

[jgabry]

@avehtari Good catch. We can fix this in loo in the short term, but once posterior is on CRAN we can just call posterior::autocovariance() and avoid the code duplication / copy-pasting.

[jgabry]

Going to close this now that #155 is merged but will open a separate issue to remind us to switch to posterior::autocovariance() once posterior is on CRAN.