Two-sample tests based on the 2-Wasserstein distance

Testing procedures

The waddR package provides two testing procedures using the 2-Wasserstein distance to test whether two distributions \(F_A\) and \(F_B\) given in the form of samples are different by specifically testing the null hypothesis \(H_0: F_A = F_B\) against the alternative \(H_1: F_A \neq F_B\).

The first, semi-parametric (SP), procedure uses a test based on permutations combined with a generalized Pareto distribution approximation to estimate small p-values accurately.

The second procedure (ASY) uses a test based on asymptotic theory which is valid only if the samples can be assumed to come from continuous distributions.

Examples

To demonstrate the capabilities of the testing procedures, we consider models based on normal distributions here. We exemplarily construct three cases in which two distributions (samples) differ with respect to location, size, and shape, respectively, and one case without a difference. For convenience, we focus on the p-value, the value of the (squared) 2-Wasserstein distance, and the fractions of the location, size, and shape terms (in %) with respect to the (squared) 2-Wasserstein distance here, while the functions also provide additional output.

library(waddR)

spec.output<-c("pval","d.wass^2","perc.loc","perc.size","perc.shape")

We start with an example, in which the two distributions (samples) are supposed to only differ with respect to the location, and show the results for the two testing procedures (SP and ASY).

Note that the semi-parametric method “SP” with a permutation number of 10000 is used by default in wasserstein.test if nothing else is specified. The permutation procedure in the “SP” method uses a random group assignment step, and to obtain reproducible results, a seed can be set from the user environment before calling wasserstein.test.

set.seed(24)
ctrl<-rnorm(300,0,1)
dd1<-rnorm(300,1,1)
set.seed(32)
wasserstein.test(ctrl,dd1,method="SP",permnum=10000)[spec.output]
#>       pval   d.wass^2   perc.loc  perc.size perc.shape 
#>   0.000000   1.162487  98.100000   0.070000   1.830000
wasserstein.test(ctrl,dd1,method="ASY")[spec.output]
#>       pval   d.wass^2   perc.loc  perc.size perc.shape 
#>   0.000000   1.162487  98.100000   0.070000   1.830000

We obtain a very low p-value, clearly pointing at the existence of a difference, and see that differences with respect to location make up by far the major part of the 2-Wasserstein distance.

Analogously, we look at a case in which the two distributions (samples) are supposed to only differ with respect to the size.

set.seed(24)
ctrl<-rnorm(300,0,1)
dd2<-rnorm(300,0,2)
set.seed(32)
wasserstein.test(ctrl,dd2)[spec.output]
#>         pval     d.wass^2     perc.loc    perc.size   perc.shape 
#> 2.931697e-12 1.148397e+00 1.200000e-01 9.618000e+01 3.700000e+00
wasserstein.test(ctrl,dd2,method="ASY")[spec.output]
#>       pval   d.wass^2   perc.loc  perc.size perc.shape 
#>   0.000005   1.148397   0.120000  96.180000   3.700000

Similarly, we consider an example in which the two distributions (samples) are supposed to only differ with respect to the shape.

set.seed(24)
ctrl<-rnorm(300,6.5,sqrt(13.25))
sam1<-rnorm(300,3,1)
sam2<-rnorm(300,10,1)
dd3<-sapply(1:300, 
              function(n) {
                sample(c(sam1[n],sam2[n]),1,prob=c(0.5, 0.5))})
set.seed(32)                
wasserstein.test(ctrl,dd3)[spec.output]
#>         pval     d.wass^2     perc.loc    perc.size   perc.shape 
#> 2.633310e-06 2.573153e+00 1.164000e+01 0.000000e+00 8.836000e+01
wasserstein.test(ctrl,dd3,method="ASY")[spec.output]
#>       pval   d.wass^2   perc.loc  perc.size perc.shape 
#>   0.000006   2.573153  11.640000   0.000000  88.360000

Finally, we show an example in which the two distributions (samples) are supposed to be equal. We obtain a high p-value, indicating that the null hypothesis cannot be rejected.

set.seed(24)
ctrl<-rnorm(300,0,1)
nodd<-rnorm(300,0,1)
set.seed(32)
wasserstein.test(ctrl,nodd)[spec.output]
#>        pval    d.wass^2    perc.loc   perc.size  perc.shape 
#>  0.48560000  0.02772574 17.06000000  3.01000000 79.93000000
wasserstein.test(ctrl,nodd,method="ASY")[spec.output]
#>        pval    d.wass^2    perc.loc   perc.size  perc.shape 
#>  0.73892700  0.02772574 17.06000000  3.01000000 79.93000000

See also

Session info

sessionInfo()
#> R version 4.1.1 (2021-08-10)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 20.04.3 LTS
#> 
#> Matrix products: default
#> BLAS:   /home/biocbuild/bbs-3.14-bioc/R/lib/libRblas.so
#> LAPACK: /home/biocbuild/bbs-3.14-bioc/R/lib/libRlapack.so
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_GB              LC_COLLATE=C              
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats4    stats     graphics  grDevices utils     datasets  methods  
#> [8] base     
#> 
#> other attached packages:
#>  [1] BiocFileCache_2.2.0         dbplyr_2.1.1               
#>  [3] SingleCellExperiment_1.16.0 SummarizedExperiment_1.24.0
#>  [5] Biobase_2.54.0              GenomicRanges_1.46.0       
#>  [7] GenomeInfoDb_1.30.0         IRanges_2.28.0             
#>  [9] S4Vectors_0.32.0            BiocGenerics_0.40.0        
#> [11] MatrixGenerics_1.6.0        matrixStats_0.61.0         
#> [13] waddR_1.8.0                
#> 
#> loaded via a namespace (and not attached):
#>  [1] httr_1.4.2             sass_0.4.0             bit64_4.0.5           
#>  [4] jsonlite_1.7.2         splines_4.1.1          bslib_0.3.1           
#>  [7] assertthat_0.2.1       blob_1.2.2             GenomeInfoDbData_1.2.7
#> [10] yaml_2.2.1             pillar_1.6.4           RSQLite_2.2.8         
#> [13] lattice_0.20-45        glue_1.4.2             digest_0.6.28         
#> [16] XVector_0.34.0         minqa_1.2.4            htmltools_0.5.2       
#> [19] Matrix_1.3-4           pkgconfig_2.0.3        zlibbioc_1.40.0       
#> [22] purrr_0.3.4            BiocParallel_1.28.0    lme4_1.1-27.1         
#> [25] arm_1.12-2             tibble_3.1.5           generics_0.1.1        
#> [28] ellipsis_0.3.2         withr_2.4.2            cachem_1.0.6          
#> [31] magrittr_2.0.1         crayon_1.4.1           memoise_2.0.0         
#> [34] evaluate_0.14          fansi_0.5.0            nlme_3.1-153          
#> [37] MASS_7.3-54            tools_4.1.1            lifecycle_1.0.1       
#> [40] stringr_1.4.0          DelayedArray_0.20.0    eva_0.2.6             
#> [43] compiler_4.1.1         jquerylib_0.1.4        rlang_0.4.12          
#> [46] grid_4.1.1             RCurl_1.98-1.5         nloptr_1.2.2.2        
#> [49] rappdirs_0.3.3         bitops_1.0-7           rmarkdown_2.11        
#> [52] boot_1.3-28            abind_1.4-5            DBI_1.1.1             
#> [55] curl_4.3.2             R6_2.5.1               knitr_1.36            
#> [58] dplyr_1.0.7            fastmap_1.1.0          bit_4.0.4             
#> [61] utf8_1.2.2             filelock_1.0.2         stringi_1.7.5         
#> [64] parallel_4.1.1         Rcpp_1.0.7             vctrs_0.3.8           
#> [67] tidyselect_1.1.1       xfun_0.27              coda_0.19-4