library(MultiAssayExperiment)
library(HDF5Array)
library(SummarizedExperiment)
The HDF5Array
package provides an on-disk representation of large datasets
without the need to load them into memory. Convenient lazy evaluation
operations allow the user to manipulate such large data files based on
metadata. The DelayedMatrix
class in the DelayedArray
package provides a
way to connect to a large matrix that is stored on disk.
First, we create a small matrix for constructing the DelayedMatrix
class.
smallMatrix <- matrix(rnorm(10e5), ncol = 20)
We add rownames and column names to the matrix object for compatibility with
the MultiAssayExperiment
representation.
rownames(smallMatrix) <- paste0("GENE", seq_len(nrow(smallMatrix)))
colnames(smallMatrix) <- paste0("SampleID", seq_len(ncol(smallMatrix)))
Here we use the DelayedArray
constructor function to create a
DelayedMatrix
object.
smallMatrix <- DelayedArray(smallMatrix)
class(smallMatrix)
## [1] "DelayedMatrix"
## attr(,"package")
## [1] "DelayedArray"
# show method
smallMatrix
## <50000 x 20> DelayedMatrix object of type "double":
## SampleID1 SampleID2 SampleID3 ... SampleID19 SampleID20
## GENE1 0.3350712 0.7510804 0.9290145 . 0.7719760 0.7344552
## GENE2 -0.7188797 -1.2358769 -0.9130142 . 0.9046039 -0.2613202
## GENE3 1.1267346 0.7965048 0.9632856 . -0.7123481 -0.9323978
## GENE4 0.3997137 0.2712341 0.1701698 . 0.8135351 -1.5151827
## GENE5 0.1558288 -0.3455871 -1.6433194 . 0.7657454 -1.0124187
## ... . . . . . .
## GENE49996 -0.14075351 0.31541298 -0.98104291 . 1.38964092 0.39526507
## GENE49997 0.49074194 0.20071296 0.54699765 . 0.26067249 1.05889010
## GENE49998 0.98771488 0.04740033 -0.91290731 . -0.07049434 -0.47569889
## GENE49999 1.30786036 -0.73897412 -0.01204778 . -0.69686178 1.98514483
## GENE50000 -0.09547939 0.69539439 -0.73391931 . 0.88412926 -0.84657130
dim(smallMatrix)
## [1] 50000 20
Finally, the rhdf5
package stores dimnames
in a standard location.
In order to make use of this functionality, we would use writeHDF5Array
with the with.dimnames
argument:
testh5 <- tempfile(fileext = ".h5")
writeHDF5Array(smallMatrix, filepath = testh5, name = "smallMatrix",
with.dimnames = TRUE)
## <50000 x 20> HDF5Matrix object of type "double":
## SampleID1 SampleID2 SampleID3 ... SampleID19 SampleID20
## GENE1 0.3350712 0.7510804 0.9290145 . 0.7719760 0.7344552
## GENE2 -0.7188797 -1.2358769 -0.9130142 . 0.9046039 -0.2613202
## GENE3 1.1267346 0.7965048 0.9632856 . -0.7123481 -0.9323978
## GENE4 0.3997137 0.2712341 0.1701698 . 0.8135351 -1.5151827
## GENE5 0.1558288 -0.3455871 -1.6433194 . 0.7657454 -1.0124187
## ... . . . . . .
## GENE49996 -0.14075351 0.31541298 -0.98104291 . 1.38964092 0.39526507
## GENE49997 0.49074194 0.20071296 0.54699765 . 0.26067249 1.05889010
## GENE49998 0.98771488 0.04740033 -0.91290731 . -0.07049434 -0.47569889
## GENE49999 1.30786036 -0.73897412 -0.01204778 . -0.69686178 1.98514483
## GENE50000 -0.09547939 0.69539439 -0.73391931 . 0.88412926 -0.84657130
To see the file structure we use h5ls
:
h5ls(testh5)
## group name otype dclass dim
## 0 / .smallMatrix_dimnames H5I_GROUP
## 1 /.smallMatrix_dimnames 1 H5I_DATASET STRING 50000
## 2 /.smallMatrix_dimnames 2 H5I_DATASET STRING 20
## 3 / smallMatrix H5I_DATASET FLOAT 50000 x 20
Note that a large matrix from an HDF5 file can also be loaded using the
HDF5ArraySeed
and DelayedArray
functions.
hdf5Data <- HDF5ArraySeed(file = testh5, name = "smallMatrix")
newDelayedMatrix <- DelayedArray(hdf5Data)
class(newDelayedMatrix)
## [1] "HDF5Matrix"
## attr(,"package")
## [1] "HDF5Array"
newDelayedMatrix
## <50000 x 20> HDF5Matrix object of type "double":
## SampleID1 SampleID2 SampleID3 ... SampleID19 SampleID20
## GENE1 0.3350712 0.7510804 0.9290145 . 0.7719760 0.7344552
## GENE2 -0.7188797 -1.2358769 -0.9130142 . 0.9046039 -0.2613202
## GENE3 1.1267346 0.7965048 0.9632856 . -0.7123481 -0.9323978
## GENE4 0.3997137 0.2712341 0.1701698 . 0.8135351 -1.5151827
## GENE5 0.1558288 -0.3455871 -1.6433194 . 0.7657454 -1.0124187
## ... . . . . . .
## GENE49996 -0.14075351 0.31541298 -0.98104291 . 1.38964092 0.39526507
## GENE49997 0.49074194 0.20071296 0.54699765 . 0.26067249 1.05889010
## GENE49998 0.98771488 0.04740033 -0.91290731 . -0.07049434 -0.47569889
## GENE49999 1.30786036 -0.73897412 -0.01204778 . -0.69686178 1.98514483
## GENE50000 -0.09547939 0.69539439 -0.73391931 . 0.88412926 -0.84657130
DelayedMatrix
with MultiAssayExperiment
A DelayedMatrix
alone conforms to the MultiAssayExperiment
API requirements.
Shown below, the DelayedMatrix
can be put into a named list
and passed into
the MultiAssayExperiment
constructor function.
HDF5MAE <- MultiAssayExperiment(experiments = list(smallMatrix = smallMatrix))
sampleMap(HDF5MAE)
## DataFrame with 20 rows and 3 columns
## assay primary colname
## <factor> <character> <character>
## 1 smallMatrix SampleID1 SampleID1
## 2 smallMatrix SampleID2 SampleID2
## 3 smallMatrix SampleID3 SampleID3
## 4 smallMatrix SampleID4 SampleID4
## 5 smallMatrix SampleID5 SampleID5
## ... ... ... ...
## 16 smallMatrix SampleID16 SampleID16
## 17 smallMatrix SampleID17 SampleID17
## 18 smallMatrix SampleID18 SampleID18
## 19 smallMatrix SampleID19 SampleID19
## 20 smallMatrix SampleID20 SampleID20
colData(HDF5MAE)
## DataFrame with 20 rows and 0 columns
SummarizedExperiment
with DelayedMatrix
backendA more information rich DelayedMatrix
can be created when used in conjunction
with the SummarizedExperiment
class and it can even include rowRanges
.
The flexibility of the MultiAssayExperiment
API supports classes with
minimal requirements. Additionally, this SummarizedExperiment
with the
DelayedMatrix
backend can be part of a bigger MultiAssayExperiment
object.
Below is a minimal example of how this would work:
HDF5SE <- SummarizedExperiment(assays = smallMatrix)
assay(HDF5SE)
## <50000 x 20> DelayedMatrix object of type "double":
## SampleID1 SampleID2 SampleID3 ... SampleID19 SampleID20
## GENE1 0.3350712 0.7510804 0.9290145 . 0.7719760 0.7344552
## GENE2 -0.7188797 -1.2358769 -0.9130142 . 0.9046039 -0.2613202
## GENE3 1.1267346 0.7965048 0.9632856 . -0.7123481 -0.9323978
## GENE4 0.3997137 0.2712341 0.1701698 . 0.8135351 -1.5151827
## GENE5 0.1558288 -0.3455871 -1.6433194 . 0.7657454 -1.0124187
## ... . . . . . .
## GENE49996 -0.14075351 0.31541298 -0.98104291 . 1.38964092 0.39526507
## GENE49997 0.49074194 0.20071296 0.54699765 . 0.26067249 1.05889010
## GENE49998 0.98771488 0.04740033 -0.91290731 . -0.07049434 -0.47569889
## GENE49999 1.30786036 -0.73897412 -0.01204778 . -0.69686178 1.98514483
## GENE50000 -0.09547939 0.69539439 -0.73391931 . 0.88412926 -0.84657130
MultiAssayExperiment(list(HDF5SE = HDF5SE))
## A MultiAssayExperiment object of 1 listed
## experiment with a user-defined name and respective class.
## Containing an ExperimentList class object of length 1:
## [1] HDF5SE: SummarizedExperiment with 50000 rows and 20 columns
## Functionality:
## experiments() - obtain the ExperimentList instance
## colData() - the primary/phenotype DataFrame
## sampleMap() - the sample coordination DataFrame
## `$`, `[`, `[[` - extract colData columns, subset, or experiment
## *Format() - convert into a long or wide DataFrame
## assays() - convert ExperimentList to a SimpleList of matrices
## exportClass() - save data to flat files
Additional scenarios are currently in development where an HDF5Matrix
is
hosted remotely. Many opportunities exist when considering on-disk and off-disk
representations of data with MultiAssayExperiment
.
sessionInfo()
## R version 4.4.1 (2024-06-14)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 22.04.4 LTS
##
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.20-bioc/R/lib/libRblas.so
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_GB LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## time zone: America/New_York
## tzcode source: system (glibc)
##
## attached base packages:
## [1] stats4 stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] HDF5Array_1.33.6 rhdf5_2.49.0
## [3] DelayedArray_0.31.11 SparseArray_1.5.31
## [5] S4Arrays_1.5.7 abind_1.4-5
## [7] Matrix_1.7-0 survminer_0.4.9
## [9] ggpubr_0.6.0 ggplot2_3.5.1
## [11] survival_3.7-0 UpSetR_1.4.0
## [13] RaggedExperiment_1.29.2 MultiAssayExperiment_1.31.5
## [15] SummarizedExperiment_1.35.1 Biobase_2.65.0
## [17] GenomicRanges_1.57.1 GenomeInfoDb_1.41.1
## [19] IRanges_2.39.2 S4Vectors_0.43.2
## [21] BiocGenerics_0.51.0 MatrixGenerics_1.17.0
## [23] matrixStats_1.3.0 BiocStyle_2.33.1
##
## loaded via a namespace (and not attached):
## [1] gridExtra_2.3 rlang_1.1.4 magrittr_2.0.3
## [4] compiler_4.4.1 vctrs_0.6.5 reshape2_1.4.4
## [7] stringr_1.5.1 pkgconfig_2.0.3 crayon_1.5.3
## [10] fastmap_1.2.0 backports_1.5.0 magick_2.8.4
## [13] XVector_0.45.0 labeling_0.4.3 KMsurv_0.1-5
## [16] utf8_1.2.4 rmarkdown_2.28 markdown_1.13
## [19] UCSC.utils_1.1.0 tinytex_0.52 purrr_1.0.2
## [22] xfun_0.47 zlibbioc_1.51.1 cachem_1.1.0
## [25] jsonlite_1.8.8 highr_0.11 rhdf5filters_1.17.0
## [28] Rhdf5lib_1.27.0 broom_1.0.6 R6_2.5.1
## [31] bslib_0.8.0 stringi_1.8.4 car_3.1-2
## [34] jquerylib_0.1.4 Rcpp_1.0.13 bookdown_0.40
## [37] knitr_1.48 zoo_1.8-12 R.utils_2.12.3
## [40] BiocBaseUtils_1.7.1 splines_4.4.1 R.cache_0.16.0
## [43] tidyselect_1.2.1 yaml_2.3.10 ggtext_0.1.2
## [46] lattice_0.22-6 tibble_3.2.1 plyr_1.8.9
## [49] withr_3.0.1 evaluate_0.24.0 xml2_1.3.6
## [52] survMisc_0.5.6 pillar_1.9.0 BiocManager_1.30.24
## [55] carData_3.0-5 generics_0.1.3 commonmark_1.9.1
## [58] munsell_0.5.1 scales_1.3.0 xtable_1.8-4
## [61] glue_1.7.0 tools_4.4.1 data.table_1.15.4
## [64] ggsignif_0.6.4 grid_4.4.1 tidyr_1.3.1
## [67] colorspace_2.1-1 GenomeInfoDbData_1.2.12 cli_3.6.3
## [70] km.ci_0.5-6 fansi_1.0.6 dplyr_1.1.4
## [73] gtable_0.3.5 R.methodsS3_1.8.2 rstatix_0.7.2
## [76] R.rsp_0.46.0 sass_0.4.9 digest_0.6.37
## [79] farver_2.1.2 htmltools_0.5.8.1 R.oo_1.26.0
## [82] lifecycle_1.0.4 httr_1.4.7 gridtext_0.1.5