1 Introduction

HubPub provides users with functionality to help with the Bioconductor Hub structures. The package provides the ability to create a skeleton of a Hub style package that the user can then populate with the necessary information. There are also functions to help add resources to the Hub pacakge metadata files as well as publish data to the Bioconductor S3 bucket.

2 Installation

Install the most recent version from Bioconductor:

if(!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install("HubPub")

Then load HubPub:

library(HubPub)

3 HubPub

3.1 Creating a Hub styled package

The create_pkg() function creates the skeleton of a package that follows the guidelines for a Bioconductor Hub type package. More information about what are the requirements and content for a Hub style package the developer can look at the “Creating A Hub Package” vignette from this package.

create_pkg() requires a path to where the packages is to be created and the type of package that should be created (“AnnotationHub” or “ExperimentHub”). There is also a variable use_git that indicates if the package should be set up with git (default is TRUE).

NOTE: This function is intended for a developer that has not created the package yet. If the package has already been created, then this function will not benefit the developer. There are a couple other functions in this package that deal with resources that might be helpful, more on these later in the vignette.

fl <- tempdir()
create_pkg(file.path(fl, "examplePkg"), "ExperimentHub")
#> ✔ Creating '/tmp/Rtmp8rQxmQ/examplePkg/'
#> ✔ Setting active project to '/tmp/Rtmp8rQxmQ/examplePkg'
#> ✔ Creating 'R/'
#> ✔ Writing 'DESCRIPTION'
#> Package: examplePkg
#> Title: What the Package Does (One Line, Title Case)
#> Version: 0.99.0
#> Date: 2023-08-10
#> Authors@R (parsed):
#>     * First Last <first.last@example.com> [aut, cre] (YOUR-ORCID-ID)
#> Description: What the package does (one paragraph).
#> License: Artistic-2.0
#> BugReports: https://support.bioconductor.org/t/examplePkg
#> Imports:
#>     ExperimentHub
#> Suggests:
#>     ExperimentHubData
#> Encoding: UTF-8
#> Roxygen: list(markdown = TRUE)
#> RoxygenNote: 7.2.3
#> biocViews: ExperimentHub
#> ✔ Writing 'NAMESPACE'
#> ✔ Setting active project to '<no active project>'
#> ✔ Setting active project to '/tmp/Rtmp8rQxmQ/examplePkg'
#> ✔ Initialising Git repo
#> ✔ Adding '.Rproj.user', '.Rhistory', '.Rdata', '.httr-oauth', '.DS_Store', '.quarto' to '.gitignore'
#> ✔ Writing 'R/examplePkg-package.R'
#> ✔ Writing 'NEWS.md'
#> ✔ Creating 'man/'
#> ✔ Creating 'inst/scripts/'
#> ✔ Writing 'inst/scripts/make-data.R'
#> ✔ Writing 'inst/scripts/make-metadata.R'
#> ✔ Writing 'R/zzz.R'
#> ✔ Creating 'inst/extdata/'
#> ✔ Adding 'testthat' to Suggests field in DESCRIPTION
#> ✔ Adding '3' to Config/testthat/edition
#> ✔ Creating 'tests/testthat/'
#> ✔ Writing 'tests/testthat.R'
#> • Call `use_test()` to initialize a basic test file and open it for editing.
#> ✔ Writing 'tests/testthat/test_metadata.R'
#> [1] "/tmp/Rtmp8rQxmQ/examplePkg"

Once the package is created the developer can go through and make any changes to the package. For example, the DESCRIPTON file contains very basic requirements but the developer should go back and fill in the ‘Title:’ and ‘Description:’ fields.

3.2 Adding a resource to the metadata file

Another useful function in HubPub is add_resource(). This function can be useful for developers who are creating a new Hub related package or for developers who want to add a new resource to an existing Hub package. The purpose of this function is to add a hub resource to the package metadata.csv file. The function requires the name of the package (or the path to the newly created package) and a named list with the data to be added to the resource. To get the elements and content for this list look at ?hub_metadata. There is also information in the “Creating A Hub Package” vignette from this package.

metadata <- hub_metadata(
    Title = "ENCODE",
    Description = "a test entry",
    BiocVersion = "4.1",
    Genome = NA_character_,
    SourceType = "JSON",
    SourceUrl = "http://www.encodeproject.org",
    SourceVersion = "x.y.z",
    Species = NA_character_,
    TaxonomyId = as.integer(9606),
    Coordinate_1_based = NA,
    DataProvider = "ENCODE Project",
    Maintainer = "tst person <tst@email.com>",
    RDataClass = "Rda",
    DispatchClass = "Rda",
    Location_Prefix = "s3://experimenthub/",
    RDataPath = "ENCODExplorerData/encode_df_lite.rda",
    Tags = "ENCODE:Homo sapiens"
)

add_resource(file.path(fl, "examplePkg"), metadata)
#> Warning: replacing previous import 'utils::findMatches' by
#> 'S4Vectors::findMatches' when loading 'ExperimentHubData'
#> [1] "/tmp/Rtmp8rQxmQ/examplePkg/inst/extdata/metadata.csv"

Then if you want to see what the metadata file looks like you can read in the csv file like the following.

resource <- file.path(fl, "examplePkg", "inst", "extdata", "metadata.csv")
tst <- read.csv(resource)
tst
#>    Title  Description BiocVersion Genome SourceType
#> 1 ENCODE a test entry         4.1     NA       JSON
#>                      SourceUrl SourceVersion Species TaxonomyId
#> 1 http://www.encodeproject.org         x.y.z      NA       9606
#>   Coordinate_1_based   DataProvider                 Maintainer RDataClass
#> 1                 NA ENCODE Project tst person <tst@email.com>        Rda
#>   DispatchClass     Location_Prefix                            RDataPath
#> 1           Rda s3://experimenthub/ ENCODExplorerData/encode_df_lite.rda
#>                  Tags
#> 1 ENCODE:Homo sapiens

3.3 Publishing the resource to AWS S3

The final function in HubPub helps the developer with publishing data resources to an Bioconductor AWS S3. The function utilizes functions for the aws.s3 package to place files or directories on S3. The developer should have already contacted the Bioconductor hubs maintainers to get the necessary credentials to access the bucket. Once the credentials are received the developer should declare them in the system environment before running this function. The function requires a path to the file or name of the directory to be added to the bucket and a name for how the object should be named on the bucket. If adding a directory be sure there are no nested directories and only files.

The below code chunk demonstrates the use of the function using a dummy dataset. It will only work if the necessary global environments have been declared with the hub credentials.

## For publishing directories with multiple files
fl <- tempdir()
utils::write.csv(mtcars, file = file.path(fl, "mtcars1.csv"))
utils::write.csv(mtcars, file = file.path(fl, "mtcars2.csv"))
publish_resource(fl, "test_dir")
#> Warning in publish_resource(fl, "test_dir"): Not all system environment
#> variables are set, do so and rerun function.
#> copy '/tmp/Rtmp8rQxmQ/BiocStyle' to 's3://annotation-contributor/test_dir/BiocStyle'
#> copy '/tmp/Rtmp8rQxmQ/examplePkg' to 's3://annotation-contributor/test_dir/examplePkg'
#> copy '/tmp/Rtmp8rQxmQ/mtcars1.csv' to 's3://annotation-contributor/test_dir/mtcars1.csv'
#> copy '/tmp/Rtmp8rQxmQ/mtcars2.csv' to 's3://annotation-contributor/test_dir/mtcars2.csv'
#> copy '/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fbioc%2Fsrc%2Fcontrib.rds' to 's3://annotation-contributor/test_dir/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fbioc%2Fsrc%2Fcontrib.rds'
#> copy '/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fbooks%2Fsrc%2Fcontrib.rds' to 's3://annotation-contributor/test_dir/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fbooks%2Fsrc%2Fcontrib.rds'
#> copy '/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fdata%2Fannotation%2Fsrc%2Fcontrib.rds' to 's3://annotation-contributor/test_dir/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fdata%2Fannotation%2Fsrc%2Fcontrib.rds'
#> copy '/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fdata%2Fexperiment%2Fsrc%2Fcontrib.rds' to 's3://annotation-contributor/test_dir/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fdata%2Fexperiment%2Fsrc%2Fcontrib.rds'
#> copy '/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fworkflows%2Fsrc%2Fcontrib.rds' to 's3://annotation-contributor/test_dir/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fworkflows%2Fsrc%2Fcontrib.rds'
#> copy '/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fcloud.r-project.org%2Fsrc%2Fcontrib.rds' to 's3://annotation-contributor/test_dir/repos_https%3A%2F%2Fcloud.r-project.org%2Fsrc%2Fcontrib.rds'
#> $`/tmp/Rtmp8rQxmQ/BiocStyle`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/examplePkg`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/mtcars1.csv`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/mtcars2.csv`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fbioc%2Fsrc%2Fcontrib.rds`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fbooks%2Fsrc%2Fcontrib.rds`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fdata%2Fannotation%2Fsrc%2Fcontrib.rds`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fdata%2Fexperiment%2Fsrc%2Fcontrib.rds`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fbioconductor.org%2Fpackages%2F3.17%2Fworkflows%2Fsrc%2Fcontrib.rds`
#> NULL
#> 
#> $`/tmp/Rtmp8rQxmQ/repos_https%3A%2F%2Fcloud.r-project.org%2Fsrc%2Fcontrib.rds`
#> NULL

## For publishing a single file
utils::write.csv(mtcars, file = file.path(fl, "mtcars3.csv"))
publish_resource(file.path(fl, "mtcars3.csv"), "test_dir")
#> Warning in publish_resource(file.path(fl, "mtcars3.csv"), "test_dir"): Not all
#> system environment variables are set, do so and rerun function.
#> copy '/tmp/Rtmp8rQxmQ/mtcars3.csv' to 's3://annotation-contributor/test_dir/mtcars3.csv'
#> $`/tmp/Rtmp8rQxmQ/mtcars3.csv`
#> NULL

4 Session Information

sessionInfo()
#> R version 4.3.1 (2023-06-16)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 22.04.2 LTS
#> 
#> Matrix products: default
#> BLAS:   /home/biocbuild/bbs-3.17-bioc/R/lib/libRblas.so 
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_GB              LC_COLLATE=C              
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> time zone: America/New_York
#> tzcode source: system (glibc)
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] futile.logger_1.4.3 HubPub_1.8.1        BiocStyle_2.28.0   
#> 
#> loaded via a namespace (and not attached):
#>   [1] sys_3.4.2                     rstudioapi_0.15.0            
#>   [3] jsonlite_1.8.7                magrittr_2.0.3               
#>   [5] GenomicFeatures_1.52.1        rmarkdown_2.23               
#>   [7] fs_1.6.3                      BiocIO_1.10.0                
#>   [9] zlibbioc_1.46.0               vctrs_0.6.3                  
#>  [11] memoise_2.0.1                 Rsamtools_2.16.0             
#>  [13] RCurl_1.98-1.12               askpass_1.1                  
#>  [15] base64enc_0.1-3               htmltools_0.5.5              
#>  [17] S4Arrays_1.0.5                usethis_2.2.2                
#>  [19] progress_1.2.2                lambda.r_1.2.4               
#>  [21] AnnotationHub_3.8.0           curl_5.0.1                   
#>  [23] sass_0.4.7                    bslib_0.5.0                  
#>  [25] desc_1.4.2                    testthat_3.1.10              
#>  [27] futile.options_1.0.1          cachem_1.0.8                 
#>  [29] available_1.1.0               GenomicAlignments_1.36.0     
#>  [31] whisker_0.4.1                 mime_0.12                    
#>  [33] lifecycle_1.0.3               pkgconfig_2.0.3              
#>  [35] Matrix_1.6-0                  R6_2.5.1                     
#>  [37] fastmap_1.1.1                 BiocCheck_1.36.1             
#>  [39] GenomeInfoDbData_1.2.10       MatrixGenerics_1.12.3        
#>  [41] shiny_1.7.4.1                 digest_0.6.33                
#>  [43] AnnotationDbi_1.62.2          S4Vectors_0.38.1             
#>  [45] OrganismDbi_1.42.0            rprojroot_2.0.3              
#>  [47] ExperimentHub_2.8.1           aws.signature_0.6.0          
#>  [49] GenomicRanges_1.52.0          RSQLite_2.3.1                
#>  [51] filelock_1.0.2                fansi_1.0.4                  
#>  [53] httr_1.4.6                    abind_1.4-5                  
#>  [55] compiler_4.3.1                bit64_4.0.5                  
#>  [57] withr_2.5.0                   biocViews_1.68.1             
#>  [59] BiocParallel_1.34.2           DBI_1.1.3                    
#>  [61] R.utils_2.12.2                biomaRt_2.56.1               
#>  [63] openssl_2.1.0                 rappdirs_0.3.3               
#>  [65] DelayedArray_0.26.7           rjson_0.2.21                 
#>  [67] tools_4.3.1                   interactiveDisplayBase_1.38.0
#>  [69] httpuv_1.6.11                 R.oo_1.25.0                  
#>  [71] glue_1.6.2                    restfulr_0.0.15              
#>  [73] R.cache_0.16.0                promises_1.2.1               
#>  [75] grid_4.3.1                    stringdist_0.9.10            
#>  [77] generics_0.1.3                R.methodsS3_1.8.2            
#>  [79] hms_1.1.3                     xml2_1.3.5                   
#>  [81] utf8_1.2.3                    XVector_0.40.0               
#>  [83] BiocGenerics_0.46.0           BiocVersion_3.17.1           
#>  [85] pillar_1.9.0                  stringr_1.5.0                
#>  [87] later_1.3.1                   dplyr_1.1.2                  
#>  [89] BiocFileCache_2.8.0           lattice_0.21-8               
#>  [91] AnnotationHubData_1.30.0      rtracklayer_1.60.0           
#>  [93] bit_4.0.5                     RBGL_1.76.0                  
#>  [95] tidyselect_1.2.0              Biostrings_2.68.1            
#>  [97] knitr_1.43                    biocthis_1.10.3              
#>  [99] bookdown_0.35                 IRanges_2.34.1               
#> [101] SummarizedExperiment_1.30.2   stats4_4.3.1                 
#> [103] xfun_0.40                     Biobase_2.60.0               
#> [105] credentials_1.3.2             brio_1.1.3                   
#> [107] matrixStats_1.0.0             stringi_1.7.12               
#> [109] yaml_2.3.7                    evaluate_0.21                
#> [111] codetools_0.2-19              tibble_3.2.1                 
#> [113] BiocManager_1.30.22           graph_1.78.0                 
#> [115] cli_3.6.1                     xtable_1.8-4                 
#> [117] jquerylib_0.1.4               styler_1.10.1                
#> [119] roxygen2_7.2.3                Rcpp_1.0.11                  
#> [121] GenomeInfoDb_1.36.1           gert_1.9.3                   
#> [123] dbplyr_2.3.3                  png_0.1-8                    
#> [125] RUnit_0.4.32                  XML_3.99-0.14                
#> [127] parallel_4.3.1                ellipsis_0.3.2               
#> [129] blob_1.2.4                    prettyunits_1.1.1            
#> [131] aws.s3_0.3.21                 AnnotationForge_1.42.2       
#> [133] bitops_1.0-7                  ExperimentHubData_1.26.0     
#> [135] purrr_1.0.2                   crayon_1.5.2                 
#> [137] rlang_1.1.1                   formatR_1.14                 
#> [139] KEGGREST_1.40.0