Contents

1 Overview of the analysis

This package provides an integrated analysis workflow for robust and reproducible analysis of mass spectrometry proteomics data for differential protein expression or differential enrichment. It requires tabular input (e.g. txt files) as generated by quantitative analysis softwares of raw mass spectrometry data, such as MaxQuant or IsobarQuant. Functions are provided for data preparation, filtering, variance normalization and imputation of missing values, as well as statistical testing of differentially enriched / expressed proteins. It also includes tools to check intermediate steps in the workflow, such as normalization and missing values imputation. Finally, visualization tools are provided to explore the results, including heatmap, volcano plot and barplot representations. For scientists with limited experience in R, the package also contains wrapper functions that entail the complete analysis workflow and generate a report (see the chapter on Workflow functions). Even easier to use are the interactive Shiny apps that are provided by the package (see the chapter on Shiny apps).

2 Installation

The most recent version of R can be downloaded from CRAN. Additional system requirements are Pandoc, as well as Rtools for Windows and NetCDF for Linux.

Start R and install the DEP package:

if (!requireNamespace("BiocManager", quietly=TRUE))
    install.packages("BiocManager")
BiocManager::install("DEP")

library("DEP")

3 Interactive analysis using the DEP Shiny apps

The package contains shiny apps, which allow for the interactive analysis entailing the entire workflow described below. These apps are especially relevant for users with limited or no experience in R. Currently, there are two different apps. One for label-free quantification (LFQ)-based analysis (output from MaxQuant) and the second for tandem-mass-tags (TMT)-based analysis (output from IsobarQuant). To run the apps, simply run this single command:

# For LFQ analysis
run_app("LFQ")

# For TMT analysis
run_app("TMT")

4 Differential analysis

4.1 Example dataset: Ubiquitin interactors

We analyze a proteomics dataset in which Ubiquitin-protein interactors were characterized (Zhang et al. Mol Cell 2017). The raw mass spectrometry data were first analyzed using MaxQuant (Cox and Mann, Nat Biotech 2007) and the resulting “proteinGroups.txt” file is used as input for the downstream analysis.

4.2 Loading of the data

# Loading a package required for data handling
library("dplyr")

# The data is provided with the package
data <- UbiLength

# We filter for contaminant proteins and decoy database hits, which are indicated by "+" in the columns "Potential.contaminants" and "Reverse", respectively. 
data <- filter(data, Reverse != "+", Potential.contaminant != "+")

The data.frame has the following dimensions:

dim(data)
## [1] 2941   23

The data.frame has the following column names:

colnames(data)
##  [1] "Protein.IDs"             "Majority.protein.IDs"   
##  [3] "Protein.names"           "Gene.names"             
##  [5] "Fasta.headers"           "Peptides"               
##  [7] "Razor...unique.peptides" "Unique.peptides"        
##  [9] "LFQ.intensity.Ubi4_1"    "LFQ.intensity.Ubi4_2"   
## [11] "LFQ.intensity.Ubi4_3"    "LFQ.intensity.Ubi6_1"   
## [13] "LFQ.intensity.Ubi6_2"    "LFQ.intensity.Ubi6_3"   
## [15] "LFQ.intensity.Ctrl_1"    "LFQ.intensity.Ctrl_2"   
## [17] "LFQ.intensity.Ctrl_3"    "LFQ.intensity.Ubi1_1"   
## [19] "LFQ.intensity.Ubi1_2"    "LFQ.intensity.Ubi1_3"   
## [21] "Only.identified.by.site" "Reverse"                
## [23] "Potential.contaminant"

The “LFQ.intensity” columns will be used for subsequent analysis.

4.3 Data preparation

The dataset has unique Uniprot identifiers, however those are not immediately informative. The associated gene names are informative, however these are not always unique.

# Are there any duplicated gene names?
data$Gene.names %>% duplicated() %>% any()
## [1] TRUE
# Make a table of duplicated gene names
data %>% group_by(Gene.names) %>% summarize(frequency = n()) %>% 
  arrange(desc(frequency)) %>% filter(frequency > 1)
## # A tibble: 51 x 2
##    Gene.names frequency
##    <chr>          <int>
##  1 ""                 7
##  2 ATXN2              4
##  3 ATXN2L             4
##  4 SF1                4
##  5 HSPA8              3
##  6 RBM33              3
##  7 UGP2               3
##  8 ACTL6A             2
##  9 BCLAF1             2
## 10 BRAP               2
## # … with 41 more rows

For further analysis these proteins must get unique names. Additionally, some proteins do not have an annotated gene name and for those we will use the Uniprot ID.

# Make unique names using the annotation in the "Gene.names" column as primary names and the annotation in "Protein.IDs" as name for those that do not have an gene name.
data_unique <- make_unique(data, "Gene.names", "Protein.IDs", delim = ";")

# Are there any duplicated names?
data$name %>% duplicated() %>% any()
## [1] FALSE

4.4 Generate a SummarizedExperiment object

Many Bioconductor packages use SummarizedExperiment objects as input and/or output. This class of objects contains and coordinates the actual (assay) data, information on the samples as well as feature annotation. We can generate the SummarizedExperiment object from our data using two different approaches. We can extract sample information directly from the column names or we add sample information using an experimental design template. An experimental design is included in the package for our example dataset.

The experimental design must contain ‘label’, ‘condition’ and ‘replicate’ columns. The ‘label’ column contains the identifiers of the different samples and they should correspond to the column names containing the assay data. The ‘condition’ and ‘replicate’ columns contain the annotation of these samples as defined by the user.

label condition replicate
Ubi4_1 Ubi4 1
Ubi4_2 Ubi4 2
Ubi4_3 Ubi4 3
Ubi6_1 Ubi6 1
Ubi6_2 Ubi6 2
Ubi6_3 Ubi6 3
Ctrl_1 Ctrl 1
Ctrl_2 Ctrl 2
Ctrl_3 Ctrl 3
Ubi1_1 Ubi1 1
Ubi1_2 Ubi1 2
Ubi1_3 Ubi1 3 
# Generate a SummarizedExperiment object using an experimental design
LFQ_columns <- grep("LFQ.", colnames(data_unique)) # get LFQ column numbers
experimental_design <- UbiLength_ExpDesign
data_se <- make_se(data_unique, LFQ_columns, experimental_design)

# Generate a SummarizedExperiment object by parsing condition information from the column names
LFQ_columns <- grep("LFQ.", colnames(data_unique)) # get LFQ column numbers
data_se_parsed <- make_se_parse(data_unique, LFQ_columns)

# Let's have a look at the SummarizedExperiment object
data_se
## class: SummarizedExperiment 
## dim: 2941 12 
## metadata(0):
## assays(1): ''
## rownames(2941): RBM47 UBA6 ... ATXN2.3 X6RHB9
## rowData names(13): Protein.IDs Majority.protein.IDs ... name ID
## colnames(12): Ubi4_1 Ubi4_2 ... Ubi1_2 Ubi1_3
## colData names(4): label ID condition replicate

4.5 Prerequisites of the SummarizedExperiment object

The make_se and make_se_parse functions generate a SummarizedExperiment object that has a couple of specifications. The assay data is log2-transformed and its rownames depict the protein names. The rowData contains, amongst others, the ‘name’ and ‘ID’ columns that were generated by make_unique. The colData contains the experimental design and thereby the sample annotation. Thereby the colData includes the ‘label’, ‘condition’ and ‘replicate’ columns as well as a newly generated ‘ID’ column. The log2-transformed assay data and the specified rowData and colData columns are prerequisites for the subsequent analysis steps.

4.6 Filter on missing values

The dataset contains proteins which are not quantified in all replicates. Some proteins are even only quantified in a single replicate.

# Plot a barplot of the protein identification overlap between samples
plot_frequency(data_se)

This leaves our dataset with missing values, which need to be imputed. However, this should not be done for proteins that contain too many missing values. Therefore, we first filter out proteins that contain too many missing values. This is done by setting the threshold for the allowed number of missing values per condition in the filter_missval function.

# Filter for proteins that are identified in all replicates of at least one condition
data_filt <- filter_missval(data_se, thr = 0)

# Less stringent filtering:
# Filter for proteins that are identified in 2 out of 3 replicates of at least one condition
data_filt2 <- filter_missval(data_se, thr = 1)

After filtering, the number of identified proteins per sample can be plotted as well as the overlap in identifications between samples.

# Plot a barplot of the number of identified proteins per samples
plot_numbers(data_filt)

# Plot a barplot of the protein identification overlap between samples
plot_coverage(data_filt)

4.7 Normalization

The data is background corrected and normalized by variance stabilizing transformation (vsn).

# Normalize the data
data_norm <- normalize_vsn(data_filt)

The normalization can be inspected by checking the distributions of the samples before and after normalization.

# Visualize normalization by boxplots for all samples before and after normalization
plot_normalization(data_filt, data_norm)

4.8 Impute data for missing values

The remaining missing values in the dataset need to be imputed. The data can be missing at random (MAR), for example if proteins are quantified in some replicates but not in others. Data can also be missing not at random (MNAR), for example if proteins are not quantified in specific conditions (e.g. in the control samples). MNAR can indicate that proteins are below the detection limit in specific samples, which could be very well the case in proteomics experiments. For these different conditions, different imputation methods have to be used, as described in the MSnbase vignette and more specifically in the impute function descriptions.

To explore the pattern of missing values in the data, a heatmap is plotted indicating whether values are missing (0) or not (1). Only proteins with at least one missing value are visualized.

# Plot a heatmap of proteins with missing values
plot_missval(data_filt)

This heatmap indicates that missing values are highly biased to specific samples. The example dataset is an affinity enrichment dataset of ubiquitin interactors, which is likely to have proteins which are below the detection limit in specific samples. These can be proteins that are specifically enriched in the ubiquitin purifications, but are not enriched in the controls samples, or vice versa. To check whether missing values are biased to lower intense proteins, the densities and cumulative fractions are plotted for proteins with and without missing values.

# Plot intensity distributions and cumulative fraction of proteins with and without missing values
plot_detect(data_filt)

Indeed the proteins with missing values have on average low intensities. This data (MNAR and close to the detection limit) should be imputed by a left-censored imputation method, such as the quantile regression-based left-censored function (“QRILC”) or random draws from a left-shifted distribution (“MinProb” and “man”). In contrast, MAR data should be imputed with methods such as k-nearest neighbor (“knn”) or maximum likelihood (“MLE”) functions. See the MSnbase vignette and more specifically the impute function description for more information.

# All possible imputation methods are printed in an error, if an invalid function name is given.
impute(data_norm, fun = "")
## Error in match.arg(fun): 'arg' should be one of "bpca", "knn", "QRILC", "MLE", "MinDet", "MinProb", "man", "min", "zero", "mixed", "nbavg"
# Impute missing data using random draws from a Gaussian distribution centered around a minimal value (for MNAR)
data_imp <- impute(data_norm, fun = "MinProb", q = 0.01)

# Impute missing data using random draws from a manually defined left-shifted Gaussian distribution (for MNAR)
data_imp_man <- impute(data_norm, fun = "man", shift = 1.8, scale = 0.3)

# Impute missing data using the k-nearest neighbour approach (for MAR)
data_imp_knn <- impute(data_norm, fun = "knn", rowmax = 0.9)

The effect of the imputation on the distributions can be visualized.

# Plot intensity distributions before and after imputation
plot_imputation(data_norm, data_imp)

4.9 Differential enrichment analysis

Protein-wise linear models combined with empirical Bayes statistics are used for the differential enrichment analysis (or differential expression analysis). The test_diff function introduced here uses limma and automatically generates the contrasts to be tested. For the contrasts generation, the control sample has to be specified. Additionally, the types of contrasts to be produced need to be indicated, allowing the generation of all possible comparisons (“all”) or the generation of contrasts of every sample versus control (“control”). Alternatively, the user can manually specify the contrasts to be tested (type = “manual”), which need to be specified in the argument test.

# Differential enrichment analysis  based on linear models and empherical Bayes statistics

# Test every sample versus control
data_diff <- test_diff(data_imp, type = "control", control = "Ctrl")
## Tested contrasts: Ubi4_vs_Ctrl, Ubi6_vs_Ctrl, Ubi1_vs_Ctrl
# Test all possible comparisons of samples
data_diff_all_contrasts <- test_diff(data_imp, type = "all")
## Tested contrasts: Ubi4_vs_Ubi6, Ubi4_vs_Ctrl, Ubi4_vs_Ubi1, Ubi6_vs_Ctrl, Ubi6_vs_Ubi1, Ctrl_vs_Ubi1
# Test manually defined comparisons
data_diff_manual <- test_diff(data_imp, type = "manual", 
                              test = c("Ubi4_vs_Ctrl", "Ubi6_vs_Ctrl"))
## Tested contrasts: Ubi4_vs_Ctrl, Ubi6_vs_Ctrl

Finally, significant proteins are defined by user-defined cutoffs using add_rejections.

# Denote significant proteins based on user defined cutoffs
dep <- add_rejections(data_diff, alpha = 0.05, lfc = log2(1.5))

4.10 Visualization of the results

The results from the previous analysis can be easily visualized by a number of functions. These visualizations assist in the determination of the optimal cutoffs to be used, highlight the most interesting samples and contrasts, and pinpoint differentially enriched/expressed proteins.

4.10.1 PCA plot

The PCA plot can be used to get a high-level overview of the data. This can be very useful to observe batch effects, such as clear differences between replicates.

# Plot the first and second principal components
plot_pca(dep, x = 1, y = 2, n = 500, point_size = 4)

4.10.2 Correlation matrix

A correlation matrix can be plotted as a heatmap, to visualize the Pearson correlations between the different samples.

# Plot the Pearson correlation matrix
plot_cor(dep, significant = TRUE, lower = 0, upper = 1, pal = "Reds")

4.10.3 Heatmap of all significant proteins

The heatmap representation gives an overview of all significant proteins (rows) in all samples (columns). This allows to see general trends, for example if one sample or replicate is really different compared to the others. Additionally, the clustering of samples (columns) can indicate closer related samples and clustering of proteins (rows) indicates similarly behaving proteins. The proteins can be clustered by k-means clustering (kmeans argument) and the number of clusters can be defined by argument k.

# Plot a heatmap of all significant proteins with the data centered per protein
plot_heatmap(dep, type = "centered", kmeans = TRUE, 
             k = 6, col_limit = 4, show_row_names = FALSE,
             indicate = c("condition", "replicate"))

The heatmap shows a clustering of replicates and indicates that 4Ubi and 6Ubi enrich a similar repertoire of proteins. The k-means clustering of proteins (general clusters of rows) nicely separates protein classes with different binding behaviors.

Alternatively, a heatmap can be plotted using the contrasts, i.e. the direct sample comparisons, as columns. Here, this emphasises the enrichment of ubiquitin interactors compared to the control sample.

# Plot a heatmap of all significant proteins (rows) and the tested contrasts (columns)
plot_heatmap(dep, type = "contrast", kmeans = TRUE, 
             k = 6, col_limit = 10, show_row_names = FALSE)

4.10.4 Volcano plots of specific contrasts

Volcano plots can be used to visualize a specific contrast (comparison between two samples). This allows to inspect the enrichment of proteins between the two samples (x axis) and their corresponding adjusted p value (y axis). The add_names argument can be set to FALSE if the protein labels should be omitted, for example if there are too many names.

# Plot a volcano plot for the contrast "Ubi6 vs Ctrl""
plot_volcano(dep, contrast = "Ubi6_vs_Ctrl", label_size = 2, add_names = TRUE)

4.10.5 Barplots of a protein of interest

It can also be useful to plot the data of a single protein, for example if this protein is of special interest.

# Plot a barplot for USP15 and IKBKG
plot_single(dep, proteins = c("USP15", "IKBKG"))

# Plot a barplot for the protein USP15 with the data centered
plot_single(dep, proteins = "USP15", type = "centered")

4.10.6 Frequency plot of significant proteins and overlap of conditions

Proteins can be differentially enriched/expressed in multiple comparisons. To visualize the distribution of significant conditions per protein and the overlap between conditions, the plot_cond function can be used.

# Plot a frequency plot of significant proteins for the different conditions
plot_cond(dep)

4.11 Results table

To extract a table containing the essential results, the get_results function can be used.

# Generate a results table
data_results <- get_results(dep)

# Number of significant proteins
data_results %>% filter(significant) %>% nrow()
## [1] 272

The resulting table contains the following columns:

# Column names of the results table
colnames(data_results)
##  [1] "name"                     "ID"                      
##  [3] "Ubi1_vs_Ctrl_p.val"       "Ubi4_vs_Ctrl_p.val"      
##  [5] "Ubi6_vs_Ctrl_p.val"       "Ubi1_vs_Ctrl_p.adj"      
##  [7] "Ubi4_vs_Ctrl_p.adj"       "Ubi6_vs_Ctrl_p.adj"      
##  [9] "Ubi1_vs_Ctrl_significant" "Ubi4_vs_Ctrl_significant"
## [11] "Ubi6_vs_Ctrl_significant" "significant"             
## [13] "Ubi1_vs_Ctrl_ratio"       "Ubi4_vs_Ctrl_ratio"      
## [15] "Ubi6_vs_Ctrl_ratio"       "Ctrl_centered"           
## [17] "Ubi1_centered"            "Ubi4_centered"           
## [19] "Ubi6_centered"

Of these columns, the p.val and p.adj columns contain the raw and adjusted p values, respectively, for the contrast as depicted in the column name. The ratio columns contain the average log2 fold changes. The significant columns indicate whether the protein is differentially enriched/expressed, as defined by the chosen cutoffs. The centered columns contain the average log2 fold changes scaled by protein-wise centering.

4.12 Generate a data.frame from the resulting SummarizedExperiment object

You might want to obtain an ordinary data.frame of the results. For this purpose, the package provides functions to convert SummarizedExperiment objects to data.frames. get_df_wide will generate a wide table, whereas get_df_long will generate a long table.

# Generate a wide data.frame
df_wide <- get_df_wide(dep)
# Generate a long data.frame
df_long <- get_df_long(dep)

4.13 Save your results object for reuse

To facilitate future analysis and/or visualization of your current data, saving your analyzed data is highly recommended. We save the final data object (dep) as well as intermediates of the analysis, i.e. the initial SummarizedExperiment object (data_se), normalized data (data_norm), imputed data (data_imp) and differentially expression analyzed data (data_diff). This allows us to easily change parameters in future analysis.

# Save analyzed data
save(data_se, data_norm, data_imp, data_diff, dep, file = "data.RData")
# These data can be loaded in future R sessions using this command
load("data.RData")

5 Workflow functions for the entire analysis

The package contains workflow functions that entail the complete analysis and generate a report.

5.1 LFQ-based DEP analysis

Differential enrichment analysis of label-free proteomics data can be performed using the LFQ workflow function.

# The data is provided with the package 
data <- UbiLength
experimental_design <- UbiLength_ExpDesign

# The wrapper function performs the full analysis
data_results <- LFQ(data, experimental_design, fun = "MinProb", 
      type = "control", control = "Ctrl", alpha = 0.05, lfc = 1)
## [1] 0.2608395

This wrapper produces a list of objects, which can be used to create a report and/or for further analysis. The report function produces two reports (pdf and html), a results table and a RData object, which are saved in a generated “Report” folder.

# Make a markdown report and save the results
report(data_results)

The results generated by the LFQ function contain, among others, the results object (data.frame object) and the dep object (SummarizedExperiment object).

# See all objects saved within the results object
names(data_results)
## [1] "data"    "se"      "filt"    "norm"    "imputed" "diff"    "dep"    
## [8] "results" "param"

The results object contains the essential results and the dep object contains the full SummarizedExperiment object. The results table can be explored by selecting the $results object

# Extract the results table
results_table <- data_results$results

# Number of significant proteins
results_table %>% filter(significant) %>% nrow()
## [1] 232

The full data (dep object) can be used for the plotting functions as described in the chapter “Visualization of the results”, for example a heatmap.

# Extract the sign object
full_data <- data_results$dep

# Use the full data to generate a heatmap
plot_heatmap(full_data, type = "contrast", kmeans = TRUE, 
             k = 6, col_limit = 4, show_row_names = FALSE)

5.2 TMT-based DEP analysis

Differential enrichment analysis of Tandem-Mass-Tag labeled proteomics data is also supported. Protein results tables from IsobarQuant can be directly analyzed using the TMT wrapper function.

# Need example data
TMTdata <- example_data
Exp_Design <- example_Exp_Design

# The wrapper function performs the full analysis
TMTdata_results <- TMT(TMTdata, expdesign = Exp_Design, fun = "MinProb",
    type = "control", control = "Control", alpha = 0.05, lfc = 1)

6 Session information

## R version 3.6.1 (2019-07-05)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.3 LTS
## 
## Matrix products: default
## BLAS:   /home/biocbuild/bbs-3.10-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.10-bioc/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] parallel  stats4    stats     graphics  grDevices utils     datasets 
## [8] methods   base     
## 
## other attached packages:
##  [1] imputeLCMD_2.0      impute_1.60.0       pcaMethods_1.78.0  
##  [4] Biobase_2.46.0      BiocGenerics_0.32.0 norm_1.0-9.5       
##  [7] tmvtnorm_1.4-10     gmm_1.6-2           sandwich_2.5-1     
## [10] Matrix_1.2-17       mvtnorm_1.0-11      dplyr_0.8.3        
## [13] DEP_1.8.0           BiocStyle_2.14.0   
## 
## loaded via a namespace (and not attached):
##   [1] colorspace_1.4-1            rjson_0.2.20               
##   [3] ellipsis_0.3.0              circlize_0.4.8             
##   [5] XVector_0.26.0              GenomicRanges_1.38.0       
##   [7] GlobalOptions_0.1.1         clue_0.3-57                
##   [9] mzR_2.20.0                  affyio_1.56.0              
##  [11] ggrepel_0.8.1               DT_0.9                     
##  [13] fansi_0.4.0                 codetools_0.2-16           
##  [15] ncdf4_1.17                  doParallel_1.0.15          
##  [17] knitr_1.25                  zeallot_0.1.0              
##  [19] cluster_2.1.0               vsn_3.54.0                 
##  [21] png_0.1-7                   shinydashboard_0.7.1       
##  [23] shiny_1.4.0                 BiocManager_1.30.9         
##  [25] readr_1.3.1                 compiler_3.6.1             
##  [27] backports_1.1.5             assertthat_0.2.1           
##  [29] fastmap_1.0.1               lazyeval_0.2.2             
##  [31] limma_3.42.0                cli_1.1.0                  
##  [33] later_1.0.0                 htmltools_0.4.0            
##  [35] tools_3.6.1                 gtable_0.3.0               
##  [37] glue_1.3.1                  GenomeInfoDbData_1.2.2     
##  [39] affy_1.64.0                 Rcpp_1.0.2                 
##  [41] MALDIquant_1.19.3           vctrs_0.2.0                
##  [43] preprocessCore_1.48.0       iterators_1.0.12           
##  [45] xfun_0.10                   stringr_1.4.0              
##  [47] mime_0.7                    lifecycle_0.1.0            
##  [49] XML_3.98-1.20               zlibbioc_1.32.0            
##  [51] MASS_7.3-51.4               zoo_1.8-6                  
##  [53] scales_1.0.0                MSnbase_2.12.0             
##  [55] hms_0.5.1                   promises_1.1.0             
##  [57] ProtGenerics_1.18.0         SummarizedExperiment_1.16.0
##  [59] RColorBrewer_1.1-2          ComplexHeatmap_2.2.0       
##  [61] yaml_2.2.0                  gridExtra_2.3              
##  [63] ggplot2_3.2.1               stringi_1.4.3              
##  [65] highr_0.8                   S4Vectors_0.24.0           
##  [67] foreach_1.4.7               BiocParallel_1.20.0        
##  [69] shape_1.4.4                 GenomeInfoDb_1.22.0        
##  [71] rlang_0.4.1                 pkgconfig_2.0.3            
##  [73] matrixStats_0.55.0          bitops_1.0-6               
##  [75] mzID_1.24.0                 evaluate_0.14              
##  [77] lattice_0.20-38             purrr_0.3.3                
##  [79] htmlwidgets_1.5.1           labeling_0.3               
##  [81] tidyselect_0.2.5            plyr_1.8.4                 
##  [83] magrittr_1.5                bookdown_0.14              
##  [85] R6_2.4.0                    IRanges_2.20.0             
##  [87] DelayedArray_0.12.0         pillar_1.4.2               
##  [89] RCurl_1.95-4.12             tibble_2.1.3               
##  [91] crayon_1.3.4                fdrtool_1.2.15             
##  [93] utf8_1.1.4                  rmarkdown_1.16             
##  [95] GetoptLong_0.1.7            grid_3.6.1                 
##  [97] digest_0.6.22               xtable_1.8-4               
##  [99] tidyr_1.0.0                 httpuv_1.5.2               
## [101] munsell_0.5.0