---
title: "A User Guide to the POUMM R-package"
author: "Venelin Mitov"
output: 
  rmarkdown::html_vignette:
    fig_caption: yes
    toc: true
    toc_depth: 2
vignette: >
  %\VignetteIndexEntry{A User Guide to The POUMM R-package}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
bibliography: REFERENCES.bib
---

<!--
# Copyright 2015-2019 Venelin Mitov
#
# This file is part of POUMM.
#
# POUMM is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# POUMM is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with POUMM  If not, see <http://www.gnu.org/licenses/>.
-->

```{r setup, include = FALSE}
# Make results reproducible
set.seed(1)
knitr::opts_chunk$set(cache = FALSE)
options(digits = 4)

library(POUMM)

useCachedResults <- TRUE
```

Here, we introduce the R-package **POUMM** - an implementation of the Phylogenetic Ornstein-Uhlenbeck Mixed Model (POUMM) for univariate continuous traits [@Mitov:2018co;@Mitov:2017eg]. In the sections below, we demonstrate how the package works. To that end, we run a simulation of a trait according to the POUMM model. Then, we execute a maximum likelihood (ML) and a Bayesian (MCMC) POUMM fit to the simulated data. We show how to use plots and some diagnostics to assess the quality of the fit, i.e. the mixing and the convergence of the MCMC, as well as the consistency of the POUMM fit with the true POUMM parameters from the simulation. Once you are familiar with these topics, we recommend reading the vignette [Interpreting the POUMM](./InterpretingPOUMM.html).

But before we start, we need to install and load a few R-packages from CRAN:

```{r install-CRAN-packages, eval=FALSE}
install.packages("data.table")
install.packages("ggplot2")
install.packages("lmtest")
install.packages("ape")
```
```{r load-CRAN-packages, eval=TRUE, echo=TRUE, message=FALSE, warning=FALSE, results="hide"}
library(ggplot2)
library(data.table)
library(lmtest)
library(ape)
```

# Installing the POUMM R-package {#Installing}
You can install the most recent version of the package from github:
```{r, eval=FALSE, echo=TRUE}
devtools::install_github(repo="venelin/POUMM")
```
The above command will install the HEAD version from the master git-branch. This  is the package branch, which evolves the fastest and gets the quickest bug-fixes.

A more stable version of the package can be installed from CRAN:
```{r, eval=FALSE, echo=TRUE}
install.packages("POUMM")
```

The above commands will install all 3rd party dependencies with one exception: the package for high precision floating point arithmetics (Rmpfr). While the POUMM can be run without Rmpfr installed, installing this package is recommended to prevent numerical instability in some extreme cases, i.e. very small positive values for the POUMM parameters alpha, sigma, and sigmae. Currently, Rmpfr is listed as "Suggests" in POUMM's DESCRIPTION, since it's installation has been problematic on some systems (i.e. Linux). If you have not done this before, you may need to enable Rcpp in your R environment. You can read how to do this in the Rcpp-FAQ vignette available from [this page](https://CRAN.R-project.org/package=Rcpp).

Bear in mind that, for CRAN compliance, some of the functionality of the package
might not be enabled on some systems, in particular, parallel likelihood calculation on 
Mac OS X El Capitan using clang compiler prior to version 6 (see the section [Parallel execution](#Parallel) below).

# Simulating trait evolution under the POUMM 

## Parameters of the simulation
First, we specify the parameters of the POUMM simulation: 
```{r}
N <- 500
g0 <- 0           
alpha <- .5        
theta <- 2        
sigma <- 0.2     
sigmae <- 0.2 
```


```{r, include=FALSE, eval=useCachedResults}
data(vignetteCachedResults)
list2env(vignetteCachedResults, globalenv())
```

We briefly explain the above parameters. The first four of them define an OU-process with initial state $g_0$, a selection strength parameter, $\alpha$, a long-term mean, $\theta$, and a stochastic time-unit standard deviation, $\sigma$. To get an intuition about the OU-parameters, one can consider random OU-trajectories using the function `rTrajectoryOU()`. On the figure below, notice that doubling $\alpha$ speeds up the convergence of the trajectory towards $\theta$ (magenta line) while doubling $\sigma$ results in bigger stochastic oscilations (blue line):

```{r, echo=FALSE, fig.height=5, fig.width=7, fig.cap="Dashed black and magenta lines denote the deterministic trend towards the long-term mean $\\theta$, fixing the stochastic parameter $\\sigma=0$."}
tStep <- 0.025
t <- seq(0, 6, by = tStep)

plot(t, rTrajectoryOU(g0, tStep, alpha, theta, sigma, length(t)), type = 'l', ylab = expression(bar(z)), ylim = c(0, 4))
#plot(t, rTrajectoryOU(g0, tStep, alpha, theta, sigma, length(t)), type = 'l', main = "Random OU trajectories", ylab = "g", ylim = c(0, 4))
lines(t, rTrajectoryOU(g0, tStep, alpha, theta, 0, length(t)), lty = 2)

lines(t, rTrajectoryOU(g0, tStep, alpha*2, theta, sigma, length(t)), col = "magenta")
lines(t, rTrajectoryOU(g0, tStep, alpha*2, theta, 0, length(t)), lty = 2, col = "magenta")

lines(t, rTrajectoryOU(g0, tStep, alpha, theta, sigma*2, length(t)), col = "blue")

abline(h=theta, lty = 3, col = "darkgrey")

legend("topleft", 
       legend = c(expression(list(alpha == .5, sigma^2 == 0.04)),
                  expression(list(alpha == .5, sigma^2 == 0.16)),
                  expression(list(alpha == .5, sigma^2 == 0)),
                  
                  expression(list(alpha == 1, sigma^2 == 0.04)),
                  expression(list(alpha == 1, sigma^2 == 0)),
                  
                  expression(theta == 2)),
       # legend = c(expression(list(alpha == .5, theta == 2, sigma == 0.2)),
       #            expression(list(alpha == .5, theta == 2, sigma == 0.4)),
       #            expression(list(alpha == .5, theta == 2, sigma == 0)),
       #            
       #            expression(list(alpha == 1, theta == 2, sigma == 0.2)),
       #            expression(list(alpha == 1, theta == 2, sigma == 0)),
       #            
       #            expression(theta == 2)),
       lty = c(1, 1, 2, 1, 2, 3), 
       col = c("black", "blue", "black", "magenta", "magenta", "darkgrey"))
```


The POUMM models the evolution of a continuous trait, $z$, along a phylogenetic tree, assuming that $z$ is the sum of a genetic (heritable) component, $g$, and an independent non-heritable (environmental) component, $e\sim N(0,\sigma_e^2)$. At every branching in the tree, the daughter lineages inherit the $g$-value of their parent, adding their own environmental component $e$. The POUMM assumes the genetic component, $g$, evolves along each lineage according to an OU-process with initial state the $g$ value inherited from the parent-lineage and global parameters $\alpha$, $\theta$ and $\sigma$. 

## Simulating the phylogeny
Once the POUMM parameters are specified, we use the **ape** R-package to generate a random tree with `r N` tips:

```{r simulate-tree, results="hide", eval=!useCachedResults}
# Number of tips
tree <- rtree(N)

plot(tree, show.tip.label = FALSE)
```

## Simulating trait evolution on the phylogeny
Starting from the root value $g_0$, we simulate the genotypic values, $g$, and the environmental contributions, $e$, at all internal nodes down to the tips of the phylogeny:


```{r simulate-gez-OU, eval=!useCachedResults}
# genotypic (heritable) values
g <- rVNodesGivenTreePOUMM(tree, g0, alpha, theta, sigma)

# environmental contributions
e <- rnorm(length(g), 0, sigmae)

# phenotypic values
z <- g + e
```

## Visualizing the data
In most real situations, only the phenotypic value, at the tips, i.e. \code{z[1:N]} will be observable. One useful way to visualize the observed trait-values is to cluster the tips in the tree according to their root-tip distance, and to use box-whisker or violin plots to visualize the trait distribution in each group. This allows to visually assess the trend towards uni-modality and normality of the values - an important prerequisite for the POUMM. 

```{r violin-plots, fig.show = "hold", fig.height=4, fig.width=7, fig.cap="Distributions of the trait-values grouped according to their root-tip distances."}
# This is easily done using the nodeTimes utility function in combination with
# the cut-function from the base package.
data <- data.table(z = z[1:N], t = nodeTimes(tree, tipsOnly = TRUE))
data <- data[, group := cut(t, breaks = 5, include.lowest = TRUE)]

ggplot(data = data, aes(x = t, y = z, group = group)) + 
  geom_violin(aes(col = group)) + geom_point(aes(col = group), size=.5)
```


# Fitting the POUMM
Once all simulated data is available, it is time proceed with a first POUMM fit. This is done easily by calling the function `POUMM()`: 

```{r MaintainCache, echo=FALSE, warning=FALSE, results="hide", message=FALSE, eval=TRUE}
if(!useCachedResults) {
  # Perform the heavy fits locally. 
  # set up a parallel cluster on the local computer for parallel MCMC:
  cluster <- parallel::makeCluster(parallel::detectCores(logical = FALSE))
  doParallel::registerDoParallel(cluster)
  
  fitPOUMM <- POUMM(z[1:N], tree, spec=list(thinMCMC = 1000, parallelMCMC=TRUE), verbose = TRUE)
  fitPOUMM2 <- POUMM(z[1:N], tree, spec=list(nSamplesMCMC = 4e5, thinMCMC = 1000, parallelMCMC=TRUE))
  
  specH2tMean <- specifyPOUMM_ATH2tMeanSeG0(z[1:N], tree, 
                                                   nSamplesMCMC = 4e5, 
                                                   thinMCMC = 1000, parallelMCMC=TRUE)
  fitH2tMean <- POUMM(z[1:N], tree, spec = specH2tMean)
  
  
  # Don't forget to destroy the parallel cluster to avoid leaving zombie worker-processes.
  parallel::stopCluster(cluster)

  # remove the pruneInfo because it is not serializable. To be restored after loading the cachedResultsFile.
  fitPOUMM$pruneInfo <- fitPOUMM2$pruneInfo <- fitH2tMean$pruneInfo <- NULL
  
  vignetteCachedResults <- list(
    g = g,
    z = z,
    tree = tree,
    e = e,
    fitPOUMM = fitPOUMM,
    fitPOUMM2 = fitPOUMM2,
    fitH2tMean = fitH2tMean)
  # use list2env(vignetteCachedResults, globalenv())
  # to restore the objects in the global environment
  usethis::use_data(vignetteCachedResults, overwrite = TRUE)
  
  # save(g, z, tree, e, 
  #      fitPOUMM2, fitPOUMM, fitH2tMean,
  #      file = cachedResultsFile)
} 
```

```{r RestorePruneInfo, echo=FALSE, warning=FALSE, results="hide", message=FALSE, eval=TRUE}
# restore the pruneInfo since it is needed afterwards.
fitPOUMM$pruneInfo <- fitPOUMM2$pruneInfo <- fitH2tMean$pruneInfo <- pruneTree(tree, z[1:length(tree$tip.label)])
```


```{r fitPOUMM-1, results="hide", message=FALSE, warning=FALSE, eval=FALSE}
fitPOUMM <- POUMM(z[1:N], tree)
```

The above code runs for about 5 minutes on a MacBook Pro Retina (late 2013) with a 2.3 GHz Intel Core i7 processor. Using default settings, it performs a maximum likelihood (ML) and a Bayesian (MCMC) fit to the data. First the ML-fit is done. Then, three MCMC chains are run as follows: the first MCMC chain samples from the default prior distribution, i.e. assuming a constant POUMM likelihood; the second and the third chains perform adaptive Metropolis sampling from the posterior parameter distribution conditioned on the default prior and the data. By default each chain is run for $10^5$ iterations. This and other default POUMM settings are described in the help-page for the function `specifyPOUMM()`. 

The strategy of executing three MCMC chains instead of one allows to assess: 

* the quality of the MCMC fit: a mismatch between the sampling distributions of the second and third chains suggests that at least one of the chains has not converged to a region of high posterior density (HPD). 
* the presence of signal for the POUMM parameters in the data: a close match between prior and posterior distributions suggests lack of signal in the data. 

We plot traces and posterior sample densities from the MCMC fit:

```{r, fig.height=5.4, fig.show="hold", fig.width=7.2, warning=FALSE, fig.cap="MCMC traces from a POUMM MCMC-fit.", results="hide", eval=TRUE}

# get a list of plots 
plotList <- plot(fitPOUMM, showUnivarDensityOnDiag = TRUE, doPlot = FALSE)
plotList$traceplot
```

```{r, fig.height=5.4, fig.show="hold", fig.width=7.2, warning=FALSE, fig.cap="MCMC univariate density plots. Black dots on the x-axis indicate the ML-fit."}
plotList$densplot
```

A mismatch of the posterior sample density plots from chains 2 and 3, in particular for the phylogenetic heritability, $H_{\bar{t}}^2$, indicates that the chains have not converged. This can be confirmed quantitatively by the Gelman-Rubin statistic (column called G.R.) in the summary of the fit:

```{r, warning=FALSE, eval=TRUE}
summary(fitPOUMM)
```

The G.R. diagnostic is used to check whether two random samples originate from the same distribution. Values that are substantially different from 1.00 (in this case greater than 1.01) indicate significant difference between the two samples and possible need to increase the number of MCMC iterations. Therefore, we rerun the fit specifying that each chain should be run for $4 \times 10^5$ iterations:

```{r fitPOUMM-2, results="hide", eval=FALSE}
fitPOUMM2 <- POUMM(z[1:N], tree, spec=list(nSamplesMCMC = 4e5))  
```

Now, both the density plots and the G.R. values indicate nearly perfect convergence of the second and third chains. The agreement between the ML-estimates (black dots on the density plots) and the posterior density modes (approximate location of the peak in the density curves) shows that the prior does not inflict a bias on the MCMC sample. The mismatch between chain 1 and chains 2 and 3 suggests that the information about the POUMM parameters contained in the data disagrees with or significantly improves our prior knowledge about these parameters. This is the desired outcome of a Bayesian fit, in particular, in the case of a weak (non-informed) prior, such as the default one. 

```{r, fig.height=5.4, fig.show="hold", fig.width=7.2, warning=FALSE, results="hide", eval=TRUE}
plotList <- plot(fitPOUMM2, doPlot = FALSE)
plotList$densplot
```


```{r, warning=FALSE, eval=TRUE}
summary(fitPOUMM2)
```

## Consistency of the fit with the "true" simulation parameters
The 95% high posterior density (HPD) intervals contain the true values for all five POUMM parameters ($\alpha$, $\theta$, $\sigma$, $\sigma_e$ and $g_0$). This is also true for the derived statistics. To check this, we calculate the true derived statistics from the true parameter values and check that these are well within the corresponding HPD intervals:

```{r}
tMean <- mean(nodeTimes(tree, tipsOnly = TRUE))
tMax <- max(nodeTimes(tree, tipsOnly = TRUE))

c(# phylogenetic heritability at mean root-tip distance: 
  H2tMean = H2(alpha, sigma, sigmae, t = tMean),
  # phylogenetic heritability at long term equilibirium:
  H2tInf = H2(alpha, sigma, sigmae, t = Inf),
  # empirical (time-independent) phylogenetic heritability, 
  H2e = H2e(z[1:N], sigmae),
  # genotypic variance at mean root-tip distance: 
  sigmaG2tMean = varOU(t = tMean, alpha, sigma),
  # genotypic variance at max root-tip distance: 
  sigmaG2tMean = varOU(t = tMax, alpha, sigma),
  # genotypic variance at long-term equilibrium:
  sigmaG2tInf = varOU(t = Inf, alpha, sigma)
  )
```

Finally, we compare the ratio of empirical genotypic to total phenotypic variance with the HPD-interval for the phylogenetic heritability. 

```{r, warning=FALSE}
c(H2empirical = var(g[1:N])/var(z[1:N]))
summary(fitPOUMM2)["H2e"==stat, unlist(HPD)]
```

# Parallel execution {#Parallel}
On computers with multiple core processors, it is possible to speed-up the POUMM-fit by parallelization. The POUMM package supports parallelization on two levels:

* parallelizing the POUMM likelihood calculation. The POUMM package parallelizes the likelihood calculation through the SPLITT library for parallel lineage traversal [@Mitov:2017eg]. This is a fine grain parallelization, which can benefit from modern single instruction multiple data (SIMD) processors as well as multiple physical cores. Parallelization on multiple cores becomes beneficial on trees exceeding several hundreds tips. For this parallelization to work, the POUMM package must be compiled from source-code using an OpenMP 4.0-enabled C++ compiler. Open MP 4.0 is supported by several modern C++ compilers including Gnu-g++, Intel-icpc and clang version 6.0 or above (see also the [SPLITT Get started guide](https://venelin.github.io/SPLITT/articles/SPLITT.html#prerequisites) for details). 

To control the maximum number of threads (defaults to all physical or virtual cores on the system) by specifying the environment variable OMP_NUM_THREADS, before starting R e.g.:
```
export OMP_NUM_THREADS=4
```

* parallelizing the MCMC-chains - this can be done by creating a *cluster* using the R-package ```parallel```. With the default settings of the MCMC-fit (executing two MCMC chains sampling from the posterior distribution and one MCMC chain sampling from the prior), this parallelization can result in about two times speed-up of the POUMM fit on a computer with at least two available physical cores. 

```{r, echo=TRUE, eval=FALSE}
# set up a parallel cluster on the local computer for parallel MCMC:
cluster <- parallel::makeCluster(parallel::detectCores(logical = FALSE))
doParallel::registerDoParallel(cluster)

fitPOUMM <- POUMM(z[1:N], tree, spec=list(parallelMCMC = TRUE))

# Don't forget to destroy the parallel cluster to avoid leaving zombie worker-processes.
parallel::stopCluster(cluster)
```

It is possible to use this parallelization in combination with parallelization of the likelihood calculation. This, however, has not been tested and, presumably, would be slower than a parallelization on the likelihood level only. It may be appropriate to parallelize the MCMC chains on small trees, since for small trees, the parallel likelihood calculation is not likely to be much faster than a serial mode calculation. In this case the SPLITT library would switch automatically to serial mode, so there will be no parallel CPU at the likelihood level.

# Packages used
```{r create-references, echo=FALSE, include=FALSE, eval=TRUE}
likCalculation <- c("Rcpp", "Rmpfr")
mcmcSampling <- c("adaptMCMC")
mcmcDiagnosis <- c("coda")
otherPackages <- c("parallel", "foreach", "data.table", "Matrix")
treeProcessing <- c("ape")
reporting <- c("data.table", "ggplot2", "lmtest")
testing <- c("testthat", "mvtnorm")
 
packagesUsed <- c(likCalculation, mcmcDiagnosis, otherPackages, treeProcessing, reporting, testing)

printPackages <- function(packs) {
  res <- ""
  for(i in 1:length(packs)) {
    res <- paste0(res, paste0(packs[i], ' v', packageVersion(packs[i]), ' [@R-', packs[i], ']'))
    if(i < length(packs)) {
      res <- paste0(res, ', ')
    }
  }
  res
}

# Write bib information (this line is executed manually and the bib-file is edited manually after that)
# knitr::write_bib(packagesUsed, file = "./REFERENCES-R.bib")
```


Apart from base R functionality, the POUMM package uses a number of 3rd party R-packages:

* For likelihood calculation: `r printPackages(likCalculation)`;
* For mcmcSampling: `r printPackages(mcmcSampling)`;
* For MCMC convergence diagnostics, calculation of MCMC effective sizes and HPD-intervals: `r printPackages(mcmcDiagnosis)`;
* For other purposes (parameter transformations and summary statistics): `r printPackages(otherPackages)`);
* For tree processing: `r printPackages(treeProcessing)`;
* For reporting: `r printPackages(reporting)`;
* For testing: `r printPackages(testing)`.

# References