Integration testing vignette for the Shiny Dev center.

integration-testing.Rmd

---
title: "Integration Testing in Shiny"
output: github_document
editor_options:
  chunk_output_type: console
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

<div style="border: 1px solid #EBB; border-radius: 3px; background-color: #FCFCFC; padding: 1em;" markdown="1">

<div style="color: #A00;" markdown="1">
> This section describes features that are still in development and not yet available in the version of Shiny that is on CRAN. If you want to try these features, you'll need to install Shiny from GitHub using: `remotes::install_github("rstudio/shiny")`
</div>

There are many different types of automated testing, but a comprehensive testing strategy for a complex Shiny app might cover:

1. **unit testing** to test the individual functions in your app ([testthat](https://testthat.r-lib.org/) or a similar package)
2. **integration testing** to test the interactions between different functions in your app
3. **functional testing** to test the overall function of the app from the user's perspective in a browser ([shinytest](https://rstudio.github.io/shinytest/articles/shinytest.html), [shinyloadtest](https://rstudio.github.io/shinyloadtest/))

The `testModule()` and `testServer()` functions in Shiny address the "integration testing" point above for modules and Shiny server functions, respectively. We'll take a look at both below, but the mechanics of writing the tests are identical regardless of whether you're testing a module or an application.

(If you haven't seen modules before, we recommend that you go take a look at the [article on modules](https://shiny.rstudio.com/articles/modules.html) first, as this article will presume familiarity.)

```{r}
library(shiny)
module <- function(input, output, session) {
  myreactive <- reactive({
    input$x * 2
  })
  output$txt <- renderText({
    paste0("I am ", myreactive())
  })
}
```

This module

  - depends on one input (`x`),
  - creates an intermediate reactive (`myreactive`), and
  - updates an output (`txt`) reactively.

It would be nice to write tests that confirm that the module behaves the way we expect. We can do so using the `testModule` function.

```{r}
testModule(module, {
  cat("Initially, input$x is NULL, right?", is.null(input$x), "\n")

  # Give input$x a value.
  session$setInputs(x = 1)

  cat("Now that x is set to 1, myreactive is: ", myreactive(), "\n")
  cat("\tand output$txt is: ", output$txt, "\n")
  
  # Now update input$x to a new value
  session$setInputs(x = 2)

  cat("After updating x to 2, myreactive is: ", myreactive(), "\n")
  cat("\tand output$txt is: ", output$txt, "\n")
})
```

There are a few things to notice in this example.

First, the test expression provided here assumes the existence of some variables -- specifically, `input`, `output`, and `myreactive`. This is safe because the test code provided to `testModule` is run in a child environment of the module. This means that any parameters passed in to your module (such as `input`, `output`, and `session`) are readily available, as are any intermediate objects or reactives that you define in the module. However, because it's a child environment, your test code is less likely to accidentally modify anything in the module itself.

Second, you'll need to give values to any inputs that you want to be defined; by default, they're all `NULL`. We do that using the `session$setInputs()` method. The real `session` object that you'd see when running a Shiny app does not have this method; however, the `session` object used in `testModule` differs from the real `session` object Shiny uses. This allows us to tailor it to be more suitable for testing purposes by modifying or creating new methods such as `setInputs()`.

Last, you're likely used to assigning to `output`, but here we're reading from `output$txt` in order to check its value. When running inside `testModule`, you can simply reference an output and it will give the value produced by the `render` function.

## Automated Tests

Realistically, we don't want to just print the values for manual inspection; we'll want to leverage them in automated tests. That way, we'll be able to build up a collection of tests that we can run against our module in the future to confirm that it always behaves correctly. You can use whatever testing framework you'd like (or none at all!), but we'll use the `expect_*` functions from the testthat package in this example.

```{r}
# Bring in testthat just for its expectations
library(testthat)
testModule(module, {
  session$setInputs(x = 1)
  expect_equal(myreactive(), 2)
  expect_equal(output$txt, "I am 2")
  session$setInputs(x = 2)
  expect_equal(myreactive(), 4)
  expect_equal(output$txt, "I am 4")
})
```

If there's no error, then we know our tests ran successfully. If there were a bug, we'd see an error printed. For example:

```{r}
tryCatch({
  testModule(module, {
    session$setInputs(x = 1)

    # This expectation will fail
    expect_equal(myreactive(), 99)
  })
}, error = function(e){
  print("There was an error!")
  print(e)
})
```

## Testing Shiny Apps

In addition to testing Shiny modules, you can also test Shiny applications. The `testServer` function will automatically extract the server portion given an application's directory and you can test it just like you do any other module.

```{r defineAppLines, echo=FALSE}
appLines <- c(
  "# app.R -- written into tempdir()", "ui <- fluidPage(",
  " textInput(\"name\", \"Name: \"),", " textOutput(\"greeting\")",
  ")", "", 
  "server <- function(input, output) {", " output$greeting <- renderText({",
  " paste0(\"Hello, \", input$name, \"!\")", " })",
  "}", "",
  "shinyApp(ui = ui, server = server)"
)

# write out to a tempdir
dir <- tempdir()
writeLines(appLines, file.path(dir, "app.R"))
```

```{r, echo=FALSE, results='asis'}
# Render the appLines variable above as if it had been inlined as an R code chunk

code_chunk <- function(output, language=""){
  cat(paste0("```",language,"\n"))
  cat(output)
  cat("\n```\n")
}

#' Include an external R file with syntax highlighting in the doc
code_chunk(paste0(appLines, collapse="\n"), language="r")
```

```{r}
testServer({
  session$setInputs(name = "Shiny User")
  expect_equal(output$greeting, "Hello, Shiny User!")
  
  session$setInputs(name = "New Name")
  expect_equal(output$greeting, "Hello, New Name!")
}, appDir = tempdir())
```

## Promises

`testModule` can handle promises inside of render functions.

```{r}
library(promises)
library(future)
plan(multisession)
module <- function(input, output, session){
  output$async <- renderText({
    # Stash the value since you can't do reactivity inside of a promise
    # See https://rstudio.github.io/promises/articles/shiny.html#shiny-specific-caveats-and-limitations
    t <- input$times
    
    # A promise chain that repeats the letter A and then collapses it into a string
    future({ rep("A", times = t) }) %...>%
      paste(collapse = "")
  })
}
testModule(module, {
  session$setInputs(times = 3)
  expect_equal(output$async, "AAA")

  session$setInputs(times = 5)
  expect_equal(output$async, "AAAAA")
})
```

As you can see, no special precautions were required for a `render` function that uses promises. Behind-the-scenes, the code in `testModule` will stall when trying to read from an `output` that returned a promise. This allows you to interact with asynchronous outputs in your tests as if they were synchronous.

<!--TODO: What about internal reactives that are promise-based? We don't do anything special for them...-->

## Modules with additional inputs

`testModule` can also handle modules that accept additional arguments such as this one.

```{r}
module <- function(input, output, session, arg1, arg2){
  output$txt1 <- renderText({ arg1 })

  output$txt2 <- renderText({ arg2 })
}
```

Additional arguments should be passed after the test expression as named parameters.

```{r}
testModule(module, {
  expect_equal(output$txt1, "val1")
  expect_equal(output$txt2, "val2")
}, arg1 = "val1", arg2 = "val2")
```

## Accessing a module's returned value

Some modules return reactive data as an output. For such modules, it can be helpful to test the returned value as well. The returned value from the module is made available as a property on the mock `session` object as demonstrated in this example.

```{r}
module <- function(input, output, session){
  reactive({
    return(input$a + input$b)
  })
}
testModule(module, {
  session$setInputs(a = 1, b = 2)
  expect_equal(session$returned(), 3)
  # And retains reactivity
  session$setInputs(a = 2)
  expect_equal(session$returned(), 4)
})
```

## Timer and Polling

Testing behavior that relies on timing is notoriously difficult. Modules will behave differently on different machines and under different conditions. In order to make testing with time more deterministic, `testModule` uses simulated time that you control, rather than the actual computer time. Let's look at what happens when you try to use "real" time in your testing.

```{r}
module <- function(input, output, session){
  rv <- reactiveValues(x=0)

  observe({
    # Cause the observer to invalidate every 0.1 seconds
    invalidateLater(100)
    isolate(rv$x <- rv$x + 1)
  })
}
testModule(module, {
  expect_equal(rv$x, 1) # The observer runs once at initialization

  Sys.sleep(1) # Sleep for a second

  expect_equal(rv$x, 1) # The value hasn't changed
})
```

This behavior may be surprising. It seems like `rv$x` should have been incremented 10 times (or perhaps 9, due to computational overhead). But in truth, it hasn't changed at all. This is because `testModule` doesn't consider the actual time on your computer, but instead it uses simulated time.

In order to cause `testModule` to progress through time, instead of `Sys.sleep`, we'll use `session$elapse` -- another method that exists only on our mocked session object. Using the same module object as above:

```{r}
testModule(module, {
  expect_equal(rv$x, 1) # The observer runs once at initialization

  session$elapse(100) # Simulate the passing of 100ms

  expect_equal(rv$x, 2) # The observer was invalidated and the value updated!

  # You can even simulate multiple events in a single elapse
  session$elapse(300)
  expect_equal(rv$x, 5)
})
```

As you can see, using `session$elapse` caused `testModule` to recognize that (simulted) time had passed which triggered the reactivity as we'd expect. This approach allows you to deterministically control time in your tests while avoiding expensive pauses that would slow down your tests. Using this approach, this test can complete in only a fraction of the 100ms that it simulates.

You should note that only certain time-based functions are aware of this mocked time that can be managed via `elapse`. Shiny functions like `reactivePoll`, `invalidateLater`, and `reactiveTimer` will all abide by this simulated time, but time-based functions in other packages (such as `later::later` are not aware of mocked time and will not behave according to these rules.)

## Complex Outputs (plots, htmlwidgets)

> **Work in progress** -- We intend to add more helpers to make it easier to inspect and validate the raw HTML/JSON content. Our integration testing tools do not yet provide functionality for validating such output.

Thus far, we've seen how to validate simple outputs like numeric or text values. Real Shiny modules and applications often use more complex outputs such as plots or htmlwidgets. Validating the correctness of these is not as simple, but is doable.

You can access the data for even complex outputs in `testModule`, but the structure of the output may initially be foreign to you.

```{r}
module <- function(input, output, session){
  output$plot <- renderPlot({
    df <- data.frame(length = iris$Petal.Length, width = iris$Petal.Width)
    plot(df)
  })
}
testModule(module, {
  print(str(output$plot))
})
```

As you can see, there are a lot of internal details that go into a plot. Behind-the-scenes, these are all the details that Shiny will use to correctly display a plot in a user's browser. You don't need to learn about all of these properties -- and they're all subject to change.

In terms of your testing strategy, you shouldn't bother yourself with "is Shiny generating the correct structure so that the plot will render in the browser?" That's a question that the Shiny package itself needs to answer (and one for which we have our own tests). The goal for your tests should be to ask "is the code that I wrote producing the plot I want?" There are two components to that question:

  1. Does the plot generate without producing an error?
  2. Is the plot visually correct?

`testModule` is great for assessing the first component here. By merely referencing `output$plot` in your test, you'll confirm that the plot was generated without an error. The second component is better suited for a [shinytest](https://rstudio.github.io/shinytest/articles/shinytest.html) test which actually loads the Shiny app in a headless browser and confirms that the content visually appears the same as it did previously. Doing this kind of test in `testModule` would be complex and may not be reliable as graphics devices differ slightly from platform to platform; i.e. the exact bits in the `src` field of your plot will not necessarily be reproducible between different versions of R or different operating systems.

For htmlwidgets, you can adopt a similar strategy. The goal is not to confirm that the htmlwidget's render function is behaving properly -- but rather that the data that you intend to render is indeed getting passed through.

We could modify the above example to better represent this approach.

```{r}
module <- function(input, output, session){
  # Move any complex logic into a separate reactive which can be tested comprehensively
  plotData <- reactive({
    data.frame(length = iris$Petal.Length, width = iris$Petal.Width)
  })

  # And leave the `render` function to be as simple as possible to lessen the need for
  # integration tests.
  output$plot <- renderPlot({
    plot(plotData())
  })
}
testModule(module, {
  # Confirm that the data reactive is behaving as expected
  expect_equal(nrow(plotData()), 150)
  expect_equal(ncol(plotData()), 2)
  expect_equal(colnames(plotData()), c("length", "width"))

  # And now the plot function is so simple that there's not much need for
  # automated testing. If we did wish to evaluate the plot visually, we could
  # do so using the shinytest package.
  output$plot # Just confirming that the plot can be accessed without an error
})
```

You could adopt a similar strategy with other plots or htmlwidgets: move the complexity into reactives that can be tested, and leave the `render` functions as simple as possible.

## Flushing Reactives

Reactivity differs from imperative programming in that the processing required to update reactives can be deferred and batched together. While this is a boon for the computational speed of a reactive system, it does create some ambiguity about *when* the reactives should be processed or "flushed".

`testModule` will do its best to automatically "flush" the reactives at the right time. There are two triggers that will cause a reactive flush:

1. Calling `session$setInputs()` - After setting the updated inputs, the reactives will be flushed.
2. Calling `session$elapse()` - After the scheduled callbacks are executed, reactives will be flushed.

However, there may be other times that a Shiny module author might want to trigger a reactive flush. For instance, you might want to flush the reactives after updating an element in a `reactiveValues` in your module like this one.

```{r}
module <- function(input, output, session){
  rv <- reactiveValues(a=1)
  output$txt <- renderText({
    rv$a
  })
}

testModule(module, {
  expect_equal(output$txt, "1")
  
  rv$a <- 2
  
  # testModule has no innate knowledge of our `rv` variable; 
  # therefore, it hasn't been updated
  expect_equal(output$txt, "1")
  
  # We'll need to manually force a flush of the reactives
  session$flushReact()
  
  expect_equal(output$txt, "2")
})
```

As you can see, we can use `session$flushReact()` to trigger a reactive flush at any point we'd like. In this example, `testModule` knows  nothing about our `rv` variable. Therefore if we want to observe reactive changes that occur after manually updating this variable, we'd need to explicitly flush the reactives.

</div>