Skip to content

Instantly share code, notes, and snippets.

@cnolanminich

cnolanminich/how_we_set_up_our_shiny_server.md

Created September 21, 2018 19:03
Show Gist options
  • Save cnolanminich/52cba3788f0074b682370f40d65ce089 to your computer and use it in GitHub Desktop.
Save cnolanminich/52cba3788f0074b682370f40d65ce089 to your computer and use it in GitHub Desktop.
Download ZIP
How we set up our shiny server
Raw
how_we_set_up_our_shiny_server.md

How We Set Up Our Shiny Servey

Author: Christian Minich

Date: 2017-01-18

Recently we stood up a series of Shiny apps on our linux server, three of which you can find below:

  • SERVER/apps/or_prediction
  • SERVER/apps/rocqi_tutorials
  • SERVER/apps/spc_template

The other one we developed, which is locked down to specific users, is here:

  • SERVER/apps/onco_pro

We are using this for a couple of different use cases that our current set of tools doesn't support very well:

  • To create interactive onboarding or training tutorials
  • To merge the language of predictive model creation with the language to visualize its performance
  • To expose internally-created packages (e.g. rocqi) to non-coding audiences)
  • As a light-weight web application prototyping framekwork.

Some of these could be public-facing, but many need some level of secutiy built in to them.

The skills I needed to obtain in order to do this

  • Sudo access to a linux server (in our case, RHEL 7)
  • A basic understanding of Bash (shell scripting language for linux)
  • A basic familiarity with Linux OS / file systems

A couple of notes:

Here is one example with instructions on how to do it from "scratch."

I used this Ansible playbook, created by Will Struebing. It defaults to setting up an R Studio server on port 8787 and Shiny on 3838.

I had some trouble with a few of the steps (I think it was the Firewall step? Had to manually start/stop / restart the Firewalld service).

The server I used (SERVER) already had something listening on port 3838, so the main page (SERVER:3838) would serve up, but the Shiny apps wouldn't render.

We wanted to set up the Shiny server behind an Nginx reverse proxy. You can follow something like this to see how to do it.

Major roadblocks

  • How can I make the apps only viewable to specific people?
    • R Studio supports this in their paid version, but we have a way around it, in two steps
    • To segment access on a per app basis, we initially tried to use the PAM.d configuration, but as noted, that solution seemed only to work if we the users had both access to the server AND were authenticated via LDAP. We instead used the third-party nginx module nginx-auth-ldap. For that to work with per app access, we set up one "LDAP Server" in the configuration file per AD Group combination. This will become very annoying as we add many applications. It also means we need to add a new location / block per Shiny app, which is an additional step, in order for the reverse proxy to work. Here is what our default.conf file in /etc/nginx/conf.d to look like, minus the password to CHOP's LDAP for the serveice account:
# we may disable this caching, as it could be problematic for some applications
auth_ldap_cache_enabled on;
auth_ldap_cache_expiration_time 10000;
auth_ldap_cache_size 1000;
#error_log /var/log/nginx/error_ldap.log debug;
ldap_server all {
         url "ldap://[LDAPSERVER]?sAMAccountName?sub?";
         binddn "cn=[username],ou=[],ou=[],dc=[]";
         binddn_passwd "[OUR PASSWORD]";
         require valid_user;
         satisfy any;
         connections 1;
         #group_attribute member;
         #group_attribute_is_dn on;
         #require group "cn=[],dc=[],dc=[]";
    }
ldap_server qmr {
         url "ldap://[LDAPSERVER]?sAMAccountName?sub?";
         binddn "cn=[]ou=[],ou=p[],dc=[]],dc=[]";
         binddn_passwd "[OUR PASSWORD]";
         require valid_user;
         satisfy any;
         connections 1;
         group_attribute member;
         group_attribute_is_dn on;
         require group "CN=[],OU=[],OU=Users,OU=[],DC=[],DC=[]]";
         require group CN=[],OU=[],OU=Users,OU=[],DC=[],DC=[]]";
}

server {
    listen 80;
# app live in production -- only requires user to have something within the broader CHOP.edu LDAP server
    location /apps/or_prediction/ {
      proxy_pass http://127.0.0.1:3839/apps/or_prediction/;
      proxy_redirect off;
      #proxy_redirect http://127.0.0.1:3839/ $scheme://$host/;
      proxy_http_version 1.1;
      proxy_read_timeout 20d;
      proxy_buffering off;
      auth_ldap "Enter AD credentials e.g. 'minichc'";
      auth_ldap_servers all;
        }

 location /apps/wmd/ {
      proxy_pass http://127.0.0.1:3839/apps/wmd/;
      proxy_redirect off;
      #proxy_redirect http://127.0.0.1:3839/ $scheme://$host/;
      proxy_http_version 1.1;
	       proxy_read_timeout 20d;
      proxy_buffering off;
      auth_ldap "Enter AD credentials e.g. 'minichc'";
      auth_ldap_servers all;
        }

# test app endpoint with 2 groups having access
 location /apps/wmd/ {
      proxy_pass http://127.0.0.1:3839/apps/wmd/;
      proxy_redirect off;
      #proxy_redirect http://127.0.0.1:3839/ $scheme://$host/;
      proxy_http_version 1.1;
      proxy_read_timeout 20d;
      proxy_buffering off;
      auth_ldap "Enter AD credentials e.g. 'minichc'";
      auth_ldap_servers qmr;
        }
		
# test app endpoint with 1 group having access 
 location /apps/wmd_2/ {
      proxy_pass http://127.0.0.1:3839/apps/wmd_2/;
      proxy_redirect off;
      #proxy_redirect http://127.0.0.1:3839/ $scheme://$host/;
      proxy_http_version 1.1;
      proxy_read_timeout 20d;
      proxy_buffering off;
      auth_ldap "Enter AD credentials e.g. 'minichc'";
      auth_ldap_servers ear;
 #     auth_ldap "Enter AD credentials e.g. 'minichc'";
 #     auth_ldap_servers test;

        }
}

  • How do I add packages so that shiny can use them?

    • You'll need Sudo access (so talk to Christian at the moment), but you have to install it globally for all users (otherwise it gets stored in you /home/R/x86_64-redhat-linux-gnu-library directory, which is only readable by you. Any apps Shiny is running won't be able to see those packages).
    • Here is the code: sudo su - -c "R -e \"install.packages('flexdashboard', repos='http://cran.rstudio.com/')\"". If you have a list of packages: sudo su - -c "R -e \"install.packages(c('flexdashboard', 'shiny'), repos='http://cran.rstudio.com/')\""
    • You can always install local packages for testing, or locally run R code within the server by either going to the R Rstudio server and logging in (not everyone currently has access to log in) at [SERVER], or by ssh-ing into the server and typing R

  • How do I add an app?

    • Right now, it's not super easy. The current workflow is
      • scp the files or folder into [SERVER] like so: cd to your directory with the file on your local machine, type scp or_prediction.Rmd ssh [USER]@SERVER:/home/minichc (replace with your log in and username after home). More here
      • ssh into our server (like: ssh [USERNAME]@[USERNAME])
      • Use the mkdir command to create a directory to house your R files, with the name you want the app to be called.
      • Copy the .Rmd into the locaiton where Shiny knows to look for the apps like so: cp or_prediction.Rmd /srv/shiny-server/apps/or_prediction
      • Go to the Nginx config file and add a proxy re-direct that looks like this (the parts to edit are the /location directory, and the proxy_pass line)
      location /apps/or_prediction/ {
  proxy_pass http://127.0.0.1:3839/apps/or_prediction/;
  proxy_redirect off;
      #proxy_redirect http://127.0.0.1:3839/ $scheme://$host/; proxy_http_version 1.1; proxy_read_timeout 20d; proxy_buffering off; } 
      * Restart the Nginx service to reflect your changes with `sudo service nginx restart`
* Go to `[SERVER]/apps/[YOUR APP NAME]` to see if it loads!

  • How do I hit Netezza?

    • The drivers are installed. Currently this is done in a temporary hack-y way (using the LD_LIBRARY_PATH to load the shared objects located in /usr/local/nz). This "overrides" some other shared objects specifically, the ones that use the curl library to interact with the internet). We will fix that soon.

  • How do I reference files on GitHub?

    • Currently, due to the shared object SNAFU referenced above, you cant'.

  • How do I update a Shiny app?

    • Follow the same process as adding an app, except that you don't need to add the directory to /srv/shiny-server/apps

  • How can I test to make sure the app will run / debug?

    • One good method (for RMDs): ssh in to your server, enter R, and the run `rmarkdown::render('[YOUR FILE HERE.Rmd]').
    • You can also go to /var/log/shiny-server/, run ls -a and run sudo cat FILENAME. The log file names look like this: log

  • Where are all the important files that I should know about?

    • Shiny server admin reference guide
    • /etc/shiny-server/shiny-server.conf Has the configuration file that tells Shiny how to be configured. To see changes reflected on the server, run sudo service shiny-server restart
    • /etc/nginx/conf.d/default.conf Has the configuration file for Nginx. You'll make changes here when

  • What temporary hacks has Christian done that we'll need to fix?

    • in the /etc/shiny-server/shiny-server.conf, I edited the run_as user to be me (minichc), which means that Shiny apps are running as my username. In the future, we would want it to run through the service account shiny
    • Likewise, we should be using a service account to query the CDW. Currently we have one I can use, but I have to set it up and figure out how to manage password storage.
    • Netezza credentials are currently stored in /home/minichc/.bash_profile as the following variables: NZ_PASS, NZ_USER, NZ_HOST, NZ_DBMS_PORT, NZ_DATABASE
    • I put the following in my .bash_profile variable LD_LIBRARY_PATH: LD_LIBRARY_PATH=/usr/local/lib:/lib64:/usr/local/nz/lib:/usr/local/nz/lib64. What this does is dynamically override the shared objects that R looks for at runtime. This is so that R can use the Netezza drivers. But this is a hack! This creates a conflcit between shared objects, which results in anything in R that references the curl library failing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment