Skip to content

Instantly share code, notes, and snippets.

@duncan-brown

duncan-brown/SciTokenSetup.md

Last active January 28, 2022 02:59
Show Gist options
  • Save duncan-brown/fb5e83b86814baeda001316a6bdfcc3b to your computer and use it in GitHub Desktop.
Save duncan-brown/fb5e83b86814baeda001316a6bdfcc3b to your computer and use it in GitHub Desktop.
Download ZIP
Description of how SciTokens was set up in Syracuse
Raw
SciTokenSetup.md

Syracuse SciTokens Setup

Install software

Switch to Condor 8.8 in /etc/yum.repos.d/htcondor-stable-rhel7.repo then upgrade Condor with

systemctl stop condor
yum upgrade condor
systemctl start condor

Install OSG 3.4 RPMs for xrootd and scitokens

rpm -e osg-release-3.3-7.osg33.el7.noarch
rpm -ivh https://repo.opensciencegrid.org/osg/3.4/osg-3.4-el7-release-latest.rpm
yum install xrootd
yum install xrootd-scitokens
yum install xrootd-multiuser
yum install stashcp

Set up the XrootD config file

Edit the file /etc/xrootd/xrootd-http.cfg add the following two lines to configure https://github.com/opensciencegrid/xrootd-multiuser

xrootd.fslib libXrdMultiuser.so default
multiuser.umask 0022

Then add the following lines to set up https://github.com/scitokens/xrootd-scitokens

ofs.authorize
ofs.authlib libXrdAccSciTokens.so

# Pass the bearer token to the Xrootd authorization framework.
http.header2cgi Authorization authz

The cmsd@http.service service reads this file and chokes on the xrd.protocol line, so edit the file to protect this with an if statement:

if exec xrootd
xrd.protocol XrdHttp:8000 /usr/lib64/libXrdHttp-4.so
fi

To get cmsd working, add the lines

all.role server
xrd.port 1094

all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd

Make sure that these directories exist and have the right permissions:

mkdir -p /var/run/xrootd/http
chowm -R xrootd:xrootd /var/run/xrootd

To register with the OSG redirector, add the line

all.manager redirector.osgstorage.org+ 1213

For debugging, it is also helpful to add

ofs.trace all
oss.trace all

Set up SciTokens config file for XrootD

Create a file named /etc/xrootd/scitokens.cfg contining the lines

[Global]
audience = sugwg-scitokens.phy.syr.edu

[Issuer Scitokens-Jeff]
issuer = https://surge.ncsa.illinois.edu/scitokens-server
base_path = /
map_subject = True

[Issuer Demo]
issuer = https://demo.scitokens.org
base_path = /
map_subject = True

Create the directory structure for XrootD

This assumes that the files really live in /stash/user.

mkdir -p /stash/user/dbrown
chown dbrown:lsc /stash/user/dbrown
mkdir /mnt/stash/osg
cd /mnt/stash/osg
ln -s /stash sugwg

Start XrootD and CMSD

Start the priveleged XrootD server with

systemctl start xrootd-privileged@http
systemctl start cmsd@http.service

Test XrootD

Do the following as an unpriveleged user:

echo "Hello, World!" > hello.txt

Test a write

Go to https://demo.scitokens.org/ and create a token with the following additional lines in the payload

  "scope": "write:/osg/sugwg/user/dbrown",
  "sub": "dbrown",
  "aud": "sugwg-scitokens.phy.syr.edu"

Now try and write a file to the XrootD server with

curl -v --connect-timeout 30 --speed-limit 1024 -X PUT --fail --upload-file hello.txt -H "Authorization: Bearer TOKEN" http://sugwg-scitokens.phy.syr.edu:8000/osg/sugwg/user/dbrown/hello.txt

where TOKEN is the token created by https://demo.scitokens.org (Note: the token must be RS256 encoded). You should see

* About to connect() to sugwg-scitokens.phy.syr.edu port 8000 (#0)
*   Trying 128.230.146.14...
* Connected to sugwg-scitokens.phy.syr.edu (128.230.146.14) port 8000 (#0)
> PUT /osg/sugwg/user/dbrown/hello.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sugwg-scitokens.phy.syr.edu:8000
> Accept: */*
> Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImtleS1yczI1NiJ9.eyJpc3MiOiJodHRwczovL2RlbW8uc2NpdG9rZW5zLm9yZyIsImV4cCI6MTU0NzY1Mjg1OSwiaWF0IjoxNTQ3NjUyMjU5LCJuYmYiOjE1NDc2NTIyNTksImp0aSI6Ijk5NGM1ZWRmLTA1ZDYtNDFiYi04NGZhLTgwNDRjOWRhN2YwZiIsInNjb3BlIjoid3JpdGU6L29zZy9zdWd3Zy91c2VyL2Ricm93biIsInN1YiI6ImRicm93biIsImF1ZCI6InN1Z3dnLXNjaXRva2Vucy5waHkuc3lyLmVkdSJ9.tzRRtP6IJ1e86gIVif8i-e94scJXSaMxeWiogFAUo99P3btNlHqou7gwp5KvxDK8jGOp_8OquNjiPJwSaDyqYpGMrx40JguRTkk5b9TBRzWpiWQm1mJwQGiyOnVQ15je8FgPdkNjF8Sn8q7l26dEEEyWvWKOrlEKqq7xf6KXG-uIOa9pdcFErzi5UWGATXaZORIZiHtQuIMRXojOWMNwPoMgR3gRCWAqjasgOX7BNpJbZVNMGMQpgArqXNiGoP0OARotN9kTWS9cf_0BKQ7IpPtYmztRhK3X2vV7KtUHoJPEjioO7wdTG9wpverbAk46__u6xAE6OmCVsC_d2BCFqA
> Content-Length: 13
> Expect: 100-continue
> 
< HTTP/1.1 100 Unknown
< Connection: Keep-Alive
< Content-Length: 0
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Connection: Close
< Content-Length: 3
< 
* Closing connection 0
:-)

Test a read

Go to https://demo.scitokens.org/ and create a token with the following additional lines in the payload

  "scope": "read:/osg/sugwg/user/dbrown",
  "sub": "dbrown",
  "aud": "sugwg-scitokens.phy.syr.edu"

Grab the token generated and use it to download a file, e.g.

curl -H "Authorization: Bearer TOKEN" http://sugwg-scitokens.phy.syr.edu:8000/osg/sugwg/user/dbrown/hello.txt

This should return

Hello, world!

Install Condor CredMon

These instructions assume that you want to run the Condor CredMon on port 8080 on the submit host under apache. To work around a Condor bug, this requires the magic knob ZKMHOST to be set in the config file.

Install the Condor CredMon by running the following command as root:

pip install git+https://github.com/htcondor/scitokens-credmon

Register with Jeff's SciToken Server

Go to https://surge.ncsa.illinois.edu/scitokens-server/.well-known Register at https://surge.ncsa.illinois.edu/scitokens-server/register Record the client ID and secret for use in /etc/condor/config.d/10-credmon.conf below.

Set up the CredMon

Then create file /etc/condor/config.d/10-credmon.conf containing the lines below, setting using the information from the previous step to replace ID_FROM_JEFF and SECRET_FROM_JEFF:

# Set up the credmon and credential directory
SEC_CREDENTIAL_MONITOR = /usr/bin/condor_credmon
SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credential
SEC_CREDENTIAL_MONITOR_LOG = /var/log/condor/CredmonLog
TOKENS = true
ZKMHOST = sugwg-scitokens.phy.syr.edu:8080

# Define OAuth token providers
SCITOKENS_CLIENT_ID = ID_FROM_JEFF
SCITOKENS_CLIENT_SECRET = SECRET_FROM_JEFF
SCITOKENS_RETURN_URL_SUFFIX = /return/scitokens
SCITOKENS_AUTHORIZATION_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/authorize
SCITOKENS_TOKEN_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/token
SCITOKENS_USER_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/userinfo

Edit condor_config.local and add CREDD to the DAEMON_LIST:

# Enable the CredD
DAEMON_LIST = $(DAEMON_LIST), CREDD

Create the credentials directory and fix the permissions

mkdir -p /var/lib/condor/credential
chown condor:condor /var/lib/condor/credential
chmod 700 /var/lib/condor/credential

Starting the CredMon

The manager currently isn't supervising the CredMon, so start it manually with the command

su - condor
nohup condor_credmon </dev/null &>/dev/null &
exit

Configure apache for the CredMon web page

Create a file /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi containing the lines

# Set up the credmon and credential directory
SEC_CREDENTIAL_MONITOR = /usr/bin/condor_credmon
SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credential
SEC_CREDENTIAL_MONITOR_LOG = /var/log/condor/CredmonLog
TOKENS = true
ZKMHOST = sugwg-scitokens.phy.syr.edu:8080

# Define OAuth token providers
SCITOKENS_CLIENT_ID = myproxy:oa4mp,2012:/client_id/3dc18747287b0f29805bcee37f2fd3c8
SCITOKENS_CLIENT_SECRET = WEzL7U5wW9lnCzezv9mOV_mdHcwnH74zj-vKSbkOfmplOmTbfTKkHQ
SCITOKENS_RETURN_URL_SUFFIX = /return/scitokens
SCITOKENS_AUTHORIZATION_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/authorize
SCITOKENS_TOKEN_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/token
SCITOKENS_USER_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/userinfo
[dbrown@sugwg-scitokens stash_cp_test]$ cat /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi
#
# Configure Logging
#

from credmon.utils import setup_logging, get_cred_dir
import os
import logging

cred_dir = get_cred_dir()
logger = setup_logging(log_path = os.path.join(cred_dir, 'oauth_credmon_webserver.log'),
                       log_level = logging.DEBUG)


#
# Start Service
#
from credmon.CredentialMonitors.OAuthCredmonWebserver import app
app.secret_key = 'HereComeTheBadgers'

application = app

Create a file /etc/httpd/conf.d/condor-credmon.conf containing

Listen 128.230.146.14:8080

<VirtualHost 128.230.146.14:8080>
  ServerName sugwg-scitokens.phy.syr.edu
  ServerAdmin dabrown@syr.edu

  ## Vhost docroot
  DocumentRoot /var/www/cgi-bin/wsgi/condor-credmon

  <Directory "/var/www/cgi-bin/wsgi/condor-credmon">
    Options -Indexes
    AllowOverride None
    Order allow,deny
    Allow from all
    SSLOptions +StdEnvVars +ExportCertData
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/condor-credmon_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/condor-credmonhttp_access.log" combined

  WSGIDaemonProcess     CondorCredmon user=condor group=condor processes=2 threads=25 python-path=/usr/lib64/pegasus/externals/python
  WSGIProcessGroup      CondorCredmon
  WSGIScriptAlias       / /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi

  SSLEngine on
  SSLProtocol all -SSLv2 -SSLv3
  SSLHonorCipherOrder on
  SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"

  SSLCertificateFile /etc/grid-security/hostcert.pem
  SSLCertificateKeyFile /etc/grid-security/hostkey.pem
  SSLCertificateChainFile /etc/grid-security/igtf-ca-bundle.crt
  BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
</VirtualHost>

Restart apache

systemctl restart httpd

Testing the CredMon

To test the CredMon, create a submit file containg the lines:

universe = vanilla

output = scitokens_test.$(cluster).$(process).out
error = scitokens_test.$(cluster).$(process).err
log = scitokens_test.$(cluster).log

executable = /usr/bin/env

use_oauth_services = scitokens

scitokens_oauth_permissions = read:/user/dbrown
scitokens_oauth_resource = sugwg-scitokens.phy.syr.edu

requirements = TARGET.CondorVersion is "$CondorVersion: 8.8.0 Jan 03 2019 BuildID: 457757 PackageID: 8.8.0-1 $"

queue

Then submit this job in the usual way:

condor_submit token-test.sub

Instead of submitting a job, condor will respond asking you to visit a web page:

Submitting job(s)
Hello, dbrown.
Please visit: https://sugwg-scitokens.phy.syr.edu:8080/key/3e7286b3cd209baecb2e25f08b4ba9a57d7cb8ce4795930072190ccd3255d64b

There is currently a condor bug that incoreectly sets the ownership of the credential file to root. If you try and visit this URL, you will get a 500 error as the CredMon (runing as condor) can't read this file. You need to manually fix this as root by running the command:

chown condor:condor /var/lib/condor/credential/3e7286b3cd209baecb2e25f08b4ba9a57d7cb8ce4795930072190ccd3255d64b

Then visit the web page and authenticate. Once you have authenticated, you can re-submit with

condor_submit token-test.sub

and you will get a valid submission with a job ID:

Submitting job(s).
1 job(s) submitted to cluster 6509688.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment