Skip to content

Instantly share code, notes, and snippets.

Last active January 28, 2022 02:59
Show Gist options
  • Save duncan-brown/fb5e83b86814baeda001316a6bdfcc3b to your computer and use it in GitHub Desktop.
Save duncan-brown/fb5e83b86814baeda001316a6bdfcc3b to your computer and use it in GitHub Desktop.
Description of how SciTokens was set up in Syracuse

Syracuse SciTokens Setup

Install software

Switch to Condor 8.8 in /etc/yum.repos.d/htcondor-stable-rhel7.repo then upgrade Condor with

systemctl stop condor
yum upgrade condor
systemctl start condor

Install OSG 3.4 RPMs for xrootd and scitokens

rpm -e osg-release-3.3-7.osg33.el7.noarch
rpm -ivh
yum install xrootd
yum install xrootd-scitokens
yum install xrootd-multiuser
yum install stashcp

Set up the XrootD config file

Edit the file /etc/xrootd/xrootd-http.cfg add the following two lines to configure

xrootd.fslib default
multiuser.umask 0022

Then add the following lines to set up


# Pass the bearer token to the Xrootd authorization framework.
http.header2cgi Authorization authz

The cmsd@http.service service reads this file and chokes on the xrd.protocol line, so edit the file to protect this with an if statement:

if exec xrootd
xrd.protocol XrdHttp:8000 /usr/lib64/

To get cmsd working, add the lines

all.role server
xrd.port 1094

all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd

Make sure that these directories exist and have the right permissions:

mkdir -p /var/run/xrootd/http
chowm -R xrootd:xrootd /var/run/xrootd

To register with the OSG redirector, add the line

all.manager 1213

For debugging, it is also helpful to add

ofs.trace all
oss.trace all

Set up SciTokens config file for XrootD

Create a file named /etc/xrootd/scitokens.cfg contining the lines

audience =

[Issuer Scitokens-Jeff]
issuer =
base_path = /
map_subject = True

[Issuer Demo]
issuer =
base_path = /
map_subject = True

Create the directory structure for XrootD

This assumes that the files really live in /stash/user.

mkdir -p /stash/user/dbrown
chown dbrown:lsc /stash/user/dbrown
mkdir /mnt/stash/osg
cd /mnt/stash/osg
ln -s /stash sugwg

Start XrootD and CMSD

Start the priveleged XrootD server with

systemctl start xrootd-privileged@http
systemctl start cmsd@http.service

Test XrootD

Do the following as an unpriveleged user:

echo "Hello, World!" > hello.txt

Test a write

Go to and create a token with the following additional lines in the payload

  "scope": "write:/osg/sugwg/user/dbrown",
  "sub": "dbrown",
  "aud": ""

Now try and write a file to the XrootD server with

curl -v --connect-timeout 30 --speed-limit 1024 -X PUT --fail --upload-file hello.txt -H "Authorization: Bearer TOKEN"

where TOKEN is the token created by (Note: the token must be RS256 encoded). You should see

* About to connect() to port 8000 (#0)
*   Trying
* Connected to ( port 8000 (#0)
> PUT /osg/sugwg/user/dbrown/hello.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host:
> Accept: */*
> Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImtleS1yczI1NiJ9.eyJpc3MiOiJodHRwczovL2RlbW8uc2NpdG9rZW5zLm9yZyIsImV4cCI6MTU0NzY1Mjg1OSwiaWF0IjoxNTQ3NjUyMjU5LCJuYmYiOjE1NDc2NTIyNTksImp0aSI6Ijk5NGM1ZWRmLTA1ZDYtNDFiYi04NGZhLTgwNDRjOWRhN2YwZiIsInNjb3BlIjoid3JpdGU6L29zZy9zdWd3Zy91c2VyL2Ricm93biIsInN1YiI6ImRicm93biIsImF1ZCI6InN1Z3dnLXNjaXRva2Vucy5waHkuc3lyLmVkdSJ9.tzRRtP6IJ1e86gIVif8i-e94scJXSaMxeWiogFAUo99P3btNlHqou7gwp5KvxDK8jGOp_8OquNjiPJwSaDyqYpGMrx40JguRTkk5b9TBRzWpiWQm1mJwQGiyOnVQ15je8FgPdkNjF8Sn8q7l26dEEEyWvWKOrlEKqq7xf6KXG-uIOa9pdcFErzi5UWGATXaZORIZiHtQuIMRXojOWMNwPoMgR3gRCWAqjasgOX7BNpJbZVNMGMQpgArqXNiGoP0OARotN9kTWS9cf_0BKQ7IpPtYmztRhK3X2vV7KtUHoJPEjioO7wdTG9wpverbAk46__u6xAE6OmCVsC_d2BCFqA
> Content-Length: 13
> Expect: 100-continue
< HTTP/1.1 100 Unknown
< Connection: Keep-Alive
< Content-Length: 0
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Connection: Close
< Content-Length: 3
* Closing connection 0

Test a read

Go to and create a token with the following additional lines in the payload

  "scope": "read:/osg/sugwg/user/dbrown",
  "sub": "dbrown",
  "aud": ""

Grab the token generated and use it to download a file, e.g.

curl -H "Authorization: Bearer TOKEN"

This should return

Hello, world!

Install Condor CredMon

These instructions assume that you want to run the Condor CredMon on port 8080 on the submit host under apache. To work around a Condor bug, this requires the magic knob ZKMHOST to be set in the config file.

Install the Condor CredMon by running the following command as root:

pip install git+

Register with Jeff's SciToken Server

Go to Register at Record the client ID and secret for use in /etc/condor/config.d/10-credmon.conf below.

Set up the CredMon

Then create file /etc/condor/config.d/10-credmon.conf containing the lines below, setting using the information from the previous step to replace ID_FROM_JEFF and SECRET_FROM_JEFF:

# Set up the credmon and credential directory
SEC_CREDENTIAL_MONITOR = /usr/bin/condor_credmon
SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credential
SEC_CREDENTIAL_MONITOR_LOG = /var/log/condor/CredmonLog
TOKENS = true

# Define OAuth token providers

Edit condor_config.local and add CREDD to the DAEMON_LIST:

# Enable the CredD

Create the credentials directory and fix the permissions

mkdir -p /var/lib/condor/credential
chown condor:condor /var/lib/condor/credential
chmod 700 /var/lib/condor/credential

Starting the CredMon

The manager currently isn't supervising the CredMon, so start it manually with the command

su - condor
nohup condor_credmon </dev/null &>/dev/null &

Configure apache for the CredMon web page

Create a file /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi containing the lines

# Set up the credmon and credential directory
SEC_CREDENTIAL_MONITOR = /usr/bin/condor_credmon
SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credential
SEC_CREDENTIAL_MONITOR_LOG = /var/log/condor/CredmonLog
TOKENS = true

# Define OAuth token providers
SCITOKENS_CLIENT_ID = myproxy:oa4mp,2012:/client_id/3dc18747287b0f29805bcee37f2fd3c8
SCITOKENS_CLIENT_SECRET = WEzL7U5wW9lnCzezv9mOV_mdHcwnH74zj-vKSbkOfmplOmTbfTKkHQ
[dbrown@sugwg-scitokens stash_cp_test]$ cat /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi
# Configure Logging

from credmon.utils import setup_logging, get_cred_dir
import os
import logging

cred_dir = get_cred_dir()
logger = setup_logging(log_path = os.path.join(cred_dir, 'oauth_credmon_webserver.log'),
                       log_level = logging.DEBUG)

# Start Service
from credmon.CredentialMonitors.OAuthCredmonWebserver import app
app.secret_key = 'HereComeTheBadgers'

application = app

Create a file /etc/httpd/conf.d/condor-credmon.conf containing



  ## Vhost docroot
  DocumentRoot /var/www/cgi-bin/wsgi/condor-credmon

  <Directory "/var/www/cgi-bin/wsgi/condor-credmon">
    Options -Indexes
    AllowOverride None
    Order allow,deny
    Allow from all
    SSLOptions +StdEnvVars +ExportCertData

  ## Logging
  ErrorLog "/var/log/httpd/condor-credmon_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/condor-credmonhttp_access.log" combined

  WSGIDaemonProcess     CondorCredmon user=condor group=condor processes=2 threads=25 python-path=/usr/lib64/pegasus/externals/python
  WSGIProcessGroup      CondorCredmon
  WSGIScriptAlias       / /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi

  SSLEngine on
  SSLProtocol all -SSLv2 -SSLv3
  SSLHonorCipherOrder on

  SSLCertificateFile /etc/grid-security/hostcert.pem
  SSLCertificateKeyFile /etc/grid-security/hostkey.pem
  SSLCertificateChainFile /etc/grid-security/igtf-ca-bundle.crt
  BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

Restart apache

systemctl restart httpd

Testing the CredMon

To test the CredMon, create a submit file containg the lines:

universe = vanilla

output = scitokens_test.$(cluster).$(process).out
error = scitokens_test.$(cluster).$(process).err
log = scitokens_test.$(cluster).log

executable = /usr/bin/env

use_oauth_services = scitokens

scitokens_oauth_permissions = read:/user/dbrown
scitokens_oauth_resource =

requirements = TARGET.CondorVersion is "$CondorVersion: 8.8.0 Jan 03 2019 BuildID: 457757 PackageID: 8.8.0-1 $"


Then submit this job in the usual way:

condor_submit token-test.sub 

Instead of submitting a job, condor will respond asking you to visit a web page:

Submitting job(s)
Hello, dbrown.
Please visit:

There is currently a condor bug that incoreectly sets the ownership of the credential file to root. If you try and visit this URL, you will get a 500 error as the CredMon (runing as condor) can't read this file. You need to manually fix this as root by running the command:

chown condor:condor /var/lib/condor/credential/3e7286b3cd209baecb2e25f08b4ba9a57d7cb8ce4795930072190ccd3255d64b

Then visit the web page and authenticate. Once you have authenticated, you can re-submit with

condor_submit token-test.sub 

and you will get a valid submission with a job ID:

Submitting job(s).
1 job(s) submitted to cluster 6509688.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment