Syracuse SciTokens Setup
Install software
Switch to Condor 8.8 in /etc/yum.repos.d/htcondor-stable-rhel7.repo
then upgrade Condor with
systemctl stop condor
yum upgrade condor
systemctl start condor
Install OSG 3.4 RPMs for xrootd and scitokens
rpm -e osg-release-3.3-7.osg33.el7.noarch
rpm -ivh https://repo.opensciencegrid.org/osg/3.4/osg-3.4-el7-release-latest.rpm
yum install xrootd
yum install xrootd-scitokens
yum install xrootd-multiuser
yum install stashcp
Set up the XrootD config file
Edit the file /etc/xrootd/xrootd-http.cfg
add the following two lines to configure https://github.com/opensciencegrid/xrootd-multiuser
xrootd.fslib libXrdMultiuser.so default
multiuser.umask 0022
Then add the following lines to set up https://github.com/scitokens/xrootd-scitokens
ofs.authorize
ofs.authlib libXrdAccSciTokens.so
# Pass the bearer token to the Xrootd authorization framework.
http.header2cgi Authorization authz
The cmsd@http.service
service reads this file and chokes on the xrd.protocol
line, so edit the file to protect this with an if statement:
if exec xrootd
xrd.protocol XrdHttp:8000 /usr/lib64/libXrdHttp-4.so
fi
To get cmsd working, add the lines
all.role server
xrd.port 1094
all.adminpath /var/spool/xrootd
all.pidpath /var/run/xrootd
Make sure that these directories exist and have the right permissions:
mkdir -p /var/run/xrootd/http
chowm -R xrootd:xrootd /var/run/xrootd
To register with the OSG redirector, add the line
all.manager redirector.osgstorage.org+ 1213
For debugging, it is also helpful to add
ofs.trace all
oss.trace all
Set up SciTokens config file for XrootD
Create a file named /etc/xrootd/scitokens.cfg
contining the lines
[Global]
audience = sugwg-scitokens.phy.syr.edu
[Issuer Scitokens-Jeff]
issuer = https://surge.ncsa.illinois.edu/scitokens-server
base_path = /
map_subject = True
[Issuer Demo]
issuer = https://demo.scitokens.org
base_path = /
map_subject = True
Create the directory structure for XrootD
This assumes that the files really live in /stash/user
.
mkdir -p /stash/user/dbrown
chown dbrown:lsc /stash/user/dbrown
mkdir /mnt/stash/osg
cd /mnt/stash/osg
ln -s /stash sugwg
Start XrootD and CMSD
Start the priveleged XrootD server with
systemctl start xrootd-privileged@http
systemctl start cmsd@http.service
Test XrootD
Do the following as an unpriveleged user:
echo "Hello, World!" > hello.txt
Test a write
Go to https://demo.scitokens.org/ and create a token with the following additional lines in the payload
"scope": "write:/osg/sugwg/user/dbrown",
"sub": "dbrown",
"aud": "sugwg-scitokens.phy.syr.edu"
Now try and write a file to the XrootD server with
curl -v --connect-timeout 30 --speed-limit 1024 -X PUT --fail --upload-file hello.txt -H "Authorization: Bearer TOKEN" http://sugwg-scitokens.phy.syr.edu:8000/osg/sugwg/user/dbrown/hello.txt
where TOKEN
is the token created by https://demo.scitokens.org (Note: the token must be RS256 encoded). You should see
* About to connect() to sugwg-scitokens.phy.syr.edu port 8000 (#0)
* Trying 128.230.146.14...
* Connected to sugwg-scitokens.phy.syr.edu (128.230.146.14) port 8000 (#0)
> PUT /osg/sugwg/user/dbrown/hello.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sugwg-scitokens.phy.syr.edu:8000
> Accept: */*
> Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImtleS1yczI1NiJ9.eyJpc3MiOiJodHRwczovL2RlbW8uc2NpdG9rZW5zLm9yZyIsImV4cCI6MTU0NzY1Mjg1OSwiaWF0IjoxNTQ3NjUyMjU5LCJuYmYiOjE1NDc2NTIyNTksImp0aSI6Ijk5NGM1ZWRmLTA1ZDYtNDFiYi04NGZhLTgwNDRjOWRhN2YwZiIsInNjb3BlIjoid3JpdGU6L29zZy9zdWd3Zy91c2VyL2Ricm93biIsInN1YiI6ImRicm93biIsImF1ZCI6InN1Z3dnLXNjaXRva2Vucy5waHkuc3lyLmVkdSJ9.tzRRtP6IJ1e86gIVif8i-e94scJXSaMxeWiogFAUo99P3btNlHqou7gwp5KvxDK8jGOp_8OquNjiPJwSaDyqYpGMrx40JguRTkk5b9TBRzWpiWQm1mJwQGiyOnVQ15je8FgPdkNjF8Sn8q7l26dEEEyWvWKOrlEKqq7xf6KXG-uIOa9pdcFErzi5UWGATXaZORIZiHtQuIMRXojOWMNwPoMgR3gRCWAqjasgOX7BNpJbZVNMGMQpgArqXNiGoP0OARotN9kTWS9cf_0BKQ7IpPtYmztRhK3X2vV7KtUHoJPEjioO7wdTG9wpverbAk46__u6xAE6OmCVsC_d2BCFqA
> Content-Length: 13
> Expect: 100-continue
>
< HTTP/1.1 100 Unknown
< Connection: Keep-Alive
< Content-Length: 0
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Connection: Close
< Content-Length: 3
<
* Closing connection 0
:-)
Test a read
Go to https://demo.scitokens.org/ and create a token with the following additional lines in the payload
"scope": "read:/osg/sugwg/user/dbrown",
"sub": "dbrown",
"aud": "sugwg-scitokens.phy.syr.edu"
Grab the token generated and use it to download a file, e.g.
curl -H "Authorization: Bearer TOKEN" http://sugwg-scitokens.phy.syr.edu:8000/osg/sugwg/user/dbrown/hello.txt
This should return
Hello, world!
Install Condor CredMon
These instructions assume that you want to run the Condor CredMon on port 8080 on the submit host under apache. To work around a Condor bug, this requires the magic knob ZKMHOST
to be set in the config file.
Install the Condor CredMon by running the following command as root:
pip install git+https://github.com/htcondor/scitokens-credmon
Register with Jeff's SciToken Server
Go to https://surge.ncsa.illinois.edu/scitokens-server/.well-known Register at https://surge.ncsa.illinois.edu/scitokens-server/register Record the client ID and secret for use in /etc/condor/config.d/10-credmon.conf below.
Set up the CredMon
Then create file /etc/condor/config.d/10-credmon.conf containing the lines below, setting using the information from the previous step to replace ID_FROM_JEFF
and SECRET_FROM_JEFF
:
# Set up the credmon and credential directory
SEC_CREDENTIAL_MONITOR = /usr/bin/condor_credmon
SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credential
SEC_CREDENTIAL_MONITOR_LOG = /var/log/condor/CredmonLog
TOKENS = true
ZKMHOST = sugwg-scitokens.phy.syr.edu:8080
# Define OAuth token providers
SCITOKENS_CLIENT_ID = ID_FROM_JEFF
SCITOKENS_CLIENT_SECRET = SECRET_FROM_JEFF
SCITOKENS_RETURN_URL_SUFFIX = /return/scitokens
SCITOKENS_AUTHORIZATION_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/authorize
SCITOKENS_TOKEN_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/token
SCITOKENS_USER_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/userinfo
Edit condor_config.local
and add CREDD
to the DAEMON_LIST
:
# Enable the CredD
DAEMON_LIST = $(DAEMON_LIST), CREDD
Create the credentials directory and fix the permissions
mkdir -p /var/lib/condor/credential
chown condor:condor /var/lib/condor/credential
chmod 700 /var/lib/condor/credential
Starting the CredMon
The manager currently isn't supervising the CredMon, so start it manually with the command
su - condor
nohup condor_credmon </dev/null &>/dev/null &
exit
Configure apache for the CredMon web page
Create a file /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi
containing the lines
# Set up the credmon and credential directory
SEC_CREDENTIAL_MONITOR = /usr/bin/condor_credmon
SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credential
SEC_CREDENTIAL_MONITOR_LOG = /var/log/condor/CredmonLog
TOKENS = true
ZKMHOST = sugwg-scitokens.phy.syr.edu:8080
# Define OAuth token providers
SCITOKENS_CLIENT_ID = myproxy:oa4mp,2012:/client_id/3dc18747287b0f29805bcee37f2fd3c8
SCITOKENS_CLIENT_SECRET = WEzL7U5wW9lnCzezv9mOV_mdHcwnH74zj-vKSbkOfmplOmTbfTKkHQ
SCITOKENS_RETURN_URL_SUFFIX = /return/scitokens
SCITOKENS_AUTHORIZATION_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/authorize
SCITOKENS_TOKEN_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/token
SCITOKENS_USER_URL = https://surge.ncsa.illinois.edu:443/scitokens-server/userinfo
[dbrown@sugwg-scitokens stash_cp_test]$ cat /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi
#
# Configure Logging
#
from credmon.utils import setup_logging, get_cred_dir
import os
import logging
cred_dir = get_cred_dir()
logger = setup_logging(log_path = os.path.join(cred_dir, 'oauth_credmon_webserver.log'),
log_level = logging.DEBUG)
#
# Start Service
#
from credmon.CredentialMonitors.OAuthCredmonWebserver import app
app.secret_key = 'HereComeTheBadgers'
application = app
Create a file /etc/httpd/conf.d/condor-credmon.conf
containing
Listen 128.230.146.14:8080
<VirtualHost 128.230.146.14:8080>
ServerName sugwg-scitokens.phy.syr.edu
ServerAdmin dabrown@syr.edu
## Vhost docroot
DocumentRoot /var/www/cgi-bin/wsgi/condor-credmon
<Directory "/var/www/cgi-bin/wsgi/condor-credmon">
Options -Indexes
AllowOverride None
Order allow,deny
Allow from all
SSLOptions +StdEnvVars +ExportCertData
</Directory>
## Logging
ErrorLog "/var/log/httpd/condor-credmon_error.log"
ServerSignature Off
CustomLog "/var/log/httpd/condor-credmonhttp_access.log" combined
WSGIDaemonProcess CondorCredmon user=condor group=condor processes=2 threads=25 python-path=/usr/lib64/pegasus/externals/python
WSGIProcessGroup CondorCredmon
WSGIScriptAlias / /var/www/cgi-bin/wsgi/condor-credmon/condor-credmon.wsgi
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS"
SSLCertificateFile /etc/grid-security/hostcert.pem
SSLCertificateKeyFile /etc/grid-security/hostkey.pem
SSLCertificateChainFile /etc/grid-security/igtf-ca-bundle.crt
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
</VirtualHost>
Restart apache
systemctl restart httpd
Testing the CredMon
To test the CredMon, create a submit file containg the lines:
universe = vanilla
output = scitokens_test.$(cluster).$(process).out
error = scitokens_test.$(cluster).$(process).err
log = scitokens_test.$(cluster).log
executable = /usr/bin/env
use_oauth_services = scitokens
scitokens_oauth_permissions = read:/user/dbrown
scitokens_oauth_resource = sugwg-scitokens.phy.syr.edu
requirements = TARGET.CondorVersion is "$CondorVersion: 8.8.0 Jan 03 2019 BuildID: 457757 PackageID: 8.8.0-1 $"
queue
Then submit this job in the usual way:
condor_submit token-test.sub
Instead of submitting a job, condor will respond asking you to visit a web page:
Submitting job(s)
Hello, dbrown.
Please visit: https://sugwg-scitokens.phy.syr.edu:8080/key/3e7286b3cd209baecb2e25f08b4ba9a57d7cb8ce4795930072190ccd3255d64b
There is currently a condor bug that incoreectly sets the ownership of the credential file to root. If you try and visit this URL, you will get a 500 error as the CredMon (runing as condor) can't read this file. You need to manually fix this as root by running the command:
chown condor:condor /var/lib/condor/credential/3e7286b3cd209baecb2e25f08b4ba9a57d7cb8ce4795930072190ccd3255d64b
Then visit the web page and authenticate. Once you have authenticated, you can re-submit with
condor_submit token-test.sub
and you will get a valid submission with a job ID:
Submitting job(s).
1 job(s) submitted to cluster 6509688.