cleanup StoRM stuck PtP requests

README.md

Summary

This bug:

https://issues.infn.it/jira/browse/STOR-779

prevents the proper cleanup of ongoing PtPs on a surl when srmRm is called on that surl on all StoRM versions >= 1.11.5.

This bug will be fixed in StoRM 1.11.8, scheduled for release on Friday, March 13th 2015.

Workaround

Cleanup the stuck SURLs by hand using the scripts in this GIST.

• count_busy_surls.sh: will display the surls that are considered to be 'stuck', i.e. for which there is an ongoing PtP request and there have been other requests resulting in 'SRM_FILE_BUSY'. Note that the script cannot tell if the PtP requests that ended up in 'SRM_FILE_BUSY' are due to the surl being stuck or due to concurrent PtP requests on the same SURL. This will have to be sorted out by the StoRM administrator.

• abort_ptp_by_token.sh: aborts a PtP given a request token. This script can take the token produced by the count_busy_surls.sh as input.

Example usage

# get the surls that are considered busy/stuck
MYSQL_USER=storm MYSQL_PASSWORD=thepassword sh count_busy_surls.sh > busy_surls.out

# Inspect the output and choose which SURLs to 'unlock'
# The following command, for instance, unlocks all SAM surls
# found in the busy surl list
for t in $(grep SAM busy_surls.out | cut -f1); do \
  MYSQL_USER=storm MYSQL_PASSWORD=thepassword sh abort_ptp_by_token.sh $t; \
done

abort_ptp_by_token.sh

#!/bin/bash
# This script aborts an ongoing PtP given a token passed as argument

MYSQL_USER=${MYSQL_USER:-root}
MYSQL_PASSWORD=${MYSQL_PASSWORD:-the_password}

TOKEN=${1:-NULL}
query="UPDATE status_Put sp JOIN (request_Put rp, request_queue rq) \
        ON sp.request_PutID=rp.ID AND rp.request_queueID=rq.ID \
        SET sp.statusCode=20, rq.status=20, sp.explanation='aborted' \
        WHERE rq.r_token = '$TOKEN' AND sp.statusCode=24";

echo $query

mysql -u$MYSQL_USER -p$MYSQL_PASSWORD storm_db -e "${query}"

count_busy_surls.sh

#!/bin/bash
# This script counts the number of apparently stuck PtP requests in the StoRM database
# and prints them in the following format
#
# token num_reqs blocked_surl first_ptp_request_timestamp
# 0abcc50a-5405-4a21-92cb-52a539572051	1199	srm://storm-fe.cr.cnaf.infn.it:8444/atlas/atlasdatadisk/SAM/testfile-GET-ATLASDATADISK.txt	2015-02-23 10:45:17
# db5a3077-fd1a-48d6-8f9c-2de390f0f231	1148	srm://storm-fe.cr.cnaf.infn.it:8444/atlas/atlasscratchdisk/SAM/testfile-GET-ATLASSCRATCHDISK.txt	2015-02-23 10:50:40
#

MYSQL_USER=${MYSQL_USER:-root}
MYSQL_PASSWORD=${MYSQL_PASSWORD:-the_password}

query="SELECT x.r_token as token, count(*) as req_count, x.targetSURL as surl, x.timestamp as time from ( select targetSURL_uniqueID, targetSURL, r_token, timestamp from request_queue rq JOIN (request_Put rp, status_Put sp) ON (rp.request_queueID = rq.ID AND sp.request_PutID = rp.ID) WHERE sp.statusCode = 24 ) AS x INNER JOIN ( select targetSURL_uniqueID, timestamp from request_queue rq2 JOIN (request_Put as rp2, status_Put as sp2) ON (rp2.request_queueID = rq2.ID AND sp2.request_PutID = rp2.ID ) WHERE sp2.statusCode = 30 ) as y on x.targetSURL_uniqueID = y.targetSURL_uniqueID where x.timestamp < y.timestamp group by x.r_token order by req_count desc"

mysql -N -u$MYSQL_USER -p$MYSQL_PASSWORD storm_db -e "${query}"