Gromacs/Lsdmap workflow on Stampede fail

grlsdmapFails2

[ExTASY-toolsOct2] ardita@tirith 114% extasy --RPconfig stampede.rcfg --Kconfig gromacslsdmap.wcfg
Session UID: 5430103554737d225a00a395
Pilot UID       : 5430103954737d225a00a397
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/amber.json
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/test.json
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/coco.json
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json
Loading kernel configurations from /users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/mdkernels/configs/namd.json
Prepare grofiles..
[Callback]: ComputePilot '5430103954737d225a00a397' state changed to Launching.
Preprocessing time :  1.41648793221
Cycle 0
Starting Simulation
[Callback]: ComputeUnit '5430103c54737d225a00a3a7' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a399' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3c9' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a8' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b7' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3af' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a9' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b9' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a39a' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3bf' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d3' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3cf' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a0' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3c0' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3ac' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d5' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3c2' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3c1' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d8' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3ae' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3ab' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3c4' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b0' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a39b' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3c5' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d0' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a39c' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b4' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b1' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a39d' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3bc' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b2' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3aa' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b3' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3c6' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3ba' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a5' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b6' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a39f' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b5' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a39e' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3ca' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3cb' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d6' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a1' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a3' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3ad' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3cd' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3c7' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a2' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3cc' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3b8' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d4' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d1' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3bd' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3be' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a4' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3c3' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3ce' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d2' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a6' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3bb' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3c8' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103d54737d225a00a3d7' state changed to PendingInputStaging.
[Callback]: ComputeUnit '5430103c54737d225a00a3a7' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a399' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3ab' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3c4' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a7' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3b0' state changed to StagingInput.
[Callback]: ComputePilot '5430103954737d225a00a397' state changed to PendingActive.
[Callback]: ComputeUnit '5430103c54737d225a00a39b' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a399' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c5' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3d0' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3ab' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a39c' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b4' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3c4' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3b1' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b0' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a39d' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3bc' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a39b' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3b2' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3aa' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b3' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3c5' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3c6' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3ba' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3d0' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a5' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b6' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a39c' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a39f' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b5' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b4' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a39e' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3ca' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b1' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3cb' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3d6' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a39d' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a1' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a3' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3ad' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3bc' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3cd' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3c7' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3aa' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3c9' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a8' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3c9' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3b7' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3af' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a8' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a9' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b9' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b7' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a39a' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3bf' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3af' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3d3' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3cf' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a0' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a9' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c0' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3ac' state changed to StagingInput.
[Callback]: ComputePilot '5430103954737d225a00a397' state changed to Active.
[Callback]: ComputeUnit '5430103c54737d225a00a3a7' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a399' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b9' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a7' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3c9' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d5' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a399' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3c9' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a8' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3a8' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b7' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b7' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3af' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3af' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a9' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b9' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b9' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3ab' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3ab' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3c4' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3c4' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b0' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3c2' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b0' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a39b' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a39b' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3c5' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3c5' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d0' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d0' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a39c' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a39c' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b4' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b4' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b1' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b1' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a39d' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a39d' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3bc' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3aa' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3aa' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a39a' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c1' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a39a' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a39a' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d8' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a7' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3c9' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a8' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b7' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3af' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3ab' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3c4' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b0' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a39b' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3c5' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d0' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a39c' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a9' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b9' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b4' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3bf' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3ae' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3bf' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b1' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a39d' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3bf' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3bc' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3aa' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a2' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a39a' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d3' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3cc' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3b8' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3d3' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3bf' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d3' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d4' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3cf' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3d3' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3cf' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d1' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3cf' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a399' state changed to StagingOutput.
[Callback]: ComputeUnit '5430103c54737d225a00a399' state changed to Failed.
            Log: Output transfer failed: Insufficient system resources: Insufficient system resources: read from process failed '[Errno 5] Input/output error' : (Shared connection to stampede.tacc.utexas.edu closed.
) ((Shared connection to stampede.tacc.utexas.edu closed.
)) (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_exceptions.py +59 (translate_exception)  :  e = se.NoSuccess ("Insufficient system resources: %s" % cmsg))
Traceback (most recent call last):
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 148, in run
    session=self._session
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/filesystem/file.py", line 86, in __init__
    _adaptor, _adaptor_state, _ttype=_ttype)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/namespace/entry.py", line 89, in __init__
    url, flags, session, ttype=_ttype)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/base.py", line 99, in __init__
    self._init_task = self._adaptor.init_instance (adaptor_state, *args, **kwargs)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/cpi/decorators.py", line 51, in wrap_function
    return sync_function (self, *args, **kwargs)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 972, in init_instance
    self.shell = self.lm.lease (lease_tgt, self.shell_creator, self.url)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/utils/lease_manager.py", line 206, in lease
    obj = self._create_object (pool_id, creator, args)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/utils/lease_manager.py", line 146, in _create_object
    obj = _LeaseObject (self, creator, args)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/utils/lease_manager.py", line 27, in __init__
    self.obj   = creator (*args)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/adaptors/shell/shell_file.py", line 965, in _shell_creator
    return sups.PTYShell (url, self.session, self._logger)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_shell.py", line 228, in __init__
    self.pty_shell  = self.factory.run_shell  (self.pty_info)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_shell_factory.py", line 418, in run_shell
    self._initialize_pty (sh_slave, info, is_shell=True)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_shell_factory.py", line 384, in _initialize_pty
    raise ptye.translate_exception (e)
NoSuccess: Insufficient system resources: Insufficient system resources: read from process failed '[Errno 5] Input/output error' : (Shared connection to stampede.tacc.utexas.edu closed.
) ((Shared connection to stampede.tacc.utexas.edu closed.
)) (/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/saga/utils/pty_exceptions.py +59 (translate_exception)  :  e = se.NoSuccess ("Insufficient system resources: %s" % cmsg))

[Callback]: ComputeUnit '5430103d54737d225a00a3cf' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3bd' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a0' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3be' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a0' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3a0' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a4' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a0' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3c0' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c3' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3c0' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3c0' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3ce' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3c0' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3ac' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3d2' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3a6' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3ac' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3ac' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d5' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3ac' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3bb' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3c8' state changed to StagingInput.
[Callback]: ComputeUnit '5430103d54737d225a00a3d5' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d5' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d5' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d7' state changed to StagingInput.
[Callback]: ComputeUnit '5430103c54737d225a00a3c2' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c1' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3d8' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c2' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3c2' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3ae' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c1' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3c1' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d8' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d8' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3ae' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3a2' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a2' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3ae' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a2' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3cc' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3b8' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3d4' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c2' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3cc' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3cc' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b8' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d1' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c1' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d4' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d8' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3ae' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d4' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d1' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3bd' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3a2' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3bd' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3be' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a4' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3c3' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3be' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3be' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3ce' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3cc' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a4' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b8' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d4' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a4' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3c3' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d2' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3d1' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3ce' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d2' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3bd' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d2' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a6' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3bb' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3c8' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3d7' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a6' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3a6' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3bb' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3be' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a4' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3c3' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3bb' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3c8' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d7' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b2' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3ce' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d7' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b2' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b3' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d2' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b3' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3c6' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3ba' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a5' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3c6' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3a6' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3c6' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3ba' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b6' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3bb' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3ba' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a5' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b6' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3c8' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d7' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b2' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b3' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a5' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b6' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a39f' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3b5' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a39e' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3ca' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3b5' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3c6' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b5' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a39e' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3cb' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3ba' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a39e' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3ca' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3cb' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3d6' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3a5' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3b6' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a39f' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3d6' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a1' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a3' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103d54737d225a00a3cd' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a3a1' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3b5' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a1' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a3' state changed to Scheduling.
[Callback]: ComputeUnit '5430103c54737d225a00a3ad' state changed to PendingExecution.
[Callback]: ComputeUnit '5430103c54737d225a00a39e' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3ca' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a3' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3cd' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3cb' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3cd' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3ad' state changed to Executing.
[Callback]: ComputeUnit '5430103d54737d225a00a3c7' state changed to Scheduling.
[Callback]: ComputeUnit '5430103d54737d225a00a3d6' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3c7' state changed to Executing.
[Callback]: ComputeUnit '5430103c54737d225a00a3a1' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3a3' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3cd' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103c54737d225a00a3ad' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
[Callback]: ComputeUnit '5430103d54737d225a00a3c7' state changed to Failed.
            Log: All FTW Input Staging Directives done - 1.
Traceback (most recent call last):
  File "/users/ardita/ExTASY-toolsOct2/bin/extasy", line 9, in <module>
    load_entry_point('radical.ensemblemd.extasy==0.1', 'console_scripts', 'extasy')()
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/extasy/bin/runme.py", line 130, in main
    Simulator(umgr,RPconfig,Kconfig,i)
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/ensemblemd/extasy/bin/Simulator/Gromacs/simulator.py", line 64, in Simulator
    with open('out%s.gro' % i, 'r') as output_file:
IOError: [Errno 2] No such file or directory: 'out0.gro'
Exception in thread OutputFileTransferWorker-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 79, in run
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 1677, in find_and_modify
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 445, in command
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 345, in _command
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in __send_message
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/unit_manager_controller.py", line 296, in run
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/unit_manager_controller.py", line 205, in call_unit_state_callbacks
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/db/database.py", line 753, in publish_compute_unit_callback_history
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 572, in update
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1136, in _send_message
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/pool.py", line 395, in maybe_return_socket
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/pool.py", line 81, in __eq__
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
Exception in thread OutputFileTransferWorker-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/output_file_transfer_worker.py", line 79, in run
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 1677, in find_and_modify
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 445, in command
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 345, in _command
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in __send_message
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread InputFileTransferWorker-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/input_file_transfer_worker.py", line 232, in run
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/collection.py", line 1677, in find_and_modify
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 445, in command
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/database.py", line 345, in _command
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in __send_message
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/controller/pilot_manager_controller.py", line 269, in run
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/radical/pilot/db/database.py", line 427, in get_pilots
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1058, in next
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 1002, in _refresh
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/cursor.py", line 915, in __send_message
  File "/users/ardita/ExTASY-toolsOct2/lib/python2.6/site-packages/pymongo/mongo_client.py", line 1207, in _send_message_with_response
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'