HTCondor kerberos debugging

client side

[phxlk@hd-38-36 test]$ _condor_TOOL_DEBUG=D_SECURITY condor_submit first.job -name lcgce02.phy.bris.ac.uk -debug
08/07/14 15:24:31 KEYCACHE: created: 0x7758e0
08/07/14 15:24:31 SECMAN: command 6 QUERY_SCHEDD_ADS to collector at <137.222.79.6:9618> from TCP port 32927 (blocking).
08/07/14 15:24:31 SECMAN: new session, doing initial authentication.
08/07/14 15:24:31 SECMAN: Auth methods: FS,PASSWORD,CLAIMTOBE,KERBEROS
08/07/14 15:24:31 HANDSHAKE: in handshake(my_methods = 'FS,PASSWORD,CLAIMTOBE,KERBEROS')
08/07/14 15:24:31 HANDSHAKE: handshake() - i am the client
08/07/14 15:24:31 HANDSHAKE: sending (methods == 582) to server
08/07/14 15:24:31 HANDSHAKE: server replied (method = 4)
08/07/14 15:24:31 AUTHENTICATE_FS: used dir /tmp/FS_XXX9tGHij, status: 0
08/07/14 15:24:31 AUTHENTICATE: method 4 (FS) failed.
08/07/14 15:24:31 HANDSHAKE: in handshake(my_methods = 'PASSWORD,CLAIMTOBE,KERBEROS')
08/07/14 15:24:31 HANDSHAKE: handshake() - i am the client
08/07/14 15:24:31 HANDSHAKE: sending (methods == 578) to server
08/07/14 15:24:31 HANDSHAKE: server replied (method = 512)
08/07/14 15:24:31 PW.
08/07/14 15:24:31 PW: getting name.
08/07/14 15:24:31 PW: Generating ra.
08/07/14 15:24:31 PW: Client sending.
08/07/14 15:24:31 Client sending: 0, 26(condor_pool@phy.bris.ac.uk), 256
08/07/14 15:24:31 PW: Client receiving.
08/07/14 15:24:31 Wrote server ra.
08/07/14 15:24:31 error: SEC_PASSWORD_FILE must be owned by Condor's real uid
08/07/14 15:24:31 error: SEC_PASSWORD_FILE must be owned by Condor's real uid
08/07/14 15:24:31 PW: Client setting keys.
08/07/14 15:24:31 PW: CLient sending two.
08/07/14 15:24:31 In client_send_two.
08/07/14 15:24:31 Can't send null for random string.
08/07/14 15:24:31 Client sending: 0() 0 0
08/07/14 15:24:31 Sent ok.
08/07/14 15:24:31 AUTHENTICATE: method 512 (PASSWORD) failed.
08/07/14 15:24:31 HANDSHAKE: in handshake(my_methods = 'CLAIMTOBE,KERBEROS')
08/07/14 15:24:31 HANDSHAKE: handshake() - i am the client
08/07/14 15:24:31 HANDSHAKE: sending (methods == 66) to server
08/07/14 15:24:31 HANDSHAKE: server replied (method = 2)
08/07/14 15:24:31 Authentication was a Success.
08/07/14 15:24:31 ZKM: setting default map to (null)
08/07/14 15:24:31 ZKM: post-map: current user is '(null)'
08/07/14 15:24:31 ZKM: post-map: current domain is '(null)'
08/07/14 15:24:31 ZKM: post-map: current FQU is '(null)'
08/07/14 15:24:31 SECMAN: successfully enabled message authenticator!
08/07/14 15:24:31 SECMAN: added session lcgce01:26755:1407421469:172764 to cache for 60 seconds (3600s lease).
08/07/14 15:24:31 SECMAN: startCommand succeeded.
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission ALLOW
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission READ
08/07/14 15:24:31 ipverify: READ optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission WRITE
08/07/14 15:24:31 ipverify: WRITE optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission NEGOTIATOR
08/07/14 15:24:31 ipverify: NEGOTIATOR optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission ADMINISTRATOR
08/07/14 15:24:31 ipverify: ADMINISTRATOR optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission OWNER
08/07/14 15:24:31 ipverify: OWNER optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission CONFIG
08/07/14 15:24:31 ipverify: CONFIG optimized to deny everyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission DAEMON
08/07/14 15:24:31 ipverify: DAEMON optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission SOAP
08/07/14 15:24:31 ipverify: SOAP optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission DEFAULT
08/07/14 15:24:31 ipverify: DEFAULT optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission CLIENT
08/07/14 15:24:31 ipverify: CLIENT optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission ADVERTISE_STARTD
08/07/14 15:24:31 ipverify: ADVERTISE_STARTD optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission ADVERTISE_SCHEDD
08/07/14 15:24:31 ipverify: ADVERTISE_SCHEDD optimized to allow anyone
08/07/14 15:24:31 IPVERIFY: Subsystem SUBMIT
08/07/14 15:24:31 IPVERIFY: Permission ADVERTISE_MASTER
08/07/14 15:24:31 ipverify: ADVERTISE_MASTER optimized to allow anyone
Submitting job(s)08/07/14 15:24:32 SECMAN: command 1112 QMGMT_WRITE_CMD to schedd at <137.222.79.12:41215> from TCP port 56342 (blocking).
08/07/14 15:24:32 SECMAN: new session, doing initial authentication.
08/07/14 15:24:32 SECMAN: Auth methods: PASSWORD,KERBEROS,FS
08/07/14 15:24:32 HANDSHAKE: in handshake(my_methods = 'PASSWORD,KERBEROS,FS')
08/07/14 15:24:32 HANDSHAKE: handshake() - i am the client
08/07/14 15:24:32 HANDSHAKE: sending (methods == 580) to server
08/07/14 15:24:32 HANDSHAKE: server replied (method = 512)
08/07/14 15:24:32 PW.
08/07/14 15:24:32 PW: getting name.
08/07/14 15:24:32 PW: Generating ra.
08/07/14 15:24:32 PW: Client sending.
08/07/14 15:24:32 Client sending: 0, 26(condor_pool@phy.bris.ac.uk), 256
08/07/14 15:24:32 PW: Client receiving.
08/07/14 15:24:32 Wrote server ra.
08/07/14 15:24:32 error: SEC_PASSWORD_FILE must be owned by Condor's real uid
08/07/14 15:24:32 error: SEC_PASSWORD_FILE must be owned by Condor's real uid
08/07/14 15:24:32 PW: Client setting keys.
08/07/14 15:24:32 PW: CLient sending two.
08/07/14 15:24:32 In client_send_two.
08/07/14 15:24:32 Can't send null for random string.
08/07/14 15:24:32 Client sending: 0() 0 0
08/07/14 15:24:32 Sent ok.
08/07/14 15:24:32 AUTHENTICATE: method 512 (PASSWORD) failed.
08/07/14 15:24:32 HANDSHAKE: in handshake(my_methods = 'KERBEROS,FS')
08/07/14 15:24:32 HANDSHAKE: handshake() - i am the client
08/07/14 15:24:32 HANDSHAKE: sending (methods == 68) to server
08/07/14 15:24:32 HANDSHAKE: server replied (method = 64)
08/07/14 15:24:32 KERBEROS: krb5_unparse_name: host/lcgce02.phy.bris.ac.uk@
08/07/14 15:24:32 KERBEROS: no user yet determined, will grab up to slash
08/07/14 15:24:32 KERBEROS: picked user: host
08/07/14 15:24:32 KERBEROS: remapping 'host' to 'condor'
08/07/14 15:24:32 Failed to map principal to user
08/07/14 15:24:32 AUTHENTICATE: method 64 (KERBEROS) failed.
08/07/14 15:24:32 HANDSHAKE: in handshake(my_methods = 'FS')
08/07/14 15:24:32 HANDSHAKE: handshake() - i am the client
08/07/14 15:24:32 HANDSHAKE: sending (methods == 4) to server
08/07/14 15:24:32 HANDSHAKE: server replied (method = 4)
08/07/14 15:24:32 AUTHENTICATE_FS: used dir /tmp/FS_XXXXV0KmY, status: 0
08/07/14 15:24:32 AUTHENTICATE: method 4 (FS) failed.
08/07/14 15:24:32 HANDSHAKE: in handshake(my_methods = '')
08/07/14 15:24:32 HANDSHAKE: handshake() - i am the client
08/07/14 15:24:32 HANDSHAKE: sending (methods == 0) to server
08/07/14 15:24:32 HANDSHAKE: server replied (method = 0)
08/07/14 15:24:32 SECMAN: required authentication with schedd at <137.222.79.12:41215> failed, so aborting command QMGMT_WRITE_CMD.

ERROR: Failed to connect to queue manager lcgce02.phy.bris.ac.uk
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using FS
AUTHENTICATE:1004:Failed to authenticate using KERBEROS
AUTHENTICATE:1004:Failed to authenticate using PASSWORD

Scheduler output

08/07/14 14:30:20 DC_AUTHENTICATE: received DC_AUTHENTICATE from <10.129.5.186:51105>
08/07/14 14:30:20 DC_AUTHENTICATE: generating 3DES key for session lcgce02:32744:1407418220:285...
08/07/14 14:30:20 SECMAN: new session, doing initial authentication.
08/07/14 14:30:20 HANDSHAKE: in handshake(my_methods = 'PASSWORD,KERBEROS,FS')
08/07/14 14:30:20 HANDSHAKE: handshake() - i am the server
08/07/14 14:30:20 HANDSHAKE: client sent (methods == 580)
08/07/14 14:30:20 HANDSHAKE: i picked (method == 512)
08/07/14 14:30:20 HANDSHAKE: client received (method == 512)
08/07/14 14:30:20 PW.
08/07/14 14:30:20 PW: Server receiving 1.
08/07/14 14:30:20 Received: 0, 26(condor_pool@phy.bris.ac.uk), 256
08/07/14 14:30:20 PW: Server fetching password.
08/07/14 14:30:20 PW: Server generating rb.
08/07/14 14:30:20 PW: Server sending.
08/07/14 14:30:20 In server_send: 0.
08/07/14 14:30:20 Calculating hkt 'condor_pool@phy.bris.ac.uk' (26), 'condor_pool@phy.bris.ac.uk' (26).
08/07/14 14:30:20 Server send 'condor_pool@phy.bris.ac.uk', 'condor_pool@phy.bris.ac.uk', 256 256 20
08/07/14 14:30:20 PW: Server receiving 2.
08/07/14 14:30:20 Error from client.
08/07/14 14:30:20 AUTHENTICATE: method 512 (PASSWORD) failed.
08/07/14 14:30:20 HANDSHAKE: in handshake(my_methods = 'PASSWORD,KERBEROS,FS')
08/07/14 14:30:20 HANDSHAKE: handshake() - i am the server
08/07/14 14:30:20 HANDSHAKE: client sent (methods == 68)
08/07/14 14:30:20 HANDSHAKE: i picked (method == 64)
08/07/14 14:30:20 HANDSHAKE: client received (method == 64)
08/07/14 14:30:20 AUTHENTICATE: method 64 (KERBEROS) failed.
08/07/14 14:30:20 HANDSHAKE: in handshake(my_methods = 'PASSWORD,KERBEROS,FS')
08/07/14 14:30:20 HANDSHAKE: handshake() - i am the server
08/07/14 14:30:20 HANDSHAKE: client sent (methods == 4)
08/07/14 14:30:20 HANDSHAKE: i picked (method == 4)
08/07/14 14:30:20 HANDSHAKE: client received (method == 4)
08/07/14 14:30:20 FS: client template is /tmp/FS_XXXXXXXXX
08/07/14 14:30:20 FS: client filename is /tmp/FS_XXXyiNlkB
08/07/14 14:30:20 AUTHENTICATE_FS: used dir /tmp/FS_XXXyiNlkB, status: 0
08/07/14 14:30:20 AUTHENTICATE: method 4 (FS) failed.
08/07/14 14:30:20 HANDSHAKE: in handshake(my_methods = 'PASSWORD,KERBEROS,FS')
08/07/14 14:30:20 HANDSHAKE: handshake() - i am the server
08/07/14 14:30:20 HANDSHAKE: client sent (methods == 0)
08/07/14 14:30:20 HANDSHAKE: i picked (method == 0)
08/07/14 14:30:20 HANDSHAKE: client received (method == 0)
08/07/14 14:30:20 DC_AUTHENTICATE: authentication of <10.129.5.186:51105> did not result in a valid mapped user name, which is required for this command (1112 QMGMT_WRITE_CMD), so aborting.
08/07/14 14:30:20 DC_AUTHENTICATE: reason for authentication failure: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using FS|FS:1004:Unable to lstat(/tmp/FS_XXXyiNlkB)|AUTHENTICATE:1004:Failed to authenticate using KERBEROS|AUTHENTICATE:1004:Failed to authenticate using PASSWORD