Skip to content

Instantly share code, notes, and snippets.

@EgorMatirov

EgorMatirov/Results of the "Pure-Go XRootD client implementation" GSoC project.md Secret

Last active August 11, 2018 14:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save EgorMatirov/303e4019606677d1838677a5a7145420 to your computer and use it in GitHub Desktop.
Save EgorMatirov/303e4019606677d1838677a5a7145420 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Results of the "Pure-Go XRootD client implementation" GSoC project.md

Proposal information

This project aims at implementing a pure-Go BSD-3 client library for XRootD using Go builtin features for concurrency and scalability. As part of that project I have worked on the XRootD client, server, and FUSE-based filesystem.

See full proposal for details.

Client

The client is implemented and supports most of the requests.

There are some specific requests that are not implemented yet: go-hep/hep#170, but some of them are fully defined by the current version of the XRootD protocol and are just placeholders.

The client needs to have some more polishing in terms of error and redirection handling:

  • What should happen if an initial connection gets aborted?
  • We need to re-issue open requests after the redirection since a file handle differs on the new server.
  • We need to change the initial session under some circumstances (see http://xrootd.org/doc/dev45/XRdv310.pdf, p. 11).
  • We need to have more tests with different kind of issues that can occur: a network connection error, a malformed answer from the server, etc.

Authentification is supported via unix protocol and Kerberos. To be done is:

  • GSI authentication - unimplemented due to lack of specification (only code is available that is GPL-licensed): issue.
  • OAuth-based authentication - see comment.

The client is already being used to implement 2 commands: xrd-cp and xrd-ls.

xrd-ls lists directory contents on a remote xrootd server:

$ /xrd-ls  root://eospublic.cern.ch:1094//eos/opendata/atlas/OutreachDatasets/2016-07-29/MC/
/eos/opendata/atlas/OutreachDatasets/2016-07-29/MC
/eos/opendata/atlas/OutreachDatasets/2016-07-29/MC/mc_105985.WW.root
/eos/opendata/atlas/OutreachDatasets/2016-07-29/MC/mc_105986.ZZ.root
...

xrd-cp copies files and directories from a remote xrootd server to local storage:

$ xrd-cp  root://eospublic.cern.ch:1094//eos/opendata/atlas/OutreachDatasets/2016-07-29/MC/mc_173045.DYtautauM08to15.root .
$ ls -alh  mc_173045.DYtautauM08to15.root
-rw-r--r-- 1 ematirov ematirov 1.4M Aug 11 18:02 mc_173045.DYtautauM08to15.root

The client supports both Linux and Windows (OS X has not been tested).

FUSE filesystem

The FUSE filesystem is implemented and lacks only hardlink and symlink syscalls since they are not supported by the XRootD protocol. Choice was made between bazil/fuse and hanwen/go-fuse libraries. hanwen/go-fuse was choosen because of more active development.

Also, tests are written to ensure functionality of the FS. There is one remaining bug, but I have had no success with debugging it.

Example of usage:

$ xrd-fuse root://eospublic.cern.ch:1094//eos/opendata/ /tmp/eospublic

Server

The server is in an early stage of work, but it supports some of the most important file operations: go-hep/hep#176. Major remaining milestones are:

  • Authentication.
  • Support of multiple socket connections (the bind request).
  • Proxying \ balancing queries to the other servers.

The server supports both Linux and Windows (OS X has not been tested).

Commits

Github search of all my commits in repository.

List of all commits:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment