EC update, March 18

gistfile1.txt

Dates are coming up very quickly, which is both exciting and
terrifying. By the end of next week, we'll be wrapping up the
feature/ec development work that will be in the beta. Then we'll have
about 2 weeks to do the merge to master.

Areas of coverage that are currently broken in functional tests for
EC: range requests, conditional requests (If-*Match), and some copy
requests. Ideally, we'll have completely passing functional tests for
a cluster with an EC default storage policy in the beta release.

Tracking on master:
(1) https://review.openstack.org/#/c/156825/
Landing this patch does a few good things. First, it makes the huge
PUT method better. But more than that, it's also on the feature/ec
branch (and thus will land with the feature/ec branch too). That means
that landing it on master now makes the feature/ec merge better and
shows more clearly what existing functionality in the proxy is touched
and what is not touched by feature/ec.

There are several patch chains on the feature/ec branch that need to be addressed.

(2) https://review.openstack.org/#/c/164380/
This patch chain has refactorings to clearly separate off the EC code
from the replication code in the object server. The reconstructor will
be rebased onto the end of this patch chain.

(3) https://review.openstack.org/164561
This is the same refactoring that is proposed to master, but the
others in the chain have to do with separating the EC code from the
replication code in the proxy server.

(4) https://review.openstack.org/163620
Sam has been working on getting range requests to work for EC code.
This patch is for single range requests. He's currently working on a
patch for multi-range requests (nearly done). This will bring range
functionality for EC to parity with replicated data.

(5) https://review.openstack.org/#/c/163209
This is to allow things like the account reaper and container sync to
use internal client instead of directly calling storage nodes (what's
on a storage node isn't an "object" in an EC context). These patches
are necessary for a seamless upgrade to using EC. Without it,
deployers will have the requirement of creating and deploying a new
config file/section before doing the post-upgrade restart of
processes.

(6) https://review.openstack.org/159205
Right now, the only way to do functional tests against EC is to set
the default storage policy to EC. Alistair's patch here enables
setting the in-process functional tests to run against a particular
storage policy.

These are all referenced on the priority reviews page and starred in
the review dashboard. These are listed in priority-order above.

With these four patch chains landed on feature/ec, we'll be in very
good shape to have a beta in Kilo. Please jump in and help out
wherever you can.

Here are the dates we're looking at:
April 30: OpenStack Kilo release -- this is a hard deadline
April 10: RC1 for Swift
March 27: freeze master for other development and begin the merge of
feature/ec into master

The RC and merge dates are not set in stone, but there isn't a ton of
wiggle room there. We'll need to be very close to those dates to
deliver on our public commitment. We've got a lot to do, and not a lot
of time. Thank you for working hard on this and helping out.

Here's a list of stuff that are known to not be in the EC beta:
* small file optimization - there are a lot of good ideas here, and
  we'll get to it later
* only run reconstructors if there is an EC policy (e.g. with
  `swift-init all start`)

This list will almost certainly grow, e.g. with stuff like If-*Match
support or cross-policy copy support or account reaper.

Please stay active in #openstack-swift so questions are answered
quickly and we can keep momentum up.