Skip to content

Instantly share code, notes, and snippets.

@alistairncoles

alistairncoles/gist:a255f6b5d8f03c77e2d0 Secret

Created March 25, 2016 12:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save alistairncoles/a255f6b5d8f03c77e2d0 to your computer and use it in GitHub Desktop.
Save alistairncoles/a255f6b5d8f03c77e2d0 to your computer and use it in GitHub Desktop.
Download ZIP
Running functional tests against a partially upgraded swift cluster
Raw
gistfile1.txt
Running functional tests against a partially upgraded swift cluster
===================================================================
Methodology:
------------
Run different swift services on different versions in separate virtualenvs.
Note: same config in /etc/swift used by all service regardless of version.
Setup:
------
Copy swift repo to two separate dirs, e.g. swift1 and swift2
In each dir:
% git checkout <version>
% tox -r -e venv -- python
>> import swift; swift.__version__ (to double-check version in use)
% source .tox/venv/bin/activate
% swift-init restart object-server container-server proxy-server
or
% swift-init restart proxy-server
In a separate dir:
% git checkout <version>
% nosetests ./test/functional
From proxy.log:
Mar 25 12:27:58 anc-vm-10 proxy-server: Pipeline is "gatekeeper catch_errors healthcheck proxy-logging cache tempurl formpost ratelimit list-endpoints container_sync tempauth staticweb container-quotas account-quotas dlo slo versioned_writes proxy-logging proxy-server"
(i.e. using tempauth, versioned_writes explicitly configured, replication default policy)
Results
-------
liberty = origin/stable/liberty = commit 8c1976aa = 2.5.1.dev2
mitaka = origin/stable/mitaka = commit e0bac5e = 2.7.0
proxy-server a/c/o server functional tests
============ ============ ================
mitaka mitaka mitaka Ran 422 tests OK (SKIP=21)
liberty mitaka mitaka Ran 422 tests FAILED (SKIP=21, errors=8, failures=4) [1]
mitaka liberty mitaka Ran 422 tests FAILED (SKIP=21, failures=24) [1]
liberty mitaka liberty Ran 380 tests OK (SKIP=17)
mitaka liberty liberty Ran 380 tests FAILED (SKIP=17, failures=5) [2]
[1] failures with mitaka test suite
-----------------------------------
Expect newer func test suite to have some failures when run against older versions, and
note that number of passing tests is at least greater with mitaka test suite than when running
the liberty test suite.
[2] proxy @ mitaka, a/c/o @ liberty - analysis of failed tests...
-------------------------------------------------------------------
# The first two are due to commit 0e5cc89da56f550468512af6f3479382a1894815
# https://review.openstack.org/#/c/234391 "Have versioned_writes use reverse listings"
# which changes middleware to use the reverse param when requesting list of versions,
# but the param is not supported by a liberty container server, so on delete the versioned
# object is replaced with the *oldest* version :/
======================================================================
FAIL: test_overwriting (test.functional.tests.TestObjectVersioning)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/swift/swift/test/functional/tests.py", line 2932, in test_overwriting
self.assertEqual("ccccc", versioned_obj.read())
AssertionError: 'ccccc' != 'aaaaa'
"'ccccc' != 'aaaaa'" = '%s != %s' % (safe_repr('ccccc'), safe_repr('aaaaa'))
"'ccccc' != 'aaaaa'" = self._formatMessage("'ccccc' != 'aaaaa'", "'ccccc' != 'aaaaa'")
>> raise self.failureException("'ccccc' != 'aaaaa'")
======================================================================
FAIL: test_overwriting (test.functional.tests.TestObjectVersioningUTF8)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/swift/swift/test/functional/tests.py", line 2932, in test_overwriting
self.assertEqual("ccccc", versioned_obj.read())
AssertionError: 'ccccc' != 'aaaaa'
"'ccccc' != 'aaaaa'" = '%s != %s' % (safe_repr('ccccc'), safe_repr('aaaaa'))
"'ccccc' != 'aaaaa'" = self._formatMessage("'ccccc' != 'aaaaa'", "'ccccc' != 'aaaaa'")
>> raise self.failureException("'ccccc' != 'aaaaa'")
# This next failure is most likely due to test coupling, with the previous failed test
# leaving unexpected objects in the versions container due to lack of cleanup in the teardown
# method. This cleanup (or lack of) got fixed in commit 87fc21c7cfcce2a3e23a84fddcfd1309cd884716
# "Speed up functional testing". Note that only TestObjectVersioningUTF8 suffered from lack of
# cleanup (due to multiple inheritance precedence) hence test_versioning_dlo only fails in that
# class. The coupling can be seen quickly by comparing:
% nosetests ./test/functional/tests.py -m ing
% nosetests ./test/functional/tests.py -m test_versioning_dlo
======================================================================
FAIL: test_versioning_dlo (test.functional.tests.TestObjectVersioningUTF8)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/swift/swift/test/functional/tests.py", line 2953, in test_versioning_dlo
self.assertEqual(3, versions_container.info()['object_count'])
AssertionError: 3 != 5
'3 != 5' = '%s != %s' % (safe_repr(3), safe_repr(5))
'3 != 5' = self._formatMessage('3 != 5', '3 != 5')
>> raise self.failureException('3 != 5')
# The next two are due to commit c0866ce which changed the response code for
# self-referencing SLO manifest to be 400:
======================================================================
FAIL: test_slo_overwrite_segment_with_manifest (test.functional.tests.TestSlo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/swift/swift/test/functional/tests.py", line 2457, in test_slo_overwrite_segment_with_manifest
self.assertEqual(409, err.status)
AssertionError: 409 != 400
'409 != 400' = '%s != %s' % (safe_repr(409), safe_repr(400))
'409 != 400' = self._formatMessage('409 != 400', '409 != 400')
>> raise self.failureException('409 != 400')
======================================================================
FAIL: test_slo_overwrite_segment_with_manifest (test.functional.tests.TestSloUTF8)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/swift/swift/test/functional/tests.py", line 2457, in test_slo_overwrite_segment_with_manifest
self.assertEqual(409, err.status)
AssertionError: 409 != 400
'409 != 400' = '%s != %s' % (safe_repr(409), safe_repr(400))
'409 != 400' = self._formatMessage('409 != 400', '409 != 400')
>> raise self.failureException('409 != 400')
----------------------------------------------------------------------
Ran 380 tests in 259.564s
FAILED (SKIP=17, failures=5)
@alistairncoles
Copy link
Author

Update: applied commit 9cfe9e12d240c48659c3af14dda9f6cd93273333 from https://review.openstack.org/#/c/299686/2 to proxy-server, ran functional tests from liberty branch, and the three failing tests from TestObjectVersioning[UTF8] now pass ok.

Still two failing tests but these are explained above by change in expected response code.

======================================================================
FAIL: test_slo_overwrite_segment_with_manifest (test.functional.tests.TestSlo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/swift/swift/test/functional/tests.py", line 2457, in test_slo_overwrite_segment_with_manifest
    self.assertEqual(409, err.status)
AssertionError: 409 != 400
    '409 != 400' = '%s != %s' % (safe_repr(409), safe_repr(400))
    '409 != 400' = self._formatMessage('409 != 400', '409 != 400')
>>  raise self.failureException('409 != 400')


======================================================================
FAIL: test_slo_overwrite_segment_with_manifest (test.functional.tests.TestSloUTF8)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/swift/swift/test/functional/tests.py", line 2457, in test_slo_overwrite_segment_with_manifest
    self.assertEqual(409, err.status)
AssertionError: 409 != 400
    '409 != 400' = '%s != %s' % (safe_repr(409), safe_repr(400))
    '409 != 400' = self._formatMessage('409 != 400', '409 != 400')
>>  raise self.failureException('409 != 400')


----------------------------------------------------------------------
Ran 380 tests in 202.318s

FAILED (SKIP=17, failures=2)

@alistairncoles
Copy link
Author

I checked the multi-page listing iter for patch 299686 by starting 2 container servers running liberty in a first virtual envs and then 2 container servers running HEAD in another venv. I had container listing limit set to 10 in swift.conf. Then used this func test patch (which spews cleanup errors but does pass the assertions). The idea being to have the first page of version objects expire before starting deletes, which forces a multi-page listing. Then inspecting the logs shows that some listings are a mix of reverse and non-reversed responses.

diff --git a/swift/common/middleware/versioned_writes.py b/swift/common/middleware/versioned_writes.py
index 51497c7..a94fca4 100644
--- a/swift/common/middleware/versioned_writes.py
+++ b/swift/common/middleware/versioned_writes.py
@@ -174,11 +174,13 @@ class VersionedWritesContext(WSGIContext):
         '''
         complete_listing = []
         if not failed_marker:
+            self.logger.info('ANC _in_proxy_reverse_listing RE-USE')
             # We've never gotten a reversed listing. So save a request and
             # use the failed listing.
             complete_listing.extend(failed_listing)
             marker = complete_listing[-1]['name'].encode('utf8')
         else:
+            self.logger.info('ANC _in_proxy_reverse_listing RE-START')
             # We've gotten at least one reversed listing. Have to start at
             # the beginning.
             marker = ''
@@ -245,11 +247,13 @@ class VersionedWritesContext(WSGIContext):
             last_item = sublisting[-1]['name'].encode('utf-8')
             page_is_after_marker = marker and first_item > marker
             if reverse and (first_item < last_item or page_is_after_marker):
+                self.logger.info('ANC _listing_pages_iter NOT REVERSED')
                 # Apparently there's at least one pre-2.6.0 container server
                 yield self._in_proxy_reverse_listing(
                     account_name, lcontainer, lprefix,
                     env, marker, sublisting)
                 return
+            self.logger.info('ANC _listing_pages_iter REVERSED')

             marker = last_item
             yield sublisting
diff --git a/test/functional/tests.py b/test/functional/tests.py
index d51156e..0bf6786 100644
--- a/test/functional/tests.py
+++ b/test/functional/tests.py
@@ -3542,6 +3542,46 @@ class TestObjectVersioning(Base):
         self.assertEqual(self.env.container.info()['versions'],
                          self.env.versions_container.name)

+    def test_versioning(self):
+        container = self.env.container
+        versions_container = self.env.versions_container
+        cont_info = container.info()
+        self.assertEqual(cont_info['versions'], versions_container.name)
+
+        obj_name = Utils.create_name()
+        obj_count = 30
+        for v in range(obj_count/2):
+            versioned_obj = container.file(obj_name)
+            versioned_obj.write(str(v), hdrs={'Content-Type': 'text/plain'})
+
+        for v in range(obj_count/2, obj_count):
+            versioned_obj = container.file(obj_name)
+            versioned_obj.write(str(v), hdrs={'Content-Type': 'text/plain',
+                                              'X-Delete-After': '5'})
+        versioned_obj = container.file(obj_name)
+        versioned_obj.write(str(obj_count), hdrs={'Content-Type': 'text/plain'})
+        print 'Sleeping...'
+        time.sleep(8)
+
+        self.assertEqual(str(obj_count), versioned_obj.read())
+        versioned_obj.delete()
+        for v in range(obj_count/2 - 1):
+            expect = str(obj_count/2 - 1 - v)
+            print 'Deleting %s' % expect
+            self.assertEqual(expect, versioned_obj.read())
+            versioned_obj.delete()
+        print 'OK, Cleaning up...'
+        # cleanup
+        for name in versions_container.files():
+            print 'Deleting version obj %s' % name
+            obj = versions_container.file(name)
+            try:
+                obj.delete()
+            except Exception:
+                pass
+
+
+
     def test_overwriting(self):
         container = self.env.container
         versions_container = self.env.versions_container

Sample logging:

swift@anc-vm-10:~/swift$ tail -f /var/log/swift/proxy.log|grep ANC
Mar 31 18:38:45 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter NOT REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _in_proxy_reverse_listing RE-USE
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:46 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter NOT REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _in_proxy_reverse_listing RE-START
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter NOT REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _in_proxy_reverse_listing RE-USE
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter NOT REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _in_proxy_reverse_listing RE-START
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:47 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:48 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:48 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:48 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:48 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:48 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:48 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:48 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter NOT REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _in_proxy_reverse_listing RE-START
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter NOT REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _in_proxy_reverse_listing RE-USE
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter NOT REVERSED
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _in_proxy_reverse_listing RE-USE
Mar 31 18:38:49 anc-vm-10 proxy-server: ANC _listing_pages_iter REVERSED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment