HugoKuo

https://swiftstack.zendesk.com/agent/#/tickets/526

I'm keep tracing the issue here. 
There's some quite unusual log in your node01 and node02 about the /dev/xvdb which work as the swift storage disks on node.

Let's look at the log one-by-one:

swift-node01 (ID:1023)
dmesg
[ 2057.692048] XFS (xvdb): xfs_log_force: error 5 returned.

HugoKuo

 ** [out :: ssman.foo.bar] [2013-07-10T23:12:50-04:00] INFO: Processing execute[install-swiftstack-openvpn-policy] action run (openvpn::default line 236)
 ** [out :: ssman.foo.bar]
 ** [out :: ssman.foo.bar] ================================================================================
 ** [out :: ssman.foo.bar] Error executing action `run` on resource 'execute[install-swiftstack-openvpn-policy]'
 ** [out :: ssman.foo.bar] ================================================================================
 ** [out :: ssman.foo.bar]
 ** [out :: ssman.foo.bar] Mixlib::ShellOut::ShellCommandFailed
 ** [out :: ssman.foo.bar] ------------------------------------
 ** [out :: ssman.foo.bar] Expected process to exit with [0], but received ''

HugoKuo

Successfully installed Cython gevent pyzmq greenlet
Cleaning up...
root@ss-testserver:~/bm# pip install ssbench
Downloading/unpacking ssbench
  Downloading ssbench-0.2.22.tar.gz (83Kb): 83Kb downloaded
  Running setup.py egg_info for package ssbench

    warning: no files found matching '.requirements.txt'
Requirement already satisfied (use --upgrade to upgrade): Cython in /usr/local/lib/python2.7/dist-packages (from ssbench)
Requirement already satisfied (use --upgrade to upgrade): gevent in /usr/local/lib/python2.7/dist-packages (from ssbench)

HugoKuo

root@ny-workshop:/home/swiftstack# ./create-accounts.py
Create ws000
1393219098.576729 [30110] INFO base.py:188 - Raven is not configured (logging is disabled). Please see the documentation for more information.
Traceback (most recent call last):
  File "./create-accounts.py", line 26, in <module>
    response = client.post(reverse('signup'), formdata)
  File "/usr/lib/python2.7/dist-packages/django/test/client.py", line 463, in post
    response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
  File "/usr/lib/python2.7/dist-packages/django/test/client.py", line 297, in post
    return self.request(**r)

HugoKuo

===== ERROR Msgs =====

Mar 24 01:54:28 localhost container-sync: ERROR Syncing /srv/node/d5/containers/253/414/3f469a092b89f7421aa7740e166ca414/3f469a092b89f7421aa7740e166ca414.db {'name': 'obj2', 'deleted': 0, 'created_at': '1395650480.62009', 'etag': '1f18348f32c9a4694f16426798937ae2', 'content_type': 'application/octet-stream', 'ROWID': 1, 'size': 5}: 

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/swift/container/sync.py", line 421, in container_sync_row
    proxy=self.select_http_proxy())
  File "/usr/lib/pymodules/python2.7/swift/common/internal_client.py", line 778, in put_object
    client.retry_request('PUT', **kwargs)
  File "/usr/lib/pymodules/python2.7/swift/common/internal_client.py", line 750, in retry_request

HugoKuo

ssman=> select id,state,args_json,state_data from app_workjob where id=48854;
  id   |      state       |      args_json       |                                                state_data
-------+------------------+----------------------+-----------------------------------------------------------------------------------------------------------
 48854 | COMPLETED_FAILED | {"cluster_id": 2284} | 1396250962.091366 INFO Pushing new versions                                                              +
       |                  |                      | 1396250962.291359 ERROR push_config failed                                                               +
       |                  |                      | Traceback (most recent call last):                                                                       +
       |                  |                      |   File "/deploy/ssman/current/ssman/app/tasks.py", line 96, in push_config                               +
       |                  |                    

HugoKuo

swiftstack@swiftstack-paco:~/source$ python s3-boto.py
1	2009-02-03T16:45:09.000Z

swiftstack@swiftstack-paco:~/source$ cat s3-boto.py
import boto
import boto.s3.connection
access_key = 'ss'                                          #SwiftStack_AUTH user
secret_key = '5d1efb88b61231c1e4399e00caf68f7b'               #s3key of the user

conn = boto.connect_s3(

HugoKuo

swiftstack@swiftstack-paco:~/data/test$ ls
swiftstack@swiftstack-paco:~/data/test$ mkdir dir1
swiftstack@swiftstack-paco:~/data/test$ mkdir dir2
swiftstack@swiftstack-paco:~/data/test$ mkdir dir3
swiftstack@swiftstack-paco:~/data/test$ mkdir dir-empty

swiftstack@swiftstack-paco:~/data/test$ ls
dir1  dir2  dir3  dir-empty

swiftstack@swiftstack-paco:~/data/test$ ls *

HugoKuo

username=support
api_key=xxxxxxx
authurl=https://cloud.swiftstack.com/auth/v1.0

#authurl=http://swiftstack.openstack.com.tw/auth/v1.0
#tenant=[Tenant for authentication with Keystone]
#password=password
#use_snet=[True to use Rackspace ServiceNet for connections]
#cache_timeout=[Seconds for directory caching, default 600]
verify_ssl=True

HugoKuo

tcgetattr: Inappropriate ioctl for device
 Active / Total Objects (% used)    : 52760637 / 53149905 (99.3%)
 Active / Total Slabs (% used)      : 1268292 / 1268292 (100.0%)
 Active / Total Caches (% used)     : 73 / 115 (63.5%)
 Active / Total Size (% used)       : 18809079.12K / 18886425.08K (99.6%)
 Minimum / Average / Maximum Object : 0.01K / 0.35K / 15.30K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
13511944 13511944 100%    0.88K 390402       36  12492864K xfs_inode
11930877 11844764  99%    0.19K 284079       42   2272632K dentry