https://github.com/intel-cloud/cosbench/blob/master/COSBenchUserGuide.pdf has all installation and config information.
Installation on Ubuntu:
-
apt-get install openjdk-7-headless-jre
-
Download latest COSBench release zip from https://github.com/intel-cloud/cosbench/releases
-
Unzip and extract.
-
Make .sh files executable
find . -name '*.sh' -exec chmod +x {} ;
Architecture:
- There is a controller node to drive and aggregate results.
- There are driver nodes to run test driver processes. Each driver node can host multiple driver processes.
./start_driver.sh – Run this on each driver node. Can do it on master node too if master node is itself the lone driver node. Can specify "[num_drivers][IP address of all drivers][base port]" as optional args. Each driver then runs at port [base_port + index]. Default driver base port is 18088.
./start_controller.sh – only on master node. Its conf/controller.conf should have entries for each worker node in [driver<n>] sections. Default controller port is 19088.
Configuration files:
-
conf/controller.conf – specifies how many drivers, and URLs of all driver process endpoints. If driver nodes run multiple driver processes, this should have endpoints for every process.
-
conf/driver.conf – these are actually overwritten by start-driver.sh. So no point modifying these directly. Instead modify conf/driver_template.conf.
Actual testing procedure is configured using a workload configuration file in conf/.
-
Attached a working Ceph S3 workload configuration - cosbench-ceph-s3-workload.conf - based on the default S3 workload conf conf/s3-config-sample.xml
-
Note the <storage> tag attributes.
-
endpoint should be the RGW host:port. For actual AWS S3 testing, just remove the endpoint attribute and path_style_access can be removed too.
-
path_style_access=true is extremely important for Ceph S3; Otherwise, the Amazon S3 SDK library used by COSBench defaults to virtual host style access, tries to contact bucket URLs like http://s3testqwer1.cephmon1 and if RGW host is not configured for virtual host addressing, S3 client fails with “Unknown name or service s3testqwer1.cephmon1”
-
-
Another example S3 workload configuration: http://www.spinics.net/lists/ceph-users/msg28805.html . The multiple <storage> elements are not necessary – COSBench code seems to inherit global elements in each work stage or work element.
-
For actual AWS S3 endpoints:
- Bucket names should be globally unique across all S3 users. Since default name s3testqwer<n> is likely to be created by somebody else, COSBench fails with 403 access denied errors with default
Start a test: ./cli.sh submit conf/s3test.conf
Monitor a test: From a browser, open http://<cosbench-controller-node:19088/controller/. Then open the item under active workload, and drill down into workload, work stage and missions by clicking on “view details”.
Troubleshooting:
Log files and logging levels:
-
log/system.log – the controller`s log.Logging level is set by log_level in [controller] section of conf/controller.conf. Set to DEBUG|INFO log/mission/[mission-id].log – Actual worker logs. This is where any S3 client errors are recorded. Set “log_level” to DEBUG|INFO in [driver_n_ ] sections of conf/controller.conf Set “log_level” to DEBUG|INFO in [driver] section of conf/driver_template.conf.
-
"Error 403 Access denied for actual AWS S3 endpoint" - Check if the bucket has a globally unique name. The default s3testqwer<n> bucket names in workload configs are likely to be owned by somebody else already.
-
Error "[INFO] [NoneStorage] - performing PUT at /s3testqwer1 [WARN] [S3Storage] - below exception encountered when creating bucket s3testqwer1: Unable to execute HTTP request: s3testqwer1.cephmon1... [NoneStorage] - performing PUT at /s3testqwer2/myobjects10 [S3Storage] - below exception encountered when creating object myobjects10 at s3testqwer2: Unable to execute HTTP request: s3testqwer2.cephmon1: Name or service not known"
The error here is that S3 client is attempting to use virtual host style URLs http://bucket.host but Ceph RGW is not configured to handle it by default. Instead, tell S3 client to use path style URLs, by suffixing bucket and object names to URL. Set <storage ... config=“....path_style_access=true”> in workload config file.