cloudstore dux

gistfile1.txt

~/P/R/hadoop-3.3.1 time bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose $BUCKET


Listing files under s3a://stevel-london/ with thread count 64
=============================================================

2021-06-03 20:10:37,994 [main] INFO  commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: List files under s3a://stevel-london/
2021-06-03 20:10:39,932 [main] INFO  commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: Initial list of path s3a://stevel-london/
2021-06-03 20:10:40,288 [main] INFO  commands.ExtendedDu (StoreDurationInfo.java:close(100)) - Initial list of path s3a://stevel-london/: duration 0:00:356
2021-06-03 20:10:40,290 [pool-3-thread-1] INFO  commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: List s3a://stevel-london/cloud-integration
Waiting for 3 scan to finish
2021-06-03 20:10:40,290 [pool-3-thread-2] INFO  commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: List s3a://stevel-london/user
2021-06-03 20:10:40,290 [pool-3-thread-3] INFO  commands.ExtendedDu (StoreDurationInfo.java:<init>(53)) - Starting: List s3a://stevel-london/tmp
2021-06-03 20:10:40,560 [pool-3-thread-3] INFO  commands.ExtendedDu (StoreDurationInfo.java:close(100)) - List s3a://stevel-london/tmp: duration 0:00:270
List iterator: FunctionRemoteIterator{FileStatusListingIterator[Object listing iterator against s3a://stevel-london/tmp; listing count 1; isTruncated=false; counters=((object_list_request=1) (object_continue_list_request=0) (object_list_request.failures=0) (object_continue_list_request.failures=0));
gauges=();
minimums=((object_continue_list_request.min=-1) (object_continue_list_request.failures.min=-1) (object_list_request.min=36) (object_list_request.failures.min=-1));
maximums=((object_continue_list_request.max=-1) (object_continue_list_request.failures.max=-1) (object_list_request.max=36) (object_list_request.failures.max=-1));
means=((object_list_request.mean=(samples=1, sum=36, mean=36.0000)) (object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)));
]}
2021-06-03 20:10:41,656 [pool-3-thread-2] INFO  commands.ExtendedDu (StoreDurationInfo.java:close(100)) - List s3a://stevel-london/user: duration 0:01:366
List iterator: FunctionRemoteIterator{FileStatusListingIterator[Object listing iterator against s3a://stevel-london/user; listing count 1; isTruncated=false; counters=((object_continue_list_request.failures=0) (object_list_request=1) (object_continue_list_request=0) (object_list_request.failures=0));
gauges=();
minimums=((object_list_request.failures.min=-1) (object_continue_list_request.min=-1) (object_list_request.min=38) (object_continue_list_request.failures.min=-1));
maximums=((object_list_request.failures.max=-1) (object_continue_list_request.max=-1) (object_continue_list_request.failures.max=-1) (object_list_request.max=38));
means=((object_list_request.mean=(samples=1, sum=38, mean=38.0000)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)));
]}
2021-06-03 20:10:42,878 [pool-3-thread-1] INFO  commands.ExtendedDu (StoreDurationInfo.java:close(100)) - List s3a://stevel-london/cloud-integration: duration 0:02:588
List iterator: FunctionRemoteIterator{FileStatusListingIterator[Object listing iterator against s3a://stevel-london/cloud-integration; listing count 1; isTruncated=false; counters=((object_continue_list_request=0) (object_list_request.failures=0) (object_continue_list_request.failures=0) (object_list_request=1));
gauges=();
minimums=((object_list_request.failures.min=-1) (object_continue_list_request.failures.min=-1) (object_list_request.min=64) (object_continue_list_request.min=-1));
maximums=((object_continue_list_request.failures.max=-1) (object_list_request.max=64) (object_continue_list_request.max=-1) (object_list_request.failures.max=-1));
means=((object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.mean=(samples=1, sum=64, mean=64.0000)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)));
]}
2021-06-03 20:10:42,879 [main] INFO  commands.ExtendedDu (StoreDurationInfo.java:close(100)) - List files under s3a://stevel-london/: duration 0:04:886
FileSystem: S3AFileSystem{uri=s3a://stevel-london, workingDir=s3a://stevel-london/user/stevel, inputPolicy=normal, partSize=8388608, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=524288, blockSize=33554432, multiPartThreshold=134217728, serverSideEncryptionAlgorithm='NONE', blockFactory=ByteBufferBlockFactory{buffersOutstanding=0}, metastore=DynamoDBMetadataStore{region=eu-west-2, tableName=stevel-london, tableArn=arn:aws:dynamodb:eu-west-2:152813717728:table/stevel-london}, authoritativeStore=false, authoritativePath=[], useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=192, available=192, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@4a1e3ac1[Running, pool size = 4, active threads = 0, queued tasks = 0, completed tasks = 4], credentials=AWSCredentialProviderList[refcount= 2: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@6e78fcf5] last provider: SimpleAWSCredentialsProvider, delegation tokens=S3ADelegationTokens{canonicalServiceURI=s3a://stevel-london; owner=stevel; isBoundToDT=false; token creation count=0; tokenManager=Service SessionTokens/001 in state SessionTokens/001: STARTED token kind = S3ADelegationToken/Session; token=(none)}, DirectoryMarkerRetention{policy='delete'}, instrumentation {S3AInstrumentation{}}}

Storage Statistics
==================

object_metadata_request	0
object_copy_requests	0
stream_read_seek_operations	0
stream_read_version_mismatches	0
delegation_tokens_issued.failures	0
op_exists	0
stream_read_seek_bytes_discarded	0
object_put_request_active	0
s3guard_metadatastore_authoritative_directories_updated	0
op_abort.failures	0
action_executor_acquired	0
stream_read_total_bytes	0
object_select_requests	0
stream_write_block_uploads_committed	0
stream_read_closed	0
committer_tasks_failed	0
action_executor_acquired.failures	0
committer_stage_file_upload.failures	0
committer_magic_files_created	0
files_copied	0
committer_materialize_file	0
stream_write_block_uploads	0
op_xattr_get_named	0
directories_created	0
action_http_get_request.failures	0
multipart_instantiated	0
stream_read_seek_backward_operations	0
op_list_located_status	0
stream_aborted	0
stream_write_bytes	0
op_get_file_status	2
op_get_delegation_token	0
store_io_request	4
object_put_bytes_pending	0
action_http_head_request.failures	0
op_list_files	3
object_multipart_initiated	0
op_copy_from_local_file	0
stream_write_queue_duration	0
committer_commit_job.failures	0
s3guard_metadatastore_record_writes	0
object_list_request	4
s3guard_metadatastore_initialization	1
committer_bytes_uploaded	0
committer_materialize_file.failures	0
s3guard_metadatastore_retry	0
stream_write_total_data	0
fake_directories_created	0
op_xattr_get_map	0
committer_jobs_failed	0
committer_commits_aborted	0
object_list_request.failures	0
stream_read_bytes_discarded_in_abort	0
s3guard_metadatastore_put_path_request	0
object_put_request	0
committer_tasks_completed	0
object_continue_list_request.failures	0
action_http_get_request	0
stream_read_operations_incomplete	0
committer_commits.failures	0
op_xattr_get_named_map.failures	0
multipart_upload_part_put	0
files_created	0
stream_write_block_uploads_active	0
ignored_errors	0
multipart_upload_part_put_bytes	0
stream_read_operations	0
stream_write_block_uploads_aborted	0
op_glob_status	0
s3guard_metadatastore_record_deletes	0
delegation_tokens_issued	0
object_multipart_aborted	0
op_xattr_list	0
op_delete	0
object_put_bytes	0
s3guard_metadatastore_throttled	0
object_delete_request	0
store_io_throttled	0
committer_jobs_completed	0
action_http_head_request	0
stream_write_total_time	0
stream_read_bytes_discarded_in_close	0
op_is_directory	0
stream_read_seek_policy_changed	0
stream_write_queue_duration.failures	0
committer_bytes_committed	0
op_abort	0
committer_commits_reverted	0
op_open	0
stream_write_exceptions_completing_upload	0
fake_directories_deleted	0
multipart_upload_completed	0
stream_read_fully_operations	0
op_list_status	1
op_create_non_recursive	0
files_deleted	0
stream_read_close_operations	0
stream_write_block_uploads_data_pending	0
multipart_upload_started	0
committer_commit_job	0
object_bulk_delete_request.failures	0
stream_read_seek_bytes_skipped	0
op_xattr_get_named.failures	0
multipart_upload_abort_under_path_invoked	0
object_bulk_delete_request	0
object_delete_request.failures	0
stream_read_opened	0
stream_read_exceptions	0
op_rename	0
op_xattr_get_map.failures	0
directories_deleted	0
op_create	0
op_hsync	0
object_continue_list_request	0
files_delete_rejected	0
stream_read_bytes	0
committer_commits_created	0
store_io_retry	0
committer_commits_completed	0
s3guard_metadatastore_record_reads	110
op_hflush	0
object_put_request_completed	0
op_get_file_checksum	0
op_xattr_list.failures	0
stream_read_seek_forward_operations	0
committer_stage_file_upload	0
stream_write_block_uploads_pending	0
op_is_file	0
files_copied_bytes	0
op_mkdirs	0
multipart_upload_aborted	0
object_delete_objects	0
stream_write_exceptions	0
stream_read_bytes_backwards_on_seek	0
op_xattr_get_named_map	0


path    files   size
====================

s3a://stevel-london/cloud-integration	52	131 MB
s3a://stevel-london/tmp	0	0 bytes
s3a://stevel-london/user	0	0 bytes

Found 52 files, 93.96 milliseconds per file
Data size 131 MB (138,335,743 bytes)
Average file size 2 MB (2,660,302 bytes)

________________________________________________________
Executed in    5.84 secs    fish           external
   usr time    6.77 secs  115.00 micros    6.77 secs
   sys time    0.64 secs  969.00 micros    0.64 secs