Skip to content

Instantly share code, notes, and snippets.

@bwalsh
Created October 30, 2018 21:41
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bwalsh/1afea0a2499b5e81c507dbc24521038e to your computer and use it in GitHub Desktop.
Save bwalsh/1afea0a2499b5e81c507dbc24521038e to your computer and use it in GitHub Desktop.
minio s3 config
# cat ~/.aws/credentials
[default]
aws_access_key_id = XXXXXXXXX
aws_secret_access_key = XXXXXX
# cat ~/.aws/config
[default]
region = us-east-1
s3 =
signature_version = s3v4
dvc import --verbose s3://bmeg/source/allele source/allele
```
Debug: PRAGMA user_version;
Debug: fetched: [(2,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 2;
Importing 's3://bmeg/source/allele' -> '/src/source/allele'
Debug: Downloading 'bmeg/source/allele' to '/src/source/allele'
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dvc/remote/s3.py", line 324, in download
Key=from_info['key'])['ContentLength']
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Warning: Failed to download 'bmeg/source/allele': An error occurred (403) when calling the HeadObject operation: Forbidden
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(10,)]
Debug: UPDATE state_info SET count = 10 WHERE rowid = 1
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dvc/remote/s3.py", line 103, in get_etag
obj = self.s3.head_object(Bucket=bucket, Key=key)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dvc/command/imp.py", line 9, in run
self.project.imp(self.args.url, self.args.out)
File "/usr/local/lib/python3.7/site-packages/dvc/project.py", line 324, in imp
stage.run()
File "/usr/local/lib/python3.7/site-packages/dvc/stage.py", line 454, in run
self.save()
File "/usr/local/lib/python3.7/site-packages/dvc/stage.py", line 405, in save
dep.save()
File "/usr/local/lib/python3.7/site-packages/dvc/dependency/s3.py", line 32, in save
self.info = self.remote.save_info(self.path_info)
File "/usr/local/lib/python3.7/site-packages/dvc/remote/s3.py", line 114, in save_info
path_info['key'])}
File "/usr/local/lib/python3.7/site-packages/dvc/remote/s3.py", line 105, in get_etag
raise DvcException('s3://{}/{} does not exist'.format(bucket, key))
dvc.exceptions.DvcException: s3://bmeg/source/allele does not exist
Error: Failed to import s3://bmeg/source/allele: s3://bmeg/source/allele does not exist
Having any troubles? Hit us up at dvc.org/support, we are always happy to help!
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment