Both the schema below have the following properties
- Crash-consistent during a completeMultipartUpload/abortMultipartUpload
- Concurrency-safe in presence of multiple concurrent uploads
These properties allows us to avoid fcntl(3) based locking in shared mode with FS backend.
.minio.sys.tmp
├── <uploadId> -----------------> created on initMultipartUpload
├── <eTag>.0 ---------------> created on initMultipartUpload
├── <eTag>.<partNumber> ----> created on putObjectPart
where,
<uploadId>/<eTag>.0
contains the following json object
{
"Bucket": "bucketName",
"Object": "objectName"
}
The following example contains 3 parts each of 2 concurrent uploads with uploadId
6e463bb8-35bd-4408-809e-78f509f558b3
7bed54f8-ad71-48b1-a4f8-bd2e2a8efbda
to mybucket/myobject
.
.minio.sys.tmp
├── 6e463bb8-35bd-4408-809e-78f509f558b3
│ ├── 467886be95c8ecfd71a2900e3f461b4f.0
│ ├── 467886be95c8ecfd71a2900e3f461b4f.1
│ ├── 467886be95c8ecfd71a2900e3f461b4f.2
│ └── 467886be95c8ecfd71a2900e3f461b4f.3
└── 7bed54f8-ad71-48b1-a4f8-bd2e2a8efbda
├── 467886be95c8ecfd71a2900e3f461b4f.0
├── 467886be95c8ecfd71a2900e3f461b4f.1
├── 467886be95c8ecfd71a2900e3f461b4f.2
└── 467886be95c8ecfd71a2900e3f461b4f.3
6e463bb8-35bd-4408-809e-78f509f558b3/467886be95c8ecfd71a2900e3f461b4f.0
would contain
{
"Bucket": "mybucket",
"Object": "myobject"
}
- Easier to support/debug since uploadId directory will have at most 10000 entries
- uploadId modTime will reflect last updated part for a given upload; simplifies detection of 'stale' uploads
- Limited number of entries per uploadId directory; listing during cleanup of stale upload parts will be faster
.minio.sys.tmp
├── <uploadId>.<eTag>.0 ---------------> created on initMultipartUpload
├── <uploadId>.<eTag>.<partNumber> ----> created on putObjectPart
where,
<uploadId>.<eTag>.0
contains the following json object
{
"Bucket": "bucketName",
"Object": "objectName"
}
The following example contains 3 parts each of 2 concurrent uploads with uploadId
6e463bb8-35bd-4408-809e-78f509f558b3
7bed54f8-ad71-48b1-a4f8-bd2e2a8efbda
to mybucket/myobject
.
.minio.sys.tmp
├── 6e463bb8-35bd-4408-809e-78f509f558b3.467886be95c8ecfd71a2900e3f461b4f.0
├── 6e463bb8-35bd-4408-809e-78f509f558b3.467886be95c8ecfd71a2900e3f461b4f.1
├── 6e463bb8-35bd-4408-809e-78f509f558b3.467886be95c8ecfd71a2900e3f461b4f.2
├── 6e463bb8-35bd-4408-809e-78f509f558b3.467886be95c8ecfd71a2900e3f461b4f.3
├── 7bed54f8-ad71-48b1-a4f8-bd2e2a8efbda.467886be95c8ecfd71a2900e3f461b4f.0
├── 7bed54f8-ad71-48b1-a4f8-bd2e2a8efbda.467886be95c8ecfd71a2900e3f461b4f.1
├── 7bed54f8-ad71-48b1-a4f8-bd2e2a8efbda.467886be95c8ecfd71a2900e3f461b4f.2
└── 7bed54f8-ad71-48b1-a4f8-bd2e2a8efbda.467886be95c8ecfd71a2900e3f461b4f.3
6e463bb8-35bd-4408-809e-78f509f558b3.467886be95c8ecfd71a2900e3f461b4f.0
contains
{
"Bucket": "bucketName",
"Object": "objectName"
}
- Single directory to hold all ongoing uploads including single PUT object
- Maximum object name supportable is limited by platform-specific path segment length limits. In GNU/Linux it's 255.
@balamurugana,
Yes, I shall add this in Cons for Schema-2.
This document doesn't describe how metadata (currently stored in fs.json) will be handled. If extended attributes (EA) are supported by the disk filesystem, then we can save them in EAs else user-metadata is not supported in such a setup. In both schema we can save user-metadata as EA on the first part until the time of completeMultipartUpload.