Skip to content

Instantly share code, notes, and snippets.

@rgregg
Last active May 26, 2022 09:17
Show Gist options
  • Star 20 You must be signed in to star a gist
  • Fork 8 You must be signed in to fork a gist
  • Save rgregg/37ba8929768a62131e85 to your computer and use it in GitHub Desktop.
Save rgregg/37ba8929768a62131e85 to your computer and use it in GitHub Desktop.
OneDrive Large File Upload API

Uploading Large Files

NOTICE: The OneDrive API now has native supported for large file uploads. Instead of using BITS, consider using the OneDrive API upload method instead!

Note This is preliminary documentation and is subject to change as we evolve the OneDrive API.

OneDrive handles uploading large files by supporting the BITS protocol. BITS is a simple extension to HTTP that enables resumable file uploads to OneDrive.

Where the standard PUT method for uploading a file has a 100 MB limit, using the BITS protocol will allow you to upload much larger files. BITS also provides connection tolerance by splitting the file into separate fragments which are uploaded sequentially and reassembled into the original file on the service. If a fragment transfer is interrupted, only that fragment needs to be uploaded again, previously uploaded fragments are preserved on the server.

OneDrive supports the standard BITS protocol for this scenario.

Note: Unlike the Live SDK, there is no logic to resize / resample photos that runs when photos are uploaded with BITS.

To upload a large file, a particular call pattern is required that looks like this:

  1. Request an Upload Session
  2. Upload File Fragments
  3. Close the Upload Session

Request an Upload Session

To begin, your app makes a request to OneDrive to create a new upload session.

Request

You will need to use the user's ID, a unique identifier for OneDrive, to build the URL. You can retrieve the user's ID using the Live SDK by making a request for the user's profile information. The value returned in the id field is the user's ID.

Once you have the user's ID, you can build the request to create an upload session. In this example, we're uploading the file Terwiliger.jpg into a folder named Portland in the root of the user's OneDrive.

POST https://storage.live.com/users/0x{id}/LiveFolders/Portland/Terwiliger.jpg HTTP/1.1
X-Http-Method-Override: BITS_POST
Authorization: Bearer {access-token}
BITS-Packet-Type: Create-Session
BITS-Supported-Protocols: {7df0354d-249b-430f-820d-3d2a9bef4931}

Note You can upload files based on the folder's resource ID or the path of the folder. To upload using the folder resource ID, use a URL format like this:

https://storage.live.com/items/{folder-id}/newfilename.jpg

To upload using path, the URL format should look like this:

https://storage.live.com/users/0x{id}/LiveFolders/{folder-path}/newfilename.jpg

Note: The {folder-id} used for the BITS upload is different the than folder-ids returned from the Live SDK. Live SDK returns a long-form folder-id that looks like this: folder.a5858c9cb698b77b.A5858C9CB698B77B!24220 The BITS upload API expects a short-form folder-id which is the last component of the Live SDK when separated on the period character. For example, A5858C9CB698B77B!24220.

Request Headers
Header Name Description
X-Http-Method-Override BITS uses the BITS_POST custom HTTP method. If your HTTP stack does not support custom protocols, you can use this header to change the HTTP method from POST to BITS_POST.
Authorization Set the OAuth access token received from OAuth 2.0.
BITS-Packet-Type Create-Session indicates that the request is to create a new upload session
BITS-Supported-Protocols The GUID represents the version of BITS used for the session.

Response

If successful, the service will return an HTTP 201 Created response:

HTTP/1.1 201 Created
Content-Length: 0
Date: Fri, 19 Nov 2010 10:31:16 GMT
BITS-Packet-Type: ACK
BITS-Protocol: {7df0354d-249b-430f-820d-3d2a9bef4931}
BITS-Session-Id: {session-id}
Response Headers
Header Name Description
BITS-Packet-Type ACK represents the server is acknowledging the request
BITS-Protocol The service echos back the GUID for the specific protocol implementation
BITS-Session-Id This value should be stored and included on subsequent requests to connect them to the session that was created.

This indicates the upload session has been created and is ready to receive the contents of the file. No file is created or visible in the user's OneDrive account until the upload session is complete. If the upload is not completed before the session ends, any received parts of the file are deleted from OneDrive.

Note An upload session is valid for a maximum of 24 hours. If the session has not completed within that time, the session and all uploaded data is deleted and clients must start the upload again from scratch.

Upload File Fragments

Your app or service should create a plan for how the file will be split into fragments to uploaded. OneDrive has a maximum fragment size of 60 MB.

For example, if you have a 150 MB file to upload, you may prefer to upload the file using 15 fragments each of 10 MB. This way if the connection is interrupted during the upload, the largest amount of data that would need to be uploaded again would be 10 MB.

Once your app has a plan for the number of fragments and their size, it can start uploading each fragment. To upload a fragment, the app will post to the same URL used to create the upload session, but with a different set of headers that define which upload session to use and the information about the fragment being uploaded.

Request

POST https://storage.live.com/users/0x{id}/LiveFolders/Portland/Terwiliger.jpg HTTP/1.1
X-Http-Method-Override: BITS_POST
Authorization: Bearer {access-token}
BITS-Packet-Type: Fragment
BITS-Session-Id: {session-id}
Content-Length: {length}
Content-Range: bytes {range}/{total-length}

{payload}
Request Headers
Header Name Description
X-Http-Method-Override BITS uses the BITS_POST custom HTTP method. If your HTTP stack does not support custom protocols, you can use this header to change the HTTP method from POST to BITS_POST.
Authorization Set the OAuth access token received from OAuth 2.0.
BITS-Packet-Type Fragment indicates that the request is part of the file content.
BITS-Session-Id The same value returned when creating the session should be written to this header.
Content-Length Number of bytes in the current fragment
Content-Range Bytes included in this fragment and the total length of the file. For example 0-10485759/1048576000 would indicate the fragment is the first 10 MB of a 100MB file. See RFC 2616 for more details.

Response

The server response to the uploaded fragment will provide information about how much of the file has been received and the next range of bytes expected.

HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 19 Nov 2010 10:31:16 GMT
BITS-Packet-Type: ACK
BITS-Session-Id: {session-id}
BITS-Received-Content-Range: {n}
Response Headers
Header Name Description
BITS-Packet-Type ACK represents the server is acknowledging the request
BITS-Session-Id This value connects the response to the upload session created in the initial request.
BITS-Received-Content-Range Number that indicates the number of bytes from the start of the file that have been received.

Note You MUST upload fragments of a file in order (from beginning to end of the file).

OneDrive does not support uploading bytes that have been previously uploaded. The service will respond with a 416 Range-Not-Satisfiable if the fragment request overlaps with an existing fragment that has already been received.

Closing The Upload Session

After each individual fragment has been uploaded, your application makes a request to close the upload session and commit the file to the user's OneDrive. If this call is not made, the file will not be saved and will be purged from temporary storage after 24 hours.

Request

POST https://storage.live.com/users/0x{cid}/LiveFolders/Portland/Terwiliger.jpg HTTP/1.1
X-Http-Method-Override: BITS_POST
Authorization: Bearer {access-token}
BITS-Packet-Type: Close-Session
BITS-Session-Id: {session-id}
Content-Length: 0
Request Headers
Header Name Description
X-Http-Method-Override BITS uses the BITS_POST custom HTTP method. If your HTTP stack does not support custom protocols, you can use this header to change the HTTP method from POST to BITS_POST.
Authorization Set the OAuth access token received from OAuth 2.0.
BITS-Packet-Type Close-Session indicates that the request to close the upload session.
BITS-Session-Id The same value returned when creating the session should be written to this header.
Content-Length Should always be 0 for this request.

Response

The server response to the uploaded fragment will provide information about how much of the file has been received and the next range of bytes expected.

HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 19 Nov 2010 10:31:16 GMT
BITS-Packet-Type: ACK
BITS-Session-Id: {session-id}
X-Resource-Id: {file-id}
Response Headers
Header Name Description
BITS-Packet-Type ACK represents the server is acknowledging the request
BITS-Session-Id This value connects the response to the upload session created in the initial request.
BITS-Received-Content-Range Number that indicates the number of bytes from the start of the file that have been received.
X-Resource-Id The resource ID for the uploaded file.

Note The value for X-Resource-Id is the resource identifier for the uploaded file in the user's OneDrive. You can use this value to make additional API calls for the file, for example to set additional metadata or properties on the file.

@Viktorianec
Copy link

Guys, have you example for iOS now? For big files, or working algorithm.

@rgregg
Copy link
Author

rgregg commented Feb 4, 2015

@BlackEnd11 Yes, this is only supported for OneDrive consumer. It is not supported for OneDrive for Business currently.

@like9515
Copy link

I have tested upload 10GB files and it seems OK. (takes about 4 hours)
To avoid server response 500 status, set fragment size a multiple 320KB

@xybu
Copy link

xybu commented Mar 2, 2015

What if the token expires during file transmission? The server continues to accept data till the last byte, but making a Close-Session request results in HTTP 500 with a header saying

www-authenticate: 'Bearer realm="OneDriveAPI", error="expired_token", error_description="Auth token expired. Try refreshing."'

So I try to refresh the token and then retry the Close-Session request, and the server now gives me AssertionException:

{
'bits-error-context': '0x5', 
'content-length': '0', 
'date': 'Mon, 02 Mar 2015 03:13:52 GMT', 
'bits-packet-type': 'Ack', 
'p3p': 'CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"', 
'x-asmversion': 'UNKNOWN; 19.14.0.0', 
'x-clienterrorcode': 'AssertionException', 
'server': 'Microsoft-HTTPAPI/2.0', 
'bits-error-code': '800300fd', 
'x-msnserver': 'BN1304____PAP162', 
'x-qosstats': '{"ApiId":0,"ResultType":4,"SourcePropertyId":0,"TargetPropertyId":42}', 
'x-throwsite': '5758.832a'
}

Any document about this?

I am reading this: https://msdn.microsoft.com/en-us/library/aa362712%28v=vs.85%29.aspx

The BITS client resends the Close-Session packet if the reason-code is in the range 500 through 599, unless the BITS-Error-Code header is present with a value of BG_E_SESSION_NOT_FOUND.

So retrying is valid, but didn't find clue about server-side assertion error.

@whoisprakhar
Copy link

is content-length equal to the length of fragment ? or the full content length of file ?

@whoisprakhar
Copy link

i get an error when i do a POST request to upload a fragment, but works fine with PUT, is this a issue ?

@gurramlokesh
Copy link

Does Onedrive support Parallel Fragment Upload now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment