Skip to content

Instantly share code, notes, and snippets.

@rgregg
Last active May 26, 2022 09:17
Show Gist options
  • Star 20 You must be signed in to star a gist
  • Fork 8 You must be signed in to fork a gist
  • Save rgregg/37ba8929768a62131e85 to your computer and use it in GitHub Desktop.
Save rgregg/37ba8929768a62131e85 to your computer and use it in GitHub Desktop.
OneDrive Large File Upload API

Uploading Large Files

NOTICE: The OneDrive API now has native supported for large file uploads. Instead of using BITS, consider using the OneDrive API upload method instead!

Note This is preliminary documentation and is subject to change as we evolve the OneDrive API.

OneDrive handles uploading large files by supporting the BITS protocol. BITS is a simple extension to HTTP that enables resumable file uploads to OneDrive.

Where the standard PUT method for uploading a file has a 100 MB limit, using the BITS protocol will allow you to upload much larger files. BITS also provides connection tolerance by splitting the file into separate fragments which are uploaded sequentially and reassembled into the original file on the service. If a fragment transfer is interrupted, only that fragment needs to be uploaded again, previously uploaded fragments are preserved on the server.

OneDrive supports the standard BITS protocol for this scenario.

Note: Unlike the Live SDK, there is no logic to resize / resample photos that runs when photos are uploaded with BITS.

To upload a large file, a particular call pattern is required that looks like this:

  1. Request an Upload Session
  2. Upload File Fragments
  3. Close the Upload Session

Request an Upload Session

To begin, your app makes a request to OneDrive to create a new upload session.

Request

You will need to use the user's ID, a unique identifier for OneDrive, to build the URL. You can retrieve the user's ID using the Live SDK by making a request for the user's profile information. The value returned in the id field is the user's ID.

Once you have the user's ID, you can build the request to create an upload session. In this example, we're uploading the file Terwiliger.jpg into a folder named Portland in the root of the user's OneDrive.

POST https://storage.live.com/users/0x{id}/LiveFolders/Portland/Terwiliger.jpg HTTP/1.1
X-Http-Method-Override: BITS_POST
Authorization: Bearer {access-token}
BITS-Packet-Type: Create-Session
BITS-Supported-Protocols: {7df0354d-249b-430f-820d-3d2a9bef4931}

Note You can upload files based on the folder's resource ID or the path of the folder. To upload using the folder resource ID, use a URL format like this:

https://storage.live.com/items/{folder-id}/newfilename.jpg

To upload using path, the URL format should look like this:

https://storage.live.com/users/0x{id}/LiveFolders/{folder-path}/newfilename.jpg

Note: The {folder-id} used for the BITS upload is different the than folder-ids returned from the Live SDK. Live SDK returns a long-form folder-id that looks like this: folder.a5858c9cb698b77b.A5858C9CB698B77B!24220 The BITS upload API expects a short-form folder-id which is the last component of the Live SDK when separated on the period character. For example, A5858C9CB698B77B!24220.

Request Headers
Header Name Description
X-Http-Method-Override BITS uses the BITS_POST custom HTTP method. If your HTTP stack does not support custom protocols, you can use this header to change the HTTP method from POST to BITS_POST.
Authorization Set the OAuth access token received from OAuth 2.0.
BITS-Packet-Type Create-Session indicates that the request is to create a new upload session
BITS-Supported-Protocols The GUID represents the version of BITS used for the session.

Response

If successful, the service will return an HTTP 201 Created response:

HTTP/1.1 201 Created
Content-Length: 0
Date: Fri, 19 Nov 2010 10:31:16 GMT
BITS-Packet-Type: ACK
BITS-Protocol: {7df0354d-249b-430f-820d-3d2a9bef4931}
BITS-Session-Id: {session-id}
Response Headers
Header Name Description
BITS-Packet-Type ACK represents the server is acknowledging the request
BITS-Protocol The service echos back the GUID for the specific protocol implementation
BITS-Session-Id This value should be stored and included on subsequent requests to connect them to the session that was created.

This indicates the upload session has been created and is ready to receive the contents of the file. No file is created or visible in the user's OneDrive account until the upload session is complete. If the upload is not completed before the session ends, any received parts of the file are deleted from OneDrive.

Note An upload session is valid for a maximum of 24 hours. If the session has not completed within that time, the session and all uploaded data is deleted and clients must start the upload again from scratch.

Upload File Fragments

Your app or service should create a plan for how the file will be split into fragments to uploaded. OneDrive has a maximum fragment size of 60 MB.

For example, if you have a 150 MB file to upload, you may prefer to upload the file using 15 fragments each of 10 MB. This way if the connection is interrupted during the upload, the largest amount of data that would need to be uploaded again would be 10 MB.

Once your app has a plan for the number of fragments and their size, it can start uploading each fragment. To upload a fragment, the app will post to the same URL used to create the upload session, but with a different set of headers that define which upload session to use and the information about the fragment being uploaded.

Request

POST https://storage.live.com/users/0x{id}/LiveFolders/Portland/Terwiliger.jpg HTTP/1.1
X-Http-Method-Override: BITS_POST
Authorization: Bearer {access-token}
BITS-Packet-Type: Fragment
BITS-Session-Id: {session-id}
Content-Length: {length}
Content-Range: bytes {range}/{total-length}

{payload}
Request Headers
Header Name Description
X-Http-Method-Override BITS uses the BITS_POST custom HTTP method. If your HTTP stack does not support custom protocols, you can use this header to change the HTTP method from POST to BITS_POST.
Authorization Set the OAuth access token received from OAuth 2.0.
BITS-Packet-Type Fragment indicates that the request is part of the file content.
BITS-Session-Id The same value returned when creating the session should be written to this header.
Content-Length Number of bytes in the current fragment
Content-Range Bytes included in this fragment and the total length of the file. For example 0-10485759/1048576000 would indicate the fragment is the first 10 MB of a 100MB file. See RFC 2616 for more details.

Response

The server response to the uploaded fragment will provide information about how much of the file has been received and the next range of bytes expected.

HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 19 Nov 2010 10:31:16 GMT
BITS-Packet-Type: ACK
BITS-Session-Id: {session-id}
BITS-Received-Content-Range: {n}
Response Headers
Header Name Description
BITS-Packet-Type ACK represents the server is acknowledging the request
BITS-Session-Id This value connects the response to the upload session created in the initial request.
BITS-Received-Content-Range Number that indicates the number of bytes from the start of the file that have been received.

Note You MUST upload fragments of a file in order (from beginning to end of the file).

OneDrive does not support uploading bytes that have been previously uploaded. The service will respond with a 416 Range-Not-Satisfiable if the fragment request overlaps with an existing fragment that has already been received.

Closing The Upload Session

After each individual fragment has been uploaded, your application makes a request to close the upload session and commit the file to the user's OneDrive. If this call is not made, the file will not be saved and will be purged from temporary storage after 24 hours.

Request

POST https://storage.live.com/users/0x{cid}/LiveFolders/Portland/Terwiliger.jpg HTTP/1.1
X-Http-Method-Override: BITS_POST
Authorization: Bearer {access-token}
BITS-Packet-Type: Close-Session
BITS-Session-Id: {session-id}
Content-Length: 0
Request Headers
Header Name Description
X-Http-Method-Override BITS uses the BITS_POST custom HTTP method. If your HTTP stack does not support custom protocols, you can use this header to change the HTTP method from POST to BITS_POST.
Authorization Set the OAuth access token received from OAuth 2.0.
BITS-Packet-Type Close-Session indicates that the request to close the upload session.
BITS-Session-Id The same value returned when creating the session should be written to this header.
Content-Length Should always be 0 for this request.

Response

The server response to the uploaded fragment will provide information about how much of the file has been received and the next range of bytes expected.

HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 19 Nov 2010 10:31:16 GMT
BITS-Packet-Type: ACK
BITS-Session-Id: {session-id}
X-Resource-Id: {file-id}
Response Headers
Header Name Description
BITS-Packet-Type ACK represents the server is acknowledging the request
BITS-Session-Id This value connects the response to the upload session created in the initial request.
BITS-Received-Content-Range Number that indicates the number of bytes from the start of the file that have been received.
X-Resource-Id The resource ID for the uploaded file.

Note The value for X-Resource-Id is the resource identifier for the uploaded file in the user's OneDrive. You can use this value to make additional API calls for the file, for example to set additional metadata or properties on the file.

@mrdone
Copy link

mrdone commented Dec 17, 2014

@rgregg i am using this to do the file uploading, and come across the "expired token" issue. that is: if the access token expired, the "close session" operation will never success, and i got the error code "401". and after refresh the token to a new one, the error code "500" is received from the server. any comments?

@mrdone
Copy link

mrdone commented Dec 17, 2014

@rgregg i have already fixed this problem, my solution is: before closing the session, just do the token-refresh, and use the new access token to close the session. my understanding here is: if the close session request is sent, the server will destroy the session, no matter whether it succeeded or not.

@jensanthegreat
Copy link

Hi,

I have problems on uploading the final Fragment request. I am able to upload the chunks from the beginning, but when it reaches the last one, it gives 400 and BadArgument error.

I can't find out what the problem is. Anyone got ideas?

Content-Range=[bytes 0-8191/17170] << Response 200
Content-Range=[bytes 8192-16383/17170] << Response 200
Content-Range=[bytes 16384-17169/17170] << Response 400

The other headers are just the same for each requests (of course the content-length is different). I have also tried to provide the content-length and remove the content-length. They just behave the same.

Also tried to upload 1 bytes, so that only one Fragment is requested
Content-Range=[bytes 0-0/1]
Content-Length=1
It still gives Response 400....

@ificator
Copy link

@everyone

For documentation clarifications, feel free to comment here. For issues with using the API please create a question on StackOverflow and tag it as "onedrive".

@mrdone
Copy link

mrdone commented Dec 23, 2014

@jensanthegreat
i think u may given the wrong content-length:
POST /users/0x{id}/LiveFolders/Portland/Terwiliger.jpg HTTP/1.1
Host: cid-{id}.users.storage.live.com
X-Http-Method-Override: BITS_POST
Authorization: Bearer {access-token}
BITS-Packet-Type: Fragment
BITS-Session-Id: {session-id}
Content-Length: {length}
Content-Range: bytes {range}/{total-length}

@quintonn
Copy link

@ificator
I got my upload working, thanks.

Does anyone know if there a way to also download large files using this protocol?

@ificator
Copy link

@quintonn
For downloads you can use Range header as outlined in the RFCs (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35). However, I'd also recommend getting the "Etag" header returned with your first partial download, and passing it back as the "If-Match" header to ensure the file contents doesn't change as you retrieve it. Normal responses will be "206 Partial Content", while change will result in "412 Preconditioned Failed"

GET https://8xaxda-sn3302.files-df.1drv.com/y2pUBblablah HTTP/1.1
Range: bytes=0-100

HTTP/1.1 206 Partial Content
Content-Length: 101
Etag: A905AE92F2E3E76D!1804.0
GET https://8xaxda-sn3302.files-df.1drv.com/y2pUBblablah HTTP/1.1
Range: bytes=101-200
If-Match: A905AE92F2E3E76D!1804.0

HTTP/1.1 206 Partial Content
Content-Length: 101
Etag: A905AE92F2E3E76D!1804.0
GET https://8xaxda-sn3302.files-df.1drv.com/y2pUBblablah HTTP/1.1
Range: bytes=101-200
If-Match: WRONG

HTTP/1.1 412 Precondition Failed
Content-Length: 0

The URL I used in the above examples was the direct download URL, however you can use a URL similar to what you used to upload (as documented on this page) with the caveat that you may be redirected depending on the caller, and so be sure to handle any 301s that come back.

@AndreyChernukha
Copy link

I'm having an awful issue. I described it in detail on StackOverflow http://stackoverflow.com/questions/27600577/uploading-files-to-onedrive-401-error

Let me explain it shortly here once more: After a certain period (seems like it is 24 hours but I'm not sure) of being logged in as OneDrive user I become unable to upload files using BITS protocol because I receive 401 error when trying to open upload session. At the same time the LiveConnectClient's session seems to be not expired and I can still upload files by using LiveOperation. Also I can easily get user id using the same access token which is used when opening an upload session. If I logout and login again then again everything works like charm and I'm able to create upload session and upload files. Here is the example of response that I receive:

<NSHTTPURLResponse: 0x7bc31960> { URL: https://cid-da1cedf5484811a9.users.storage.live.com/items/DA1CEDF5484811A9!373/IMG_0007.JPG } { status code: 401, headers {
"BITS-Packet-Type" = Ack;
"Content-Length" = 0;
Date = "Tue, 30 Dec 2014 12:20:27 GMT";
P3P = "CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"";
Server = "Microsoft-HTTPAPI/2.0";
"X-AsmVersion" = "UNKNOWN; 19.9.0.0";
"X-ClientErrorCode" = AccessDenied;
"X-MSNSERVER" = "SN3301____PAP243";
"X-QosStats" = "{"ApiId":0,"ResultType":2,"SourcePropertyId":0,"TargetPropertyId":42}";
"X-ThrowSite" = "4e1b.c093";
} }

@vanjasme
Copy link

please change in documentation

https://cid-{id}.users.storage.live.com/users/0x{id}/items/{folder-id}/newfilename.jpg

to correct form

https://cid-{id}.users.storage.live.com/Items/{folder-id}/newfilename.jpg

otherwise developers will get 404 error when use

https://cid-{id}.users.storage.live.com/users/0x{id}/items/{folder-id}/newfilename.jpg

@kishorethecoder
Copy link

i just getting 404 when tried with this on response

string url="https://cid-15efb69f03c7e2a0.users.storage.live.com/Items/15EFB69F03C7E2A0!107/12.txt"
string guidvalue = Guid.NewGuid().ToString();
var request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "Post";
request.ContentLength = 0;
request.Headers.Add("X-Http-Method-Override", "BITS_POST");
request.Headers.Add("Authorization", "Bearer " + _accessToken);
request.Headers.Add("BITS-Packet-Type", "Create-Session");
request.Headers.Add("BITS-Supported-Protocols", guidvalue);

using (var response = request.GetResponse())
{

        }

@quintonn
Copy link

@ificator
I am trying the download but not succeeding.
Do I need to start a session? Or should the access token be enough?
Are you able to do a download successfully?
Could you maybe share the request with a URL that is not the direct URL?
At the moment i am getting html back which contains info to login (probably login redirection).
I am trying to call this code using c#, server side. So not really able to go to a login page or anything.

@ajcsoftware
Copy link

I have the chunked uploading working fine but I have one problem. I can't upload a zero length file. If I do just a start and a close call I get a "(416) requested range not satisfiable" error and if I try a start, and a zero length fragment I get "(400) Bad Request.".

How should I upload a zero length file?

@CastleSoft
Copy link

Any chance you can do a simple file upload example in C# ? (with the Authentication/token/userid) stuff also ?

@ificator
Copy link

Here's an example program that uploads a specified file using a specified token:

https://gist.github.com/ificator/3460d7b9d0bff74eb0ff

If you don't have a token, you should be able to obtain one using the LiveSDK.

@quintonn
Copy link

@ificator
I can't get the download working without the direct download URL.
Is there any where I can get more info on the downloading.
Eg, Where can i get the direct download URL from without doing an upload first?
Is there maybe a way to query this API for available files etc?
Also, without having done an upload, how will i know how big the file is?
Any help will be much appreciated. Thanks for your support on this blog. It's amazing

@quintonn
Copy link

Ignore previous post. All this info is available using the normal Live REST Api

@BlackEnd11
Copy link

From all the examples I have seen, I tend to think the BITS API is supported only from OneDrive and not from OneDrive for Business. Can you please confirm that?

@Viktorianec
Copy link

Guys, have you example for iOS now? For big files, or working algorithm.

@rgregg
Copy link
Author

rgregg commented Feb 4, 2015

@BlackEnd11 Yes, this is only supported for OneDrive consumer. It is not supported for OneDrive for Business currently.

@like9515
Copy link

I have tested upload 10GB files and it seems OK. (takes about 4 hours)
To avoid server response 500 status, set fragment size a multiple 320KB

@xybu
Copy link

xybu commented Mar 2, 2015

What if the token expires during file transmission? The server continues to accept data till the last byte, but making a Close-Session request results in HTTP 500 with a header saying

www-authenticate: 'Bearer realm="OneDriveAPI", error="expired_token", error_description="Auth token expired. Try refreshing."'

So I try to refresh the token and then retry the Close-Session request, and the server now gives me AssertionException:

{
'bits-error-context': '0x5', 
'content-length': '0', 
'date': 'Mon, 02 Mar 2015 03:13:52 GMT', 
'bits-packet-type': 'Ack', 
'p3p': 'CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"', 
'x-asmversion': 'UNKNOWN; 19.14.0.0', 
'x-clienterrorcode': 'AssertionException', 
'server': 'Microsoft-HTTPAPI/2.0', 
'bits-error-code': '800300fd', 
'x-msnserver': 'BN1304____PAP162', 
'x-qosstats': '{"ApiId":0,"ResultType":4,"SourcePropertyId":0,"TargetPropertyId":42}', 
'x-throwsite': '5758.832a'
}

Any document about this?

I am reading this: https://msdn.microsoft.com/en-us/library/aa362712%28v=vs.85%29.aspx

The BITS client resends the Close-Session packet if the reason-code is in the range 500 through 599, unless the BITS-Error-Code header is present with a value of BG_E_SESSION_NOT_FOUND.

So retrying is valid, but didn't find clue about server-side assertion error.

@whoisprakhar
Copy link

is content-length equal to the length of fragment ? or the full content length of file ?

@whoisprakhar
Copy link

i get an error when i do a POST request to upload a fragment, but works fine with PUT, is this a issue ?

@gurramlokesh
Copy link

Does Onedrive support Parallel Fragment Upload now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment