Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save asikaria/0a806091655c6e963eea59e89fdd40a9 to your computer and use it in GitHub Desktop.
Save asikaria/0a806091655c6e963eea59e89fdd40a9 to your computer and use it in GitHub Desktop.
Doing ConcurrentAppend to Azure Data Lake Store using Java
public static void main( String[] args ) throws IOException {
ClientCredsTokenProvider tokenProvider = new ClientCredsTokenProvider("...", "...", "..."); // fill in placeholders
ADLStoreClient client = ADLStoreClient.createClient("AccountName.azureDatalakestore.net",
tokenProvider);
byte[] myBuffer = new byte[256];
// fill in myBuffer content here
RequestOptions opts = new RequestOptions();
// use ONE of the below two lines
opts.retryPolicy = new ExponentialBackoffPolicy(); // for at-least-once insertion
opts.retryPolicy = new NoRetryPolicy(); // for at-most-once insertion
OperationResponse resp = new OperationResponse();
Core.concurrentAppend("/path/to/file",
myBuffer,
0,
myBuffer.length,
true,
client,
opts,
resp
);
if (!resp.successful) {
throw client.getExceptionFromResponse(resp, "Meaningful error message");
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment