Demonstrate requesting byte ranges from encrypted objects in Manta

MantaClientByteRangeRunner.java

package co.tjcelaya.sandbox;

import com.joyent.manta.client.MantaClient;
import com.joyent.manta.client.MantaObjectResponse;
import com.joyent.manta.client.crypto.AesCtrCipherDetails;
import com.joyent.manta.client.crypto.SecretKeyUtils;
import com.joyent.manta.config.*;
import com.joyent.manta.http.MantaHttpHeaders;
import org.apache.commons.io.IOUtils;

import javax.crypto.SecretKey;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.Base64;

public class App {
    public static void main(String[] args) {
        System.out.println("\n >>>> app boot\n");

        SecretKey k = SecretKeyUtils.generate(AesCtrCipherDetails.INSTANCE_256_BIT);

        ConfigContext config = new ChainedConfigContext(
                new DefaultsConfigContext(),
                new StandardConfigContext()
                        .setMantaURL("https://us-east.manta.joyent.com")
                        .setMantaUser("tomas.celaya")
                        .setMantaKeyId("<RESULT OF `ssh-keygen -l -f /PATH/TO/PRIVATE/KEY`>")
                        .setMantaKeyPath("/PATH/TO/PRIVATE/KEY")
                        .setDisableNativeSignatures(true)
                        .setNoAuth(false)
                        .setClientEncryptionEnabled(true)
                        .setEncryptionAuthenticationMode(EncryptionAuthenticationMode.Optional) // default is Mandatory which forbids range requests
                        .setEncryptionAlgorithm("AES256/CTR/NoPadding")
                        .setEncryptionKeyId("secret")
                        .setEncryptionPrivateKeyBytes(k.getEncoded())

        );

        try (MantaClient client = new MantaClient(config)) {
            String file = "/tomas.celaya/stor/byterangetest";
            int start = 5;
            int end = 11;

            byte[] localBytes = {0xA, 0xB, 0xC, 0xD, 0xE, 0xF, 0xA, 0xB, 0xC, 0xD};

            final int sliceSize = end - start + 1; // account for the need to store the byte at idx `end`

            System.out.println("localBytes.length: " + localBytes.length);
            System.out.println("slice length: " + sliceSize);
            int alloc = sliceSize;

            if (localBytes.length <= end) {
                int correction = end - localBytes.length + 1;
                System.out.println("subtracting from slice size: " + correction);
                alloc -= correction; // account for the need to store the byte at idx `end`
            }

            System.out.println("bytes allocated: " + alloc);
            byte[] localRangeBytes = new byte[alloc];
            for (int i = start, idx = 0; i <= end && i < localBytes.length; i++, idx++) {
                localRangeBytes[idx] = localBytes[i];
            }

            System.out.println("localBytes:      " + Arrays.toString(localBytes));
            System.out.println("localRangeBytes: " + Arrays.toString(localRangeBytes));

            try {
                client.put(file, localBytes);
            } catch (IOException e) {
                e.printStackTrace();
                System.exit(1);
            }

            try (final InputStream is = client.getAsInputStream(file, new MantaHttpHeaders(), (long) start, (long) end)) {
                byte[] mantaRangeBytes =  IOUtils.toByteArray(is);
                System.out.println("mantaRangeBytes: " + Arrays.toString(mantaRangeBytes));

                if (!Arrays.equals(localRangeBytes, mantaRangeBytes)) {
                    throw new Error("there is a bug");
                }
            } catch (IOException e) {
                e.printStackTrace();
                System.exit(1);
            }

            System.out.println("looks ok from here, I wonder what the content length on the object is?");

            try {
                final MantaObjectResponse mantaObjectResponse = client.head(file);
                System.out.println("contentLength: " + mantaObjectResponse.getContentLength());
            } catch (IOException e) {
                e.printStackTrace();
                System.exit(1);
            }
        }

        System.out.println("\n <<<< app terminate\n");
    }
}

output

 >>>> app boot

localBytes.length: 10
slice length: 7
subtracting from slice size: 2
bytes allocated: 5
localBytes:      [10, 11, 12, 13, 14, 15, 10, 11, 12, 13]
localRangeBytes: [15, 10, 11, 12, 13]
mantaRangeBytes: [15, 10, 11, 12, 13]
looks ok from here, I wonder what the content length on the object is?
contentLength: 26

 <<<< app terminate

I modified the example to demonstrate a really simple way the there is a bug execution path gets taken, although I'm seeing a few errors trying to run it. The first was that the oracle-jdk wouldn't register the bouncy-castle jce for some reason. The next was a 403 when trying to put the test file to manta, which is probably an issue with the credentials I'm trying to feed it.

The changes are pretty straightforward (if you have time to test)...

int end = 11 + AesCtrCipherDetails.INSTANCE_256_BIT.getAuthenticationTagOrHmacLengthInBytes();

and

MantaMetadata metadata = new MantaMetadata();
metadata.put(MantaHttpHeaders.ENCRYPTION_PLAINTEXT_CONTENT_LENGTH, String.valueOf(Long.MIN_VALUE));

client.putMetadata(file, metadata);

I think I figured out the auth issue, but I don't have time to test at the moment. I was trying to use the key id that's read from the environment variable as the key id in the configuration context, which look like they're formatted differently.

I'll take another look at this again tonight.

Closed the PR and created a new issue. This is more an issue with us allowing users to overwrite fields that are critical for encryption to work as expected. Being able to trick the client into passing on the wrong content-length is the central issue, incorrect bytes appearing where they aren't expected is just a symptom.