2024-07-19-witness-value-extraction-notes.md

notes on witness extraction values:

An example of the witness file we want to generate.

what we're trying to find today

these 9 values are saved in the JJ golang establishHandshakeKeys method, in handshake_client_tls13.go, which is called by handshake:

• DHE sharedKey - hs.ecdheKey is the client key. is this just the PreMasterSecret?
• ES earlySecret - derived from the psk
• dES derived EarlySecret
• HS handshakeSecret
• CHTS clientSecret
• H2
• SHTS serverSecret
• dHS derived HandshakeSecret
• MS masterSecret
• mirror tracy's work on pinning the master secret and handshake hash.

what we found:

DHE sharedKey

go client:

// sharedKey = DHE, hs.ecdheKey is the client key
sharedKey, err := hs.ecdheKey.ECDH(peerKey) // old line
if err != nil {
    c.sendAlert(alertIllegalParameter)
    return errors.New("tls: invalid server key share")
}
c.SetSecret("DHE", sharedKey)

rust client: Tracy's RustCryptoBackend13::set_pre_master_secret is never called. pre_master_secret is initially set to none, and updated in Tracy's set_server_key_share (grr Tracy).

async fn set_server_key_share(&mut self, key: PublicKey) -> Result<(), BackendError> {
    let sk = self.ecdh_secret.as_ref().unwrap();
    let server_pk =
        ECDHPublicKey::from_sec1_bytes(&key.key).map_err(|_| BackendError::InvalidServerKey)?;
   
    // Start with diffie hellman dto produce the shared pre_master_secret
    let mut pms = [0u8; 32]; // NOTE: 32 bytes in both impls
    let secret = *sk.diffie_hellman(&server_pk).raw_secret_bytes();
    pms.copy_from_slice(&secret);
    self.pre_master_secret = Some(pms);

ES earlySecret and dES

go client - ES=hkdf-extract(_, _)

// earlySecret = ES
earlySecret := hs.earlySecret
if !hs.usingPSK {
    earlySecret = hs.suite.extract(nil, nil)
}
// fmt.Println("ES:", hex.EncodeToString(earlySecret))
c.SetSecret("ES", earlySecret)
// - snip -
// extract implements HKDF-Extract with the cipher suite hash.
func (c *cipherSuiteTLS13) extract(newSecret, currentSecret []byte) []byte {
	if newSecret == nil {
		newSecret = make([]byte, c.hash.Size())
	}
	return hkdf.Extract(c.hash.New, newSecret, currentSecret)
}
// - snip -
// Extract generates a pseudorandom key for use with Expand from an input secret
// and an optional independent salt.
//
// Only use this function if you need to reuse the extracted key with multiple
// Expand invocations and different context values. Most common scenarios,
// including the generation of multiple keys, should use New instead.

dES = hkdf-expand(ES, "derived", _):

// derive handshake traffic secret client and server
// dES: hs.suite.deriveSecret(earlySecret, "derived", nil)
// fmt.Println("dES:", hex.EncodeToString(hs.suite.deriveSecret(earlySecret, "derived", nil)))
c.SetSecret("dES", hs.suite.deriveSecret(earlySecret, "derived", nil))
// - snip - 
// deriveSecret implements Derive-Secret from RFC 8446, Section 7.1.
func (c *cipherSuiteTLS13) deriveSecret(secret []byte, label string, transcript hash.Hash) []byte {
	if transcript == nil {
		transcript = c.hash.New()
	}
	// expandLabel implements HKDF-Expand-Label from RFC 8446, Section 7.1.
	return c.expandLabel(secret, label, transcript.Sum(nil), c.hash.Size())
}

leading candidate for ES and dES:

let mut hasher = Sha256::new();
hasher.update(b"");
let context = hasher.finalize();
// ES
let current = self.hkdf_provider.extract_from_zero_ikm(None);
// dES cand 1
let k0_salt = hkdf_expand_label_block(current.as_ref(), b"derived", &context);

aside: transcript or ems is probably h2

Another candidate for ES is the initial value of transcript, called in ExpectServerHello::handle. It would appear handle_server_helo only toggles the state of component values, set the hs_hash_client_key_exchange (the finalized transcript value, see below) and server_key_share values, but does not obviously include components I need.

// Start our handshake hash, and input the server-hello.
// transcript is a hash of `m`
let mut transcript = self.transcript_buffer.start_hash(suite.hash_algorithm());
transcript.add_message(&m);

The next method in the rust flow is set_encrypt and set_decrypt which updates the protocol state from EarlyData to Handshake, and computes keys handshake keys with derive_keys and get_{en,de}crypter(keys).

update - see H2 below. Note that transcript is finalized by a call to set_hs_hash_client_key_exchange, which sets ems_seed. h2 may be either transcript or ems_seed.

HS handshake secret

go client HS=hkdf-extract(sharedKey, dES). Recall deriveSecret calls hkdf-expand:

handshakeSecret := hs.suite.extract(sharedKey,
    hs.suite.deriveSecret(earlySecret, "derived", nil))
c.SetSecret("HS", handshakeSecret)

// immediately following computation of dES:
// HS
let k0_secret = self.hkdf_provider.extract_from_secret(Some(k0_salt.as_ref()), &self.pre_master_secret.unwrap());

CHTS client secret, SHTS, H2

go client:

• CHTS=hkdf-expand(HS, "c hs traffic", transcript)
• note the label change from c to s SHTS=hkdf-expand(HS, "s hs traffic", transcript)
• H2=hs.transcript - note: transcript aside above in section ES.

clientSecret := hs.suite.deriveSecret(handshakeSecret,
    clientHandshakeTrafficLabel, hs.transcript)
c.SetSecret("CHTS", clientSecret)

serverSecret := hs.suite.deriveSecret(handshakeSecret,
    serverHandshakeTrafficLabel, hs.transcript)
// h2 is likely the transcript hash, see prior aside
c.SetSecret("H2", hs.transcript.Sum(nil))
c.SetSecret("SHTS", serverSecret)

// note: ems set in `handle_server_hello` by `set_hs_hash_client_key_exchange` which finalizes the transcript hash
let context = self.ems_seed.clone().unwrap();  
// CHTS
let client_secret = Some(hkdf_expand_label_block(master_secret.as_ref(), client_label, &context));
// SHTS
let server_secret = Some(hkdf_expand_label_block(master_secret.as_ref(), server_label, &context));

dHS, MS

go client:

• dHS=expand(HS, "derived", _)
• MS=extract(_, dHS)

c.SetSecret("dHS", hs.suite.deriveSecret(handshakeSecret, "derived", nil))
hs.masterSecret = hs.suite.extract(nil,
    hs.suite.deriveSecret(handshakeSecret, "derived", nil))
c.SetSecret("MS", hs.masterSecret)

Recall in rustls that HS is pronounced k0_secret.

Huh, this looks like it shouldn't work, and that at best Tracy has written this confusingly (grr) or at worst, there's bugs here and it's not obvious that this should actually work. Mm, no, it looks like this might be the former, just misleadingly written but we can be cool. We're cool. Not upset. Cool.

// if derive keys for handshake layer:
let (master_secret, client_label, server_label) =  (k0_secret, b"c hs traffic", b"s hs traffic")

// if derive keys for application layer:
let (master_secret, client_label, server_label) = {
    // dHS
    let k1_salt = hkdf_expand_label_block(k0_secret.as_ref(), b"derived", &context);
    // MS
    let k1_secret = self.hkdf_provider.extract_from_secret(Some(k1_salt.as_ref()), &[0u8; 32]);
    trace!("k1_salt={:?}", hex::encode(k1_salt.as_ref()));
    
    (k1_secret, b"c ap traffic", b"s ap traffic")
}

note: what happens next

Compute some AES values then wander back to one of the 5 places derive_keys is called.

// Finally, derive the actual AES key and IV for each secret. 
let e = self.hkdf_provider.expander_for_okm(&client_secret.unwrap());
let client_aes_key = hkdf_expand_label_aead_key(e.as_ref(), 16, b"key", &[]);
let client_aes_iv = hkdf_expand_label_aead_key(e.as_ref(), 12, b"iv", &[]);

let e = self.hkdf_provider.expander_for_okm(&server_secret.unwrap());
let server_aes_key = hkdf_expand_label_aead_key(e.as_ref(), 16, b"key", &[]);
let server_aes_iv = hkdf_expand_label_aead_key(e.as_ref(), 12, b"iv", &[]);

TlsKeys {
    client_key: client_aes_key,
    client_iv: client_aes_iv,
    server_key: server_aes_key,
    server_iv: server_aes_iv
}

now for the four remaining witness values:

• H7 - here
• CATS - here
• H3 - here
• SATS - here

In handshake, after establishHandshakeKeys is called, readServerParameters is called, in which no values are saved, then readServerCertificate is called in which h7 is set. Finally readServerFinished is called, where the remainder of the values are saved.

readServerParameters does some extension work, including ALPN settings, but sets no values as mentioned above.

h7

readServerCertificate takes the handshake certificate, performs some certificate verification logic before eventually computing h7 is computed from the sum of the transcript after reading the server certificate.

hs.transcript is not directly assigned to at any point in the method, but is referenced in methods:

• readHandshake
• signedMessage
• transcriptMsg - seems most likely to update the transcript. See here; transcriptMsg Writes data to transcript.

func transcriptMsg(msg handshakeMessage, h transcriptHash) error {
	data, err := msg.marshal()
	if err != nil { return err }
	h.Write(data) // <-- write to transcript
	return nil
}

in RusTls:

// unsure yet, see below on h3 / h7

CATS

readServerFinished defines the remainder of the values. CATS = hkdf-expand(MS, CAT, "c ap traffic", transcript or h7)

// jan: here, derive application traffic secrets!
hs.trafficSecret = hs.suite.deriveSecret(hs.masterSecret, clientApplicationTrafficLabel, hs.transcript)
c.SetSecret("CATS", hs.trafficSecret)

deriveKeys:

let client_secret = Some(hkdf_expand_label_block(master_secret.as_ref(), client_label, &context));

H3, SATS

SATS = hkdf-expand(ms, "s ap traffic", transcript or h3)

serverSecret := hs.suite.deriveSecret(hs.masterSecret, serverApplicationTrafficLabel, hs.transcript)
c.SetSecret("H3", hs.transcript.Sum(nil))
c.SetSecret("SATS", serverSecret)

in RusTls:

// h3 or h7?
let context = self.ems_seed.clone().unwrap();  
// SATS
let server_secret = Some(hkdf_expand_label_block(master_secret.as_ref(), server_label, &context));

Onto pinning and testing!