1
"substantially more efficient solution than traditional public key encryption" The main point of hybrid encryption is to process data of arbitrary size, with efficiency as a by-product (of course you could imagine things like RSA-OAEP-ECB but these make little sense)
3
"Ephemeral (E): A fresh random value meant for one-time use." This sounds like is a single value/nonce whereas it seems to instead be about an ephemeral key pair, such that E is seen as an entity.
"(skX, pkX): A KEM key pair used in role X; skX is the private key and pkX is the public key" S and R are described as "entities" rather than "roles", maybe be more consistent? (and define E as an entity/role as well?)
"encode_big_endian(x, n): An byte string encoding the integer value x as an n-byte big-endian value" s/An byte/A byte/ Comment about said encoding? e.g. wrt signed vs. unsigned Need to say that n must be at least ceil(log(x)/8) and that otherwise an error is returned (defintion of xor() mentions error case so do it here too?)
4
Primitive components lists: Say which algorithms are randomized and which arent? Say when decap algoritm fails?
Marshal/Unmarshal: sounds to me like these definitions are more about serialization, which is almost the same thing but slightly different.
KDF: definition matches HKDR but 1) are these subcomponents needed vs. a single KDF primitive? 2) a KDF doesnt necessarily follow the extract-the-expand construction, let alone one with that specific interface. I don't see why KDF should be limited to HKDF, which is probably one of the least efficient and simple constructions.
AEAD "returning plaintext message pt or the error value OpenError" Also specify possible errors and their types in KEM?
LabeledExtract/LabeledExpand Seems overcomplicated vs. defining a single KDF interface, and vs. using the info field, which is designed for such personalization.
4.1
" GenerateKeyPair(): Generate an ephemeral key pair (sk, pk) for the DH group in use DH(sk, pk): Perform a non-interactive DH exchange using the private key sk and public key pk to produce a Diffie-Hellman shared secret of length Ndh " This might be confusing to some readers, as (sk, pk) is first described as a key pair and then (sk, pk) is used to refer to keys from different pairs.
"def Encap(pkR): (...) enc = Marshal(pkE) " Why "enc" here? "enc" sounds like an abbreviation of "encryption" which does not seem the most adequate term here. Also, "zz" looks like a surprising naming choice, inconsistent with other variables' names.
Decap definition: Is it needed to specify that DH() could fail?
" curves referenced in {#ciphersuites} are as" Missing link formatting?
5
"are transported between the sender and the recipient by some application making use of HPKE" Specify what is expected from the channel? (integrity etc.)
5.1
" share a common key schedule" The term "key schedule" is usually reserved to block cipher's key-to-subkeys mechanism, doesnt seem like the right term here.
"no_psk = (psk == default_psk and pskID == default_pskID)" shouldnt it be a "or"? or just !got_psk; otherwise both got_psk and no_psk can be false.
just set "no_pkSm = !got_pkSm"?
KeySchedule: looks weird to just use "LabeledExtract" as a primitive, whereas it's just a subcomponent of HKDF created for simplifying the security proof. Just use KDFs, or PRFs if needed, but don't introduce new primitive types or rely on a given primitive's specifics.
5.1.2
"The PSK is used as the salt input to the KDF (instead of 0)" Probably not security issue but a salt is generally not a secret, as for example the HKDF RCC says: "salt optional salt value (a non-secret random value);". KDF interfaces separate on purpose secret from non-secret material, I recommend that HPKE doesnt violate this established rule, could be an issue wrt side channels, for example when expected-secret variables are wiped off memory whereas expected-public ones arent.
5.1.3
" wishes to authenticate some other identity for the sender (e.g., an email address or domain name)" I get the idea but this can be misleading to readers, since such values are identifiers, not authentication values. Maybe clarify.
5.1.2 - 5.1.4
As guidance to users/readers, maybe provide examples of use cases and/or adversarial models when one of the is suitable (or not)?
5.2
"MUST return an error if the sequence number overflows" That's a necessary, but not sufficient condition for repeated seqnums (since a number can wrap around without going back to its initial value).
Should something be said about seqnums initial value? (To hide packet index etc.)
5.3
"secret export" sounds non-standard to me as a terminology, this looks like a variable-length PRF, e.g. instantiable by a XOF of deck function.
7.1.2
Validation terms look ok.
"MUST check whether the shared secret is the all-zero value" Maybe clarify that this refers to the DH secret and not to zz (which is called "shared secret" e.g. in 5.1).
7.3
What about directly supporting a misuse resistant scheme? cf. RFCs 5297 and 8452 (SIV'd AES and AES-GCM).
8.1
"Privacy of the sender's messages, i.e., IND-CCA2 security" strictly speaking, "privacy" should be confidentiality or secrety, and only refers to the goal (captured by IND); maybe add the adversarial model (the CCA2 part).
"as long as the the underlying KEM and AEAD schemes are IND-CCA2-secure." Not fully sure, but I think the distinction CCA1/CCA2 does not appy for IND-CCA as applied to AEADs. Might note that IND-CCA is equivalent to INT-CTXT/PTXT for AEADs.
"Extract is indifferentiable from a random oracle" Inaccurate: an extract construction might be proved indiff from an RO under some assumptions on its internal building block. Instead you might say that the proof holds when Extract is modeled as an RO (which is exactly what [HPKEanalysis] says on page 10.)
"whereas X means the given mode satisfies the property" Nitpick: a cross usually means a negative result, as opposed to a check mark. I was confused when first saw that table.
8.2
Some formatting of variables is missing. Think I noticed it at other
places too.
8.4
Any simple way to turn it into a "HYPBPKE"? (hybrid password-based PKE) The obvious one is to derive key pairs via a seed taken from Argon2 but ofc that's limited.
11
Maybe https://eprint.iacr.org/2006/265.pdf could be cited/discussed?