It should be possible to deduplicate this code using traits.

pub trait Kem {
    /// Note: SEED_BYTES and SHARED_SECRET_BYTES are the same for all ML-KEM parameters
    const PUBLIC_KEY_BYTES: usize;
    const CIPHERTEXT_BYTES: usize;
    type PublicKey;
    type PrivateKey;
}

pub trait KemOps: Kem {
    fn generate() -> (Kem::PublicKey, Kem::PrivateKey) { ... }
    fn encapsulate .. 
    fn decapsulate ...
}

Hmm, I think defining the Kem and KemOps traits are worth it unless we can actually deduplicate between MLKEM768 and 1024. I think we'll either need to live with duplicated code (I'd be OK with this) or figure out something else.

I think the fundamental problem is that the API we're trying to design doesn't quite match the API of BoringSSL.

I think hash.rs in this directory may provide the inspiration we need. There is a struct Hasher that has a constructor (new) that takes in MessageDigest that defines the algorithm in use (SHA-256, SHA-384, etc.). Then the API calls for actually using the hash is the same regardless of which algorithm is used.

I think we could do the same thing for a struct Kem. Its constructor would determine whether we're using 768 or 1024, and it would have a unified API. One wrinkle is that we would need to handle the the public key and private key as byte strings, but I think that's pretty reasonable.

For example:

pub MlKemParam(...);

pub struct MlKem { ... }

impl MlKem {
   pub fn new(p: MlKemParam) -> Result<Self, ErrorStack> { ... }
   pub fn key_gen(&self) -> Result<(Vec<u8>, [SEED_BYTES; u8]), ErrorStack> { ... } // returns public key, private key
   pub fn encapsulate(&self, pk: &[u8]) -> Result<(Vec<u8>, [SHARED_SECRET_BYTES; u8], ErrorStack> { ... } // returns ciphertext
   pub fn decapsulate(&self, sk: &[u8], ciphertext: &[u8]) -> Result<[u8; SHARED_SECRET_BYTES], ErrorStack> { ... }
}

Notes:

1. Elsewhere in this crate crypto operations return ErrorStack instead of a module-specific error type. I think that's probably appropriate here as well.
2. The length of the public key and ciphertext depend on the KEM type, which is why I've handled them as a vector. Unfortunately we can't use an array without adding the length of the array as a generic const. This could work, but would clutter the API unnecessarily. An alternative that we can make alloc-free is have the user pass a &mut [u8] buffer and fail if it's the wrong size.

Oops, I forgot GitHub wants to buffer source review comments, and didn't post my comments yesterday.