We are currently using openssl-1.1.1l and try to have custom tls1.3 extensions in the certificate messages send from the server as well as the client.
According to the specification:

Extensions in the Certificate message from the server MUST
correspond to ones from the ClientHello message. Extensions in
the Certificate message from the client MUST correspond to
extensions in the CertificateRequest message from the server.

So we tried registering the custom extensions on the client as well as the server side like this:

SSL_CTX_add_custom_ext(
   ctx,
   456,
   SSL_EXT_TLS1_3_CERTIFICATE | SSL_EXT_CLIENT_HELLO | SSL_EXT_TLS1_3_CERTIFICATE_REQUEST,
   callbackAddExtension,
   callbackFreeExtension,
   nullptr,
   callbackParseExtension,
   nullptr
);

That way we can have our custom extension in the client hello so that the server can respond to that extension in its certificate message. Furthermore the server can add the extension to its certificate request so that the client can respond to the extension in its certificate message.

In the servers addExtensionCallback we add our extension data to the certificate message and append a placeholder extension to the certificate request like this:

if (extType == 456) {
	// if we are sending our certificate add our extension data
	if (context == SSL_EXT_TLS1_3_CERTIFICATE) {
		const char* customData = "Test Extension Data";
		
		*out = new unsigned char[strlen(customData)];
		memcpy((void*)*out, &customData, strlen(customData));
		*outlen = strlen(customData);
	}

	// if we are in the certificate request context, add our extension placeholder
	if (context == SSL_EXT_TLS1_3_CERTIFICATE_REQUEST) {
		*out = new unsigned char[1];
		*outlen = 1;
	}
}

On the clients side in the addExtensionCallback we do basically the same. However this time we add a placeholder extension to the client hello like this.

if (extType == 456) {
	if (context == SSL_EXT_TLS1_3_CERTIFICATE) {
		const char* customData = "Test Extension Data";
	
		*out = new unsigned char[strlen(customData)];
		memcpy((void*)*out, &customData, strlen(customData));
		*outlen = strlen(customData);
	}
	
	if (context == SSL_EXT_CLIENT_HELLO) {
		*out = new unsigned char[1];
		*outlen = 1;
	}
}

The server is correctly setting the extension data in the certificate message. The client however doesnt. The clients addExtensionCallback is never called for the SSL_EXT_TLS1_3_CERTIFICATE context although we saw that the placeholder extension was indeed correctly set in the servers SSL_EXT_TLS1_3_CERTIFICATE_REQUEST.
We also force client authentication SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, nullptr); and we also checked that the certificate sent from the client was the expected one.

From our opinion this should be the correct way to do it but maybe there is still a logical problem. If needed we can also provide a minimal runnable example of the server and the clients code.

Maybe someone got an idea whats going wrong here.
Thanks in advance.