Problem.txt

Currently getting the following stacktrace from Zookeeper. 

The script below is also used to provision BCFKS keystores with Kafka without any issues, so I'm curious what might be making Zookeeper error out on me.

ZK libs contains the following BC-FIPS JARs:
libs/bc-fips-1.0.2.jar  libs/bcpkix-fips-1.0.5.jar  libs/bctls-fips-1.0.12.2.jar

Running Zookeeper 3.63:

```
Jan 08, 2022 4:48:44 AM org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi engineInit
WARNING: Skipped default trust store
java.io.IOException: DER length more than 4 bytes: 109
        at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
        at org.bouncycastle.asn1.ASN1InputStream.readLength(Unknown Source)
        at org.bouncycastle.asn1.ASN1InputStream.readObject(Unknown Source)
        at org.bouncycastle.jcajce.provider.ProvBCFKS$BCFIPSKeyStoreSpi.engineLoad(Unknown Source)
        at java.base/java.security.KeyStore.load(KeyStore.java:1479)
        at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.getDefaultTrustStore(ProvTrustManagerFactorySpi.java:112)
        at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.engineInit(ProvTrustManagerFactorySpi.java:162)
        at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:278)
        
        
Jan 08, 2022 4:48:44 AM org.bouncycastle.jsse.provider.ProvSSLContextSpi selectX509TrustManager
WARNING: Failed to load default trust managers
java.security.KeyStoreException: Failed to load default trust store
        at org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi.engineInit(ProvTrustManagerFactorySpi.java:182)
        at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:278)
        at org.bouncycastle.jsse.provider.ProvSSLContextSpi.selectX509TrustManager(ProvSSLContextSpi.java:911)
        at org.bouncycastle.jsse.provider.ProvSSLContextSpi.engineInit(ProvSSLContextSpi.java:865)
        at java.base/javax.net.ssl.SSLContext.init(SSLContext.java:297)
        at io.netty.handler.ssl.JdkSslContext.<clinit>(JdkSslContext.java:75)
        at org.apache.zookeeper.common.SSLContextAndOptions.createNettyJdkSslContext(SSLContextAndOptions.java:105)
        at org.apache.zookeeper.server.NettyServerCnxnFactory.initSSL(NettyServerCnxnFactory.java:546)
        at org.apache.zookeeper.server.NettyServerCnxnFactory.access$1300(NettyServerCnxnFactory.java:80)
        at org.apache.zookeeper.server.NettyServerCnxnFactory$1.initChannel(NettyServerCnxnFactory.java:530)
        at org.apache.zookeeper.server.NettyServerCnxnFactory$1.initChannel(NettyServerCnxnFactory.java:522)
```


java.security file settings:

```
ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX

security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
security.provider.3=SUN


#
# Default keystore type.
#
keystore.type=BCFKS
```

zoo.cfg:
```
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080
sslQuorum=true
secureClientPort=2281
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
ssl.hostnameVerification=false

ssl.trustStore.location=/home/ubuntu/ubuntu18-kafka-fips/certs/zookeeper.truststore.bcfks
ssl.trustStore.password=testfips
ssl.trustStore.type=BCFKS
ssl.keyStore.location=/home/ubuntu/ubuntu18-kafka-fips/certs/zookeeper.keystore.bcfks
ssl.keyStore.password=testfips

ssl.quorum.hostnameVerification=false
ssl.quorum.keyStore.location=/home/ubuntu/ubuntu18-kafka-fips/certs/zookeeper.keystore.bckfs
ssl.quorum.keyStore.password=testfips
ssl.quorum.trustStore.location=/home/ubuntu/ubuntu18-kafka-fips/certs/zookeeper.truststore.bckfs
ssl.quorum.trustStore.password=testfips
```

zk-client-cfg:
```
zookeeper.ssl.client.enable=true
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
zookeeper.ssl.keystore.location=/home/ubuntu/ubuntu18-kafka-fips/certs/zookeeper.keystore.bcfks
zookeeper.ssl.keystore.password=testfips
zookeeper.ssl.truststore.location=/home/ubuntu/ubuntu18-kafka-fips/certs/zookeeper.truststore.bcfks
zookeeper.ssl.truststore.password=testfips
zookeeper.ssl.endpoint.identification.algorithm=
```

provision-bcfks-keystores.sh

#!/bin/bash -ex

FIPS_JAR="/home/ubuntu/bc-fips-1.0.2.jar"
FIPS_PROVIDER_CLASS="org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider"
KEYSTORE_PASS="testfips"
KEY_PASS="testfips"
CN_ENTITY="chen-anders"

keytool -keystore zookeeper.keystore.bcfks -alias localhost -validity 720 -genkeypair -keyalg RSA \
    -keysize 2048 -storepass "$KEYSTORE_PASS" -keypass "$KEY_PASS" -storetype BCFKS \
    -providerpath "$FIPS_JAR" -providerclass "$FIPS_PROVIDER_CLASS" \
    -dname "CN=${CN_ENTITY}"

openssl req -new -x509 -keyout ca-key -out ca-cert -days 720 -nodes -subj "/CN=${CN_ENTITY}/"

keytool -keystore zookeeper.truststore.bcfks -storetype BCFKS -alias CARoot -import \
    -file ca-cert  -storepass "$KEYSTORE_PASS" -keypass "$KEY_PASS" -providerpath "$FIPS_JAR" -providerclass "$FIPS_PROVIDER_CLASS" -noprompt

# Export the keystore certificate so it can be signed by the root CA.
keytool -keystore zookeeper.keystore.bcfks -alias localhost -storepass "$KEYSTORE_PASS" -keypass "$KEY_PASS" -storetype BCFKS \
    -certreq -file cert-file -providerpath "$FIPS_JAR" -providerclass "$FIPS_PROVIDER_CLASS" -noprompt

# Sign the keystore certificate using the root CA.
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 720 -CAcreateserial -passin "pass:$KEY_PASS"

# Import the root CA into the keystore.
keytool -keystore zookeeper.keystore.bcfks -storetype BCFKS -alias CARoot -import -file ca-cert \
    -storepass "$KEYSTORE_PASS" -providerpath "$FIPS_JAR" -providerclass "$FIPS_PROVIDER_CLASS" -noprompt

# Create certficate chain consisting of the CA and signed cert
cat ca-cert cert-signed > cert_chain

# Import the cert chain into the keystore
keytool -keystore zookeeper.keystore.bcfks -storetype BCFKS -storepass "$KEYSTORE_PASS" -keypass "$KEY_PASS" -alias localhost \
    -import -file cert_chain -providerpath "$FIPS_JAR" -providerclass "$FIPS_PROVIDER_CLASS" -noprompt