fleetserver3435.md

description

When the elastic/beats module is updated from v7.11.2 to v7.17.X elastic-agent encounters a TLS error:

Error: fail to enroll: fail to execute request to fleet-server: remote error: tls: bad certificate

Attempted solutions

1. Copy transport/tlscommon@main from elastic/elastic-agent-libs to beats@7.17 and build a new elastic-agent to test with fleet-server built with beats v7.17.18
2. Copy transport/tlscommon@main and build a new elastic-agent and fleet-server binaries
3. Copy transport/tlscommon@main and build a new elastic-agent and fleet-server binaries, and copy additional lookup logic from agent pr to make sure agent connects to localhost:8221 after installation

current

• elastic-agent changes including tlscommon update and additional debug info here
• fleet-server chagnes using updated beats (on disk) and debug info here

test steps

• build elastic-agent and fleet-server distributables
• copy to VM and unpack, put fleet-server* in the downloads dir of elastic-agent
• Create 7.17.21-SNAPSHOT deployment in qa
• create new policy with only the fleet-server integration

output

$ sudo ./elastic-agent install --url=https://192.168.1.21:8220 \
  --fleet-server-es=https://d549560f2afd4dada7aec461e08f2866.us-central1.gcp.qa.cld.elstc.co:443 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MTMzODAxNzMzMjE6Zmx4N0ZwS21SeDY3QmV4M19OLVRaZw \
  --fleet-server-policy=05370f80-fcec-11ee-a4e8-3142bce5d69f
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
2024-04-17T18:56:43.196Z	INFO	cmd/enroll_cmd.go:390	Generating self-signed certificate for Fleet Server
2024-04-17T18:56:44.969Z	INFO	cmd/enroll_cmd.go:757	Waiting for Elastic Agent to start
2024-04-17T18:56:46.974Z	INFO	cmd/enroll_cmd.go:807	Fleet Server - Starting
2024-04-17T18:56:50.981Z	INFO	cmd/enroll_cmd.go:788	Fleet Server - Running on policy with Fleet Server integration: 05370f80-fcec-11ee-a4e8-3142bce5d69f; missing config fleet.agent.id (expected during bootstrap process), server.ssl: &{Enabled:<nil> VerificationMode:full Versions:[] CipherSuites:[] CAs:[] Certificate:{Certificate:-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgICBnowDQYJKoZIhvcNAQELBQAwLDEWMBQGA1UEChMNZWxh
c3RpYy1mbGVldDESMBAGA1UEAxMJbG9jYWxob3N0MB4XDTI0MDQxNzE4NTY0M1oX
DTM0MDQxNzE4NTY0M1owMzEWMBQGA1UEChMNZWxhc3RpYy1mbGVldDEZMBcGA1UE
AxMQZmxlZXQtc2VydmVyLWRldjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAL9yp+Ff2V3INWni6ohPBRy8XsFge1ZJgsvspujs/EomiDdHC+bM6lvV3pYU
AHGi+V8XY6kKgcAMrkgxUHaWV1/7q9359thl2BoPg76FZ4tXV7jCjtEJbMeU5DOX
TePqpgBpvu1dcOJtwP1a7o4kVFX/Cqxa3CYMJIcmcgLO1Im2GGVKGIyKKJqGZyGR
+vS/qV1u5EXW2HU62XBDfTawvN7p3RozW1CaVjhCPSiil5HGestjUb+Kjr37wQYD
jQWX3UNvktejRL2ZgVO4bNe5BO341y/Hs6aFTug7Fbt6ZglNj1iEUjew8nyiby/s
67Cn1m0guPB5+Yj6S1vbWRfvNqsCAwEAAaNvMG0wDgYDVR0PAQH/BAQDAgeAMB0G
A1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAfBgNVHSMEGDAWgBQPaennOuC8
lBC85qT8wQJCEB0ZJTAbBgNVHREEFDASghBmbGVldC1zZXJ2ZXItZGV2MA0GCSqG
SIb3DQEBCwUAA4IBAQDD50EQ1XCA7CDUSApMrFLW6fh4Yu1MicpoQQ9xQdOUNXTK
F1djKsi52vCmKB+pwJ6NhQR4DxEO02mVloWcef1V3BOcupOU1ytTDr+sHPkN3pBf
CYijKGhC9KpU1V3K2HhOwQbdKACTWfBqjCEYTpe2u4mG/Jtd0y3ggZY0dv0tv7Fw
Ms7xmLd2HYhn5CnPkU4fEZ0IYqQNv3MNHRBWHWqaEI8aO9t3p6sFtbDQIv+zMAM6
c1jkrHw97VU4FAEiei7PmSUQZrMuIjqJ5OIC4GtRtCvg2pB5NAXT9EQY1B877cc7
KoKYn90iV+xwYivF5OcxdJmaM01qQguLDZWilpwK
-----END CERTIFICATE-----
 Key:-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAv3Kn4V/ZXcg1aeLqiE8FHLxewWB7VkmCy+ym6Oz8SiaIN0cL
5szqW9XelhQAcaL5XxdjqQqBwAyuSDFQdpZXX/ur3fn22GXYGg+DvoVni1dXuMKO
0Qlsx5TkM5dN4+qmAGm+7V1w4m3A/VrujiRUVf8KrFrcJgwkhyZyAs7UibYYZUoY
jIoomoZnIZH69L+pXW7kRdbYdTrZcEN9NrC83undGjNbUJpWOEI9KKKXkcZ6y2NR
v4qOvfvBBgONBZfdQ2+S16NEvZmBU7hs17kE7fjXL8ezpoVO6DsVu3pmCU2PWIRS
N7DyfKJvL+zrsKfWbSC48Hn5iPpLW9tZF+82qwIDAQABAoIBAH56IK6jw9ZRDfiG
EKs8iQQS7gaS1uiKZ/6QuTPoXEQgcJ0UMqind6xz/jjnveNU56oKqaxjZJSbzAel
jKLLb63ZdhGPsaQEpF+TIFYdi+piKXHGFtPChjrUAm6S5sJLop9jPN9Te+Aa4gR+
rlr5ZT3q4AI9hHMeUoN2wptYSNEwdtLmyg/eBC+fz24j/zO4XKsUSQbRmlVdhrna
VLkVVmozAhqVORkH9vejGhngfX1Pr0W9qXdxxN6G1/pSplDjoek1iCUcunqcsnQb
LWJR1ALVMFq0xiHAi+5uCGe2r0GjpNDqoNFB1XggtS5eCxW8G5SC3KZs8PrjjJ7L
fU931AECgYEA6FpIbr1FQcWIjThkXdtGdWa/0EglNRjezM2vhSIEBX+ln/o8Mvi8
ALyEjMMtLxS25qiTsbjDg+jIhI2/CVEq4I+Lj33+GYXD9CEUZRQxN6hp+Ex6dj+l
Nc9ACf381UTnkL9eYjcT5qAhyPGsyD+/Ddb2mkuWgygjBCXcWOQlN/sCgYEA0u6j
pPIJLqacmnF7rZwT4RHUbDEAV6LLI2Qh+p1p1XWJmxTaeHvP3ANwYwcTQnhxIty6
J5CMyaYEjsTVolXzl8c9mHqJ7NSNMkcvFxrCs/wXLVDS0QqAd2ylk9ebUyFgqJNh
bvxvaDsqSpOoz1Czg21N980iJDcX8Xw9RgzgTRECgYEA12GJLw3HuzYuZCnX76Cp
aX57Y8jcpKGPNe2cPB7R937huq+tSdqD+3LEs+7gXXooKRwu/Fk283m/FhCG+04A
HvUw+yxhdKsof46CZ0gWb29v+7h49E//Byg8Robn7ZHXzbd2+Dfx/KCHcMuFzII6
yctwAIr81EtKWjf2T3f8gC0CgYAlWdamfal8lkpC2veQnYOzpimcYuzxl8Uq5i9S
Jo+4GdEdIgsxrJL5Ha7IClexvIwkJTvdh642guxpr5c44Ml6xfA1UPWdd2HtVkeT
asyxfGFKR/acx6RKAXYsgNBjXxSzpS1qrs63gUqS3CTXWS4Ahtl4mN2D1jGqNFrR
TbRmUQKBgQC/C/7fbqJydxcFb2nDVEsqWhFPnCILER7HYdl18PhAsEzVJfe6uupq
oyBrkDYacZXJCiiJCrEjQ+xhh65BmRj2ji+QXz4W/nFUOWk579GcFS9R6xNWuDNg
nCvkUubiVOTMeaRj3F7ITQ9nBv+j+I9r/P5nq4bbuYBF5+gDjcaqKA==
-----END RSA PRIVATE KEY-----
 Passphrase: PassphrasePath:} CurveTypes:[] ClientAuth:<nil> CASha256:[]}, host: 0.0.0.0
2024-04-17T18:56:51.697Z	INFO	cmd/enroll_cmd.go:458	Starting enrollment to URL: https://fleet-server-dev:8220/
2024-04-17T18:56:51.808Z	INFO	cmd/enroll_cmd.go:485	Attempting to diagnose tls issues	{"host": "fleet-server-dev:8220"}
2024-04-17T18:56:51.813Z	INFO	cmd/enroll_cmd.go:498	Remote cert found	{"issuer_name": "CN=localhost,O=elastic-fleet", "expiry": "2034-April-17", "common_name": "localhost", "dns_names": ["fleet-server-dev"], "ips": null, "issuer": "MCwxFjAUBgNVBAoTDWVsYXN0aWMtZmxlZXQxEjAQBgNVBAMTCWxvY2FsaG9zdA==", "sig": "w+dBENVwgOwg1EgKTKxS1un4eGLtTInKaEEPcUHTlDV0yhdXYyrIudrwpigfqcCejYUEeA8RDtNplZaFnHn9VdwTnLqTlNcrUw6/rBz5Dd6QXwmIoyhoQvSqVNVdyth4TsEG3SgAk1nwaowhGE6XtruJhvybXdMt4IGWNHb9Lb+xcDLO8Zi3dh2IZ+Qpz5FOHxGdCGKkDb9zDR0QVh1qmhCPGjvbd6erBbWw0CL/szADOnNY5Kx8Pe1VOBQBInouz5klEGazLiI6ieTiAuBrUbQr4NqQeTQF0/REGNQfO+3HOyqCmJ/dIlfscGIrxeTnMXSZmjNNakILiw2VopacCg=="}
2024-04-17T18:56:51.813Z	INFO	cmd/enroll_cmd.go:504	CA info	{"cas": ["-----BEGIN CERTIFICATE-----\nMIIDSzCCAjOgAwIBAgICBnUwDQYJKoZIhvcNAQELBQAwLDEWMBQGA1UEChMNZWxh\nc3RpYy1mbGVldDESMBAGA1UEAxMJbG9jYWxob3N0MB4XDTI0MDQxNzE4NTY0M1oX\nDTM0MDQxNzE4NTY0M1owLDEWMBQGA1UEChMNZWxhc3RpYy1mbGVldDESMBAGA1UE\nAxMJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0C5m\nMcBFTUPNcfGf6k6wY1aDF8QYGxh7rzxCeTebdCQlntHLDvOz62e022m0ApwLn1Gj\nu/sYCoVHulnee6IdwLPqZQWm4XWzjr0/X/uFsbWn4VUMNQW//XkLiZHF3HiuM9b9\nMyyrHSdxNZYpPrhvcLU5ZGaHNrN+XLDGeTjlYgoMvowt478jDFYe3tNrbFB3eM6u\n2T6wtkyY8D4WnqcQVNCz9W3YgZ61cBgL3pL/uK07YU6wG6RwcQ9tmUqGKYF3gFiK\nmaJEXuAGJ1tF34fL7WGuBf4mCWV9eCkOH004wqL3mnED5JMWTGPPM/RuBIZuDOg0\nKZLnTYBQVOLCtNX6lwIDAQABo3cwdTAOBgNVHQ8BAf8EBAMCAoQwHQYDVR0lBBYw\nFAYIKwYBBQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYE\nFA9p6ec64LyUELzmpPzBAkIQHRklMBQGA1UdEQQNMAuCCWxvY2FsaG9zdDANBgkq\nhkiG9w0BAQsFAAOCAQEAA/IUmaHeJCUmUpV30Rmz1nNX35mSXXpQGdPmvQTYZcJV\nH0EPQwXW+esGYemxoPzMhT2RayYVtRJgFq7E9VlYOAdv3PdegNoiWoBbtA441vTj\nO1Lit/3hC6KLeGSuGLNo3baedxJuk3RVmkqBJm6LHDqDI58ozO6l0s9yczdaTKDh\nvqWiMBupesQdSNlrEhCpAvPbcu5r3WWfEDsaLPAvAJSZrIbzWaMTxJcy4KQYpDuf\n5fkMEgD8BNHBKqdbHTSxSGT4oTrlqtezibGnZ1BnQ25Ot+pxW1E0bD+NH1BFZ2Dl\n1XAvawstapfLwwHZie6rLOLhnaGpjPGeg3e07xA4lA==\n-----END CERTIFICATE-----\n"]}
Error: fail to enroll: fail to execute request to fleet-server: remote error: tls: bad certificate

Note

beats#22495 changes cert verification to require each cert to have at least one SAN entry, but this may not be the issue as the fleet-server reports to have a cert with dns fleet-server-dev which is the name of the vargant box

cert verification

Copying the cert, key, and ca to tmp we can see that the CA signs the cert:

$openssl verify -verbose -CAfile /tmp/ca.crt /tmp/cert.crt
/tmp/cert.crt: OK

specifying cert paths

sudo ./elastic-agent install \
  --url=https://fleet-server-dev:8220 \
  --fleet-server-es=https://d549560f2afd4dada7aec461e08f2866.us-central1.gcp.qa.cld.elstc.co:443 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MTMzODAxNzMzMjE6Zmx4N0ZwS21SeDY3QmV4M19OLVRaZw \
  --fleet-server-policy=05370f80-fcec-11ee-a4e8-3142bce5d69f \
  --fleet-server-cert=/tmp/cert.crt \
  --fleet-server-cert-key=/tmp/key.crt \
  --certificate-authorities=/tmp/ca.crt

has the same TLS issues, adding --insecure, or replacing --certificate-authorities with --insecure also has TLS issues

v7.17.21 with paths

building an elastic-agent froom the 7.17 branch without any changes and specifing the certs/ca above results in a successful install

agent change only

building an elastic-agent (from the do-not-backport branch) without the fleet-server changes, and specifing the certs results in a success The issue is with the fleet-server; elastic-agent bootstrapping differences are not a cause

update fleet-server beats to 7.12.0

Trying to update the fleet-server beats import to 7.12.0 resulted in the error occuing. Version contains the update to the golang version and other improvements v7.12.0 was the release after v7.11.2 and is where the error was introduced