How to replace expired or soon to expire CA in Fleet (Elastic Agent & Fleet Server)

[lucabelluccini]

We do not provide guidance on how to approach a CA certificates replacement in Fleet Server and/or Elasticsearch with Fleet-managed Elastic Agents.

The CA used by Elastic Agent to trust the Fleet Server cannot be provided in the policy. It is only available as a command line parameter and it points to a local file.

• Do we support multiple CAs in Elastic Agents?
• Can we hot-swap the CA (is it reloaded by Elastic Agent) or it is reloaded only at startup?

Ideally, for updating the CA in Fleet Server without downtime:

• All the Elastic Agents enrolled to the Fleet Server should be updated to trust both the OLD CA and NEW CA. How?
• Fleet Server can be restarted, replacing the Fleet Server certificates and CA. How?

Ideally, for updating the CA in Elasticsearch without downtime:

• All the Elastic Agents enrolled to the Fleet Server should be updated to trust both the OLD CA and NEW CA. Elasticsearch certs are typically defined in the Fleet UI / Output settings (via reference to a file or embedded in the policy). How?
• Fleet Server should be also configured to trust both the OLD CA and NEW CA of Elasticsearch. How?
• Elasticsearch should be roll-restarted to update their CA. How to do it it is not in the scope of the guide.

[leandrojmp]

Hello, is there any update on this?

We have a requirement to renew the certificates of our Fleet Server that is close to expiration date but I could not find any documentation about this.

[lucabelluccini]

Hello @leandrojmp, attempting to give some suggestions.

If you have a subscription, we recommend raising a support case.

This issue is to track the missing docs (and possibly a product gap).

The comments below relate to Elastic Agent <> Fleet Server connections, not to Elastic Agent <> ES or Fleet Server <> ES.

On Fleet Server, as soon as you're using the same root CA, you can just generate a new certificate from the same root CA with a newer expiration date. As long as Elastic Agents trust the root CA, it should be ok.

If instead it's the root CA expiring, then it would likely mean you will need to:

• Replace the Fleet server certificates on Fleet Server(s)
• Elastic Agents need to be re-enrolled to the Fleet Server with the new CA. Note it might trigger a reset of the registries and internal states of the Elastic Agent on which you're re-enrolling. The re-enrollment cannot be done via Fleet UI.

FYI @nimarezainia / @amitkanfer as this might highlight not only a doc gap but a product gap (ability to update certificates across the different components of Fleet & EA without downtimes / manual operations)

[leandrojmp]

Hello @lucabelluccini,

Thanks, we already opened a ticket 6 months ago reporting the lack of the documentation about renewing certificates.

At the time we needed to change the CA as well, but this time we fortunately only need to replace the certificate since they are signed by a Know CA.

I opened another ticket today because there isn't yeat any documentation about how to update the certificate before expiring.

Also, it looks to me that this is a product gap as well, having to re-enroll an agent to update the CA is way far from ideal and can be a huge impact for some users.

In my case if I needed to re-enroll thousands of hosts it could take months because of some internal process.

With certificates expiring every year and with the recommendation to expire them every 90 days, the Agent needs to be able to update the certificates and CA more easily.

[kilfoyle]

@jlind23 I can help with docs on this issue but I'd really need a draft, demo, or some sort of guidance from developers. Is there anyone who can provide that info?

[lucabelluccini]

Hello @leandrojmp - thanks for the additional info.

[leandrojmp]

Hello @lucabelluccini,

If it helps, this seems to be the internal issue for an enhancement request that we made last year: https://github.com/elastic/enhancements/issues/20562

[jlind23]

@AndersonQ as you recently worked on the mTLS issue, would you be able to assist here?
cc @pierrehilbert

[nimarezainia]

Team, can we keep this as the documentation issue. Which is merely to say that we would need a new enrollment with the new certs. This is unfortunately a gap we currently have. Will create another issue for an actual fix which will need agent+fleet changes.

@kilfoyle which section would this ideally go into?

[AndersonQ]

Hi, let me try to answer the questions:

Do we support multiple CAs in Elastic Agents?

Yes, the agent can receive a list of CAs

Can we hot-swap the CA (is it reloaded by Elastic Agent) or it is reloaded only at startup?

It's only loaded when the agent starts.

All the Elastic Agents enrolled to the Fleet Server should be updated to trust both the OLD
CA and NEW CA. How?

It's technically possible, but right now defining and loading certificates from the policy is being done for mTLS, which does not work on cloud. tl;dr: some development is needed, but with elastic/elastic-agent#2247 and the mTLS as a whole, most of the work should be done. Then it's be just to add both CAs in the policy and the agent would use them both to verify the certificate presented by fleet-server.

Fleet Server can be restarted, replacing the Fleet Server certificates and CA. How?

If the certificates are defined as a path, it should be possible to replace the contents of the file and fleet server should pick up the new ones. As far as I remember it isn't possible to load multiple certificates, only multiple CAs.

I'm not sure, but I believe the agent does not support receiving a directory for CAs or certificates. If it was supported, the contents of the directory could be changed, adding or removing certificates and CAs, and on restart the agent would pick up the updated certificates/CAs

[kilfoyle]

Thanks @AndersonQ! This helps a lot. Once the mTLS work lands it seems the process will be a lot simpler.

@nimarezainia We have this Secure connections section in the docs, but since that section is only about on-prem setups I would lean toward putting this content under "Manage Elastic Agents in Fleet" -> "Elastic Agents".

[nimarezainia]

I'm not sure, but I believe the agent does not support receiving a directory for CAs or certificates. If it was supported, the contents of the directory could be changed, adding or removing certificates and CAs, and on restart the agent would pick up the updated certificates/CAs

I'm fairly certain that this is supported but am struggling to find reference to it in docs/issues :-( We had the same concern for Logstash and trying to find a way to send each agent individual certificates. I agree that it may not be valid for CA.

[nimarezainia]

@AndersonQ not what I was looking for but in our docs HERE you can the see examples where CA and Cert can be loaded from a directory. I guess the question is whether agent/fleet-server read from this path on every new handshake (or do they rely on what's in memory):

sudo elastic-agent install --url=https://192.0.2.1:8220
--enrollment-token=
--certificate-authorities=/path/to/ca.crt

sudo ./elastic-agent install
--url=https://192.0.2.1:8220
--fleet-server-es=https://192.0.2.0:9200
--fleet-server-service-token=AAEBAWVsYXm0aWMvZmxlZXQtc2XydmVyL3Rva2VuLTE2MjM4OTAztDU1OTQ6dllfVW1mYnFTVjJwTC2ZQ0EtVnVZQQ
--fleet-server-es-ca=/path/to/elasticsearch-ca.crt
--certificate-authorities=/path/to/ca.crt
--fleet-server-cert=/path/to/fleet-server.crt
--fleet-server-cert-key=/path/to/fleet-server.key
--fleet-server-port=8220

[nimarezainia]

When you install an on-prem fleet server:

as @cmacknz mentioned in the other issue we don't expose these tls configs in the UI however the path to CA and Cert can be configured for the fleet server. Queston remains when these get read.