Difference between revisions of "How to renew the Puppet CA certificate"

From Wiki
Jump to: navigation, search
(Created page with "Many thanks to https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place for these instructions. How to renew Puppet CA and server certi...")
 
Line 2: Line 2:
  
 
How to renew Puppet CA and server certificates in place
 
How to renew Puppet CA and server certificates in place
September 1, 2017PlatformPuppet, technologyChristian Kauhaus
 
It used to run fine for years… but now the (deprecated) Puppet infrastructure at the Flying Circus is showing signs of aging. It’s not about server hardware or something like this (fully virtualized anyway) – it’s about SSL certificates of Puppet’s own SSL infrastructure. Time for a face lift.
 
  
In the following, I will describe what we did to renew both CA and Puppet server certificates. Despite that this problem should occur on every Puppet server running for a prolonged amount of time, I found remarkably few resources on the net (that did not involve completely replacing the CA) – so I’m going to share our findings.
+
If you see a message like this when running the puppet agent it is time to renew your certs.
  
 +
Certificate 'Puppet CA: server1.fcio.net' will expire on 2017-10-01T11:09:06UTC
 +
Certificate 'puppet.fcio.net' will expire on 2017-10-03T11:13:58UTC
  
Act before it’s too late
+
The CA certificate must be renewed *before* it expires or else you will need to clean and resign *all* of your client node certificates along with the CA cert.
It is important to pay immediate attention to messages like:
+
  
Certificate 'Puppet CA: server1.fcio.net' will expire on 2017-10-01T11:09:06UTC
+
To renew the existing CA cert follow these steps. *BACK UP* your CA data before doing this.
Certificate 'puppet.fcio.net' will expire on 2017-10-03T11:13:58UTC
+
As long as the certificates are only soon to expire, it is possible to roll out a new CA certificate using Puppet itself. Later on, the only rescue will be to regenerate all certificates and install them manually on each client.
+
  
But since the certificates are still valid albeit soon-to-be-expired, we have a better plan. We generate new certificates with extended expiration times based on the original keys. All certificates stay compatible to each other — a prerequisite for a slick roll over.
+
Recreate missing CSRs:
  
Fasten seat belts and make sure to back up the SSL directory before proceeding.
+
A missing CSR can be generated from an existing certificate. To regenerate the CA's CSR run this command:
  
Reverse-generate missing signing requests
+
cd /etc/puppetlabs/puppet/ssl/ca
Before getting into business, we have to check if there are certificate signing requests (CSR) available. CSRs should be found unter /var/lib/puppet/ssl/ca/ca_csr.pem and /var/lib/puppet/ssl/certificate_requests/puppet.fcio.net.pem. Ours seem to have got lost over time. No problem! A missing CSR can be generated from an existing certificate. Here is how we did it for the CA certificate:
+
openssl x509 -x509toreq -in ca_crt.pem -signkey ca_key.pem -out ca_csr.pem
  
cd /var/lib/puppet/ssl/ca
+
The procedure for generating the puppet *server* certificate is similar however you will need to follow these steps to ensure the proper SANs are included.
openssl x509 -x509toreq -in ca_crt.pem -signkey ca_key.pem \
+
  -out ca_csr.pem
+
The same procedure applies to recreating a missing CSR for a Puppet server certificate in certs/puppet.fcio.net.pem. I assume the the Puppet server’s private key lives in private_keys/puppet.fcio.net.pem and the CSR goes into certificate_requests/puppet.fcio.net.pem.  
+
  
 
Note, that those requests aren’t perfect: the certificate CA extensions are missing, but we’ll fill them in during the next step.
 
Note, that those requests aren’t perfect: the certificate CA extensions are missing, but we’ll fill them in during the next step.
  
Certificate renewal
+
Sign your new CSRs:
 +
 
 
Next, we generate new certificates from the existing private key and the signing request. To get the necessary X509v3 extensions into the CA certificate, we first create a suitable OpenSSL config file snippet:
 
Next, we generate new certificates from the existing private key and the signing request. To get the necessary X509v3 extensions into the CA certificate, we first create a suitable OpenSSL config file snippet:
  
 +
<pre>
 
cat > extension.cnf <<_EOT_
 
cat > extension.cnf <<_EOT_
 
[CA_extensions]
 
[CA_extensions]
Line 39: Line 35:
 
subjectKeyIdentifier = hash
 
subjectKeyIdentifier = hash
 
_EOT_
 
_EOT_
With the extensions config in place, we generate the new certificate:
+
</pre>
  
openssl x509 -req -days 3650 -in ca_csr.pem -signkey ca_key.pem \
+
Sign your CSR using this configuration.
  -out ca_crt.pem -extfile extension.cnf -extensions CA_extensions
+
We can now use the CA certificate to sign a new Puppet server certificate:
+
  
cd /var/lib/puppet/ssl
+
cp ca_crt.pem ca_crt.pem.backup
openssl x509 -req -days 3650 -in certificate_requests/puppet.fcio.net.pem -CA ca/ca_crt.pem \
+
openssl x509 -req -days 3650 -in ca_csr.pem -signkey ca_key.pem -out ca_crt.pem -extfile extension.cnf -extensions CA_extensions
  -CAkey ca/ca_key.pem -CAserial ca/serial -out certs/puppet.fcio.net.pem
+
 
Of course, the new certificates do nothing unless they are actually used. In our case we had to restart the Puppet server so they got picked up.
+
Now use the new CA cert to regenerate the puppet master's certificate. 
 +
 
 +
cp /opt/puppetlabs/puppet/ssl/openssl.cnf /tmp/openssl.cnf
 +
export sans="[SAN]\nsubjectAltName=DNS:puppet.example.com,DNS:puppet\n"
 +
printf $sans >> /tmp/openssl.cnf
 +
openssl req -new -sha256 -key ../private_keys/puppetmaster.example.com.pem -reqexts SAN -config /tmp/openssl.cnf -out requests/puppetmaster.example.com.pem
 +
openssl x509 -req -days 3650 -in requests/puppetmaster.example.com.pem -CA ca_crt.pem -CAkey ca_key.pem -CAserial serial -out signed/puppetmaster.example.com.pem
 +
 
 +
After the cert has been signed restart puppetserver.
 +
 
 +
systemctl restart puppetserver
  
 
Distribute the CA certificate
 
Distribute the CA certificate
 +
 
Puppet clients need the CA certificate to be locally available so that they can verify other certificates against it. We copy the newly generated ca_crt.pem into some Puppet module and let Puppet place it on all clients:
 
Puppet clients need the CA certificate to be locally available so that they can verify other certificates against it. We copy the newly generated ca_crt.pem into some Puppet module and let Puppet place it on all clients:
  
Line 58: Line 63:
 
   group => 'puppet',  
 
   group => 'puppet',  
 
  }
 
  }
 +
 
That’s it! No more warnings, everyone happy. 🙂
 
That’s it! No more warnings, everyone happy. 🙂

Revision as of 16:18, 16 August 2019

Many thanks to https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place for these instructions.

How to renew Puppet CA and server certificates in place

If you see a message like this when running the puppet agent it is time to renew your certs.

Certificate 'Puppet CA: server1.fcio.net' will expire on 2017-10-01T11:09:06UTC
Certificate 'puppet.fcio.net' will expire on 2017-10-03T11:13:58UTC

The CA certificate must be renewed *before* it expires or else you will need to clean and resign *all* of your client node certificates along with the CA cert.

To renew the existing CA cert follow these steps. *BACK UP* your CA data before doing this.

Recreate missing CSRs:

A missing CSR can be generated from an existing certificate. To regenerate the CA's CSR run this command:

cd /etc/puppetlabs/puppet/ssl/ca
openssl x509 -x509toreq -in ca_crt.pem -signkey ca_key.pem -out ca_csr.pem

The procedure for generating the puppet *server* certificate is similar however you will need to follow these steps to ensure the proper SANs are included.

Note, that those requests aren’t perfect: the certificate CA extensions are missing, but we’ll fill them in during the next step.

Sign your new CSRs:

Next, we generate new certificates from the existing private key and the signing request. To get the necessary X509v3 extensions into the CA certificate, we first create a suitable OpenSSL config file snippet:

cat > extension.cnf <<_EOT_
[CA_extensions]
basicConstraints = critical,CA:TRUE
nsComment = "Puppet Ruby/OpenSSL Internal Certificate"
keyUsage = critical,keyCertSign,cRLSign
subjectKeyIdentifier = hash
_EOT_

Sign your CSR using this configuration.

cp ca_crt.pem ca_crt.pem.backup
openssl x509 -req -days 3650 -in ca_csr.pem -signkey ca_key.pem -out ca_crt.pem -extfile extension.cnf -extensions CA_extensions

Now use the new CA cert to regenerate the puppet master's certificate.

cp /opt/puppetlabs/puppet/ssl/openssl.cnf /tmp/openssl.cnf
export sans="[SAN]\nsubjectAltName=DNS:puppet.example.com,DNS:puppet\n"
printf $sans >> /tmp/openssl.cnf
openssl req -new -sha256 -key ../private_keys/puppetmaster.example.com.pem -reqexts SAN -config /tmp/openssl.cnf -out requests/puppetmaster.example.com.pem
openssl x509 -req -days 3650 -in requests/puppetmaster.example.com.pem -CA ca_crt.pem -CAkey ca_key.pem -CAserial serial -out signed/puppetmaster.example.com.pem

After the cert has been signed restart puppetserver.

systemctl restart puppetserver

Distribute the CA certificate

Puppet clients need the CA certificate to be locally available so that they can verify other certificates against it. We copy the newly generated ca_crt.pem into some Puppet module and let Puppet place it on all clients:

file { '/var/lib/puppet/ssl/certs/ca.pem': 
  source => 'puppet:///path/to/ca_crt.pem', 
  owner => 'puppet', 
  group => 'puppet', 
}

That’s it! No more warnings, everyone happy. 🙂