#Automated Kubernetes Certificate Rotation

This guide helps you install, understand, and operate the Kubernetes Certificate Rotator in to automate the rotation of Kubernetes certificates within your clusters.

#Installation

See Cluster Plugin for installation instructions.

Note:

• Currently supported:

  • On-Premises clusters
  • DCS clusters

#Configuration

The following are the plugin's default certificate settings. The controller uses the configured renewal threshold for each certificate type and checks certificates at the configured interval.

#Certificate settings

With the default settings, certificates are renewed when the remaining validity is less than 20% or 30 days, whichever threshold is reached first. This corresponds to about 292 days for control-plane certificates and about 61 days for kubelet certificates after issuance. Actual timing can vary by one check interval. If you override the deployment values, the configured renewal thresholds take precedence.

#Operation settings

#How it works

This plugin handles automatic rotation for the following certificates.

#Rotation Process

1. Load certificate information

   The initial step involves gathering metadata for all target certificates. Since these certificates are stored in different paths on the host, their contents must be read from the respective files. To achieve this, a temporary Pod is created on the target node with the certificate directories mounted, allowing the Pod to read the information. The certificate's information is collected once per day. Certificate details (paths, expiration) are maintained in the ConfigMap cpaas-system/node-local-certs-<node-name>. The encrypted CA certificate is stored in Secret cpaas-system/kubernetes-ca.

2. Rotation Trigger Condition

   The notBefore and notAfter fields of the certificate indicate the validity period. By default, rotation is triggered when the remaining validity is less than 20% or 30 days, whichever threshold is reached first. The controller applies the configured renewal threshold for each certificate type, so overridden values take precedence. A certificate whose validity period differs from the expected duration is also renewed.

3. Rotation queue

   Certificates requiring rotation are placed in a queue for processing. The rotation program evaluates recent rotation activities and the urgency of pending tasks to decide whether to process them immediately. This prevents potential cluster health issues caused by the simultaneous rotation of multiple certificates.

4. Generate new certificates

   The rotation program generates new certificates based on internally stored CA information. The rotation process creates a temporary Pod on the target node with the necessary certificate directories mounted, allowing for controlled file modifications.

5. Restart the components

   Requiring restart:

   • kube-apiserver: It needs to be restarted to load the new certificates. During restart, it regenerates its internal loopback certificate (valid for one year, used only internally and can not be externally rotated).
   • kube-controller-manager: It needs to be restarted to reload the kubeconfig file.
   • kube-scheduler: It needs to be restarted to reload the kubeconfig file.
   • kubelet: It needs to be restarted to reload the server certificate.

   Restart method: Add annotations to the respective static Pods' YAML files to trigger the kubelet to recreate the Pods. To restart kubelet, mount the host filesystem with hostPID is true and run "systemctl restart kubelet" in the container.

   Auto-reloading:

   • Etcd can auto-reload the certificates.

#Operation Considerations

If kubelet is in an abnormal state during the rotation window and cannot rotate certificates automatically, manual rotation is required:

Operators must manually renew the certificates.

Run the following commands to renew the certificates manually:

cert-renew --ca-cert <ca-cert-path> --ca-key <ca-key-path> --days <days> <certificate or kubeconfig 1> <certificate or kubeconfig 2> ...

For example to renew the kubelet.crt:

cert-renew --ca-cert /etc/kubernetes/pki/ca.crt --ca-key /etc/kubernetes/pki/ca.key --days 91 /etc/kubernetes/pki/kubelet.crt

To download and prepare the cert-renew tool, run:

curl "$(kubectl get services -n cpaas-system frontend -o jsonpath='{.spec.clusterIP}'):8080/cluster-cert-rotator/download/cert-renew" -o ./cert-renew && chmod +x ./cert-renew

Optionally, download renew-all.sh to renew all certificates on the node:

curl "$(kubectl get services -n cpaas-system frontend -o jsonpath='{.spec.clusterIP}'):8080/cluster-cert-rotator/download/renew-all.sh" -o ./renew-all.sh