Skip to content

Troubleshooting

POD: unable to read certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

error: unable to read certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt for cfc due to open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory

You might have installed your POD in the namespace default or kube-system with automountServiceAccountToken: false. Create a new namespace to fix it.

RBAC: failed to create resource: roles.rbac.authorization.k8s.io

If you get an error during the installation of the helm chart which looks like this:

Error: INSTALLATION FAILED: failed to create resource: roles.rbac.authorization.k8s.io "XXX" is forbidden: attempt to grant extra privileges:

You have to edit the helm chart values.yaml and deactivate the creation of RBAC. As an alternative you can create a custom deployment user or use a cluster-admin for deployment.

rbac:
  create: false

HTTPD

Restarting the HTTPD process with apachectl might cause an error

This is a message you can get when using apachectl:

sh-5.1$ apachectl -k graceful
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down

The workaround is to use HTTPD:

sh-5.1$ httpd -k graceful
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.128.1.108. Set the 'ServerName' directive globally to suppress this message

Cert Manager

The stability of Cert Manager depends on the version used. Here are some tricks and tips.

Problem Routes (stuck process):

Remove all items:

NAMESPACE="ibexa"
kubectl get Routes --no-headers=true -n $NAMESPACE |sed -r "s/(\S+)\s+(\S+).*/kubectl delete Route \1 -n ${NAMESPACE}/e"

Problem issuing certificates (stuck process):

If you see stuck ordering processes of certificates like this:

[root@False xrow]# kubectl get CertificateRequest --all-namespaces                                
NAME                              READY     AGE
dieding-com-cert-814329627        False     1m
support.xrow.com-tls-4132607287   False     1m
support.xrow.net-tls-534088283    False     1m
www-dieding-com-cert-805434950    False     1m
www-xrow-com-cert-3186473128      False     1m

You can enforce the renewal by running (They can get stuck.):

kubectl get Certificate --no-headers=true --all-namespaces | grep False | sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Certificate \2/e'
kubectl get Challenges --no-headers=true --all-namespaces  |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Challenges \2/e'

Remove all temporary items:

kubectl get CertificateRequest --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete CertificateRequest \2/e'
kubectl get Orders --no-headers=true --all-namespaces  |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Orders \2/e'
kubectl get Challenges --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Challenges \2/e'

Test if a ACME record exists for a host record

[root@localhost helm-ezplatform]# dig CNAME _acme-challenge.xrow.de.

; <<>> DiG 9.16.23-RH <<>> CNAME _acme-challenge.xrow.de.
;_acme-challenge.xrow.de.       IN      CNAME
;; ANSWER SECTION:
_acme-challenge.xrow.de. 86399  IN      CNAME   dieding.net.

Show one challenge via the DNS token

kubectl get Challenges --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 describe Challenge\/\2/e' | grep -B 10 -A 30 -i 'gG8JVtAegx69ek67RfTWD'

Show current certificates:

kubectl get Certificate --no-headers=true --all-namespaces

Stop and start cert manager

kubectl scale deployment cert-manager --replicas=0
kubectl scale deployment cert-manager-cainjector --replicas=0
kubectl scale deployment cert-manager-webhook --replicas=0

kubectl scale deployment cert-manager --replicas=1
kubectl scale deployment cert-manager-cainjector --replicas=1
kubectl scale deployment cert-manager-webhook --replicas=1

Clean up directories of an empty host path provisioner

This can help you to find unused directories of an storage provisioner.

rm -Rf /root/list.txt
kubectl get pvc --all-namespaces | awk '{print $4}' > /root/list.txt
dirs=$(printf '%s ' *)
for dir in $dirs
do
  echo "$dir" | grep --null -vxFf /root/list.txt | xargs rm -rf
done