Troubleshooting¶
POD: unable to read certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt¶
error: unable to read certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt for cfc due to open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory
You might have installed your POD in the namespace default or kube-system with automountServiceAccountToken: false. Create a new namespace to fix it.
RBAC: failed to create resource: roles.rbac.authorization.k8s.io¶
If you get an error during the installation of the helm chart which looks like this:
Error: INSTALLATION FAILED: failed to create resource: roles.rbac.authorization.k8s.io "XXX" is forbidden: attempt to grant extra privileges:
You have to edit the helm chart values.yaml and deactivate the creation of RBAC. As an alternative you can create a custom deployment user or use a cluster-admin for deployment.
rbac:
create: false
HTTPD¶
Restarting the HTTPD process with apachectl might cause an error¶
This is a message you can get when using apachectl:
sh-5.1$ apachectl -k graceful
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
The workaround is to use HTTPD:
sh-5.1$ httpd -k graceful
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.128.1.108. Set the 'ServerName' directive globally to suppress this message
Cert Manager¶
The stability of Cert Manager depends on the version used. Here are some tricks and tips.
Problem Routes (stuck process)¶
Remove all items:
NAMESPACE="ibexa"
kubectl get Routes --no-headers=true -n $NAMESPACE |sed -r "s/(\S+)\s+(\S+).*/kubectl delete Route \1 -n ${NAMESPACE}/e"
Problem issuing certificates (stuck process)¶
If you see stuck ordering processes of certificates like this:
$ kubectl get CertificateRequest --all-namespaces
NAME READY AGE
dieding-com-cert-814329627 False 1m
support.xrow.com-tls-4132607287 False 1m
support.xrow.net-tls-534088283 False 1m
www-dieding-com-cert-805434950 False 1m
www-xrow-com-cert-3186473128 False 1m
They can get stuck. You can enforce the renewal by running:
kubectl get Certificate --no-headers=true --all-namespaces | grep False | sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Certificate \2/e'
kubectl get Challenges --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Challenges \2/e'
Remove all temporary items:
kubectl get CertificateRequest --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete CertificateRequest \2/e'
kubectl get Orders --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Orders \2/e'
kubectl get Challenges --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete Challenges \2/e'
Test if a ACME record exists for a host record
[root@localhost helm-ezplatform]# dig CNAME _acme-challenge.xrow.de.
; <<>> DiG 9.16.23-RH <<>> CNAME _acme-challenge.xrow.de.
;_acme-challenge.xrow.de. IN CNAME
;; ANSWER SECTION:
_acme-challenge.xrow.de. 86399 IN CNAME dieding.net.
Show one challenge via the DNS token
kubectl get Challenges --no-headers=true --all-namespaces |sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 describe Challenge\/\2/e' | grep -B 10 -A 30 -i 'gG8JVtAegx69ek67RfTWD'
Show current certificates:
kubectl get Certificate --no-headers=true --all-namespaces
Stop and start cert manager
kubectl scale deployment cert-manager --replicas=0
kubectl scale deployment cert-manager-cainjector --replicas=0
kubectl scale deployment cert-manager-webhook --replicas=0
kubectl scale deployment cert-manager --replicas=1
kubectl scale deployment cert-manager-cainjector --replicas=1
kubectl scale deployment cert-manager-webhook --replicas=1
Clean up directories of an empty host path provisioner¶
This can help you to find unused directories of an storage provisioner.
rm -Rf /root/list.txt
kubectl get pvc --all-namespaces | awk '{print $4}' > /root/list.txt
dirs=$(printf '%s ' *)
for dir in $dirs
do
echo "$dir" | grep --null -vxFf /root/list.txt | xargs rm -rf
done