MySQL Pod keeps restarting in k3s and RHEL9

Hi everyone,

I am working through the procedures for installing the Early Access version of Volt MX Go and I'm stuck with MySQL.

I set up a new virtual machine running RHEL9.1 and followed the instructions on preparing the system (incl. configuration of firewalld). k3s is running, helm was installed and Domino Keep is already running, but the MySQL pod makes some problems.

[fgbas@volt-mx-go ~]$ kubectl get pods
NAME                           READY   STATUS             RESTARTS         AGE
domino-keep-55fc8945b8-2mckp   3/3     Running            0                138m
mysql-0                        0/1     CrashLoopBackOff   30 (2m49s ago)   134m

The MySQL pod keeps restarting a few times, then goes to status "CrashLoopBackOff" and restarts again. Kubectl logs and kubectl describe pod shows the following messages:

[fgbas@volt-mx-go ~]$ kubectl logs mysql-0
mysql 12:45:12.62
mysql 12:45:12.62 Welcome to the Bitnami mysql container
mysql 12:45:12.62 Subscribe to project updates by watching https://github.com/bitnami/containers
mysql 12:45:12.62 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mysql 12:45:12.62
mysql 12:45:12.62 INFO  ==> ** Starting MySQL setup **
mysql 12:45:12.63 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 12:45:12.64 INFO  ==> Initializing mysql database
mysql 12:45:12.65 WARN  ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mysql 12:45:12.65 INFO  ==> Using persisted data
mysql 12:45:12.67 INFO  ==> Running mysql_upgrade
mysql 12:45:12.68 INFO  ==> Starting mysql in background
[fgbas@volt-mx-go ~]$ kubectl describe pod mysql-0
Name:             mysql-0
Namespace:        mxgo
Priority:         0
Service Account:  mysql
Node:             volt-mx-go/141.26.157.124
Start Time:       Fri, 05 May 2023 06:26:10 -0400
Labels:           app.kubernetes.io/component=primary
                  app.kubernetes.io/instance=mysql
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=mysql
                  controller-revision-hash=mysql-59d6db49cd
                  helm.sh/chart=mysql-9.6.0
                  statefulset.kubernetes.io/pod-name=mysql-0
Annotations:      checksum/configuration: f9e2ae2a82f163646a72409e482f2359843399b2ff618f4a421f01d863eb4f6c
Status:           Running
IP:               10.42.0.81
IPs:
  IP:           10.42.0.81
Controlled By:  StatefulSet/mysql
Containers:
  mysql:
    Container ID:   containerd://3d39f8fc2a38d749cc10db33ef39e9d38ccc546c9722f60845afdd5c2d023d60
    Image:          docker.io/bitnami/mysql:8.0.32-debian-11-r17
    Image ID:       docker.io/bitnami/mysql@sha256:44e6480abc8e21470bf15922030f6da6c605102b085639088b85831a8177c17a
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 05 May 2023 08:54:53 -0400
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Fri, 05 May 2023 08:52:36 -0400
      Finished:     Fri, 05 May 2023 08:54:51 -0400
    Ready:          False
    Restart Count:  34
    Liveness:       exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
    Startup:  exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=15s timeout=1s period=10s #success=1 #failure=10
    Environment:
      BITNAMI_DEBUG:        false
      MYSQL_ROOT_PASSWORD:  <set to the key 'mysql-root-password' in secret 'mysql'>  Optional: false
      MYSQL_USER:           dbclient
      MYSQL_PASSWORD:       <set to the key 'mysql-password' in secret 'mysql'>  Optional: false
    Mounts:
      /bitnami/mysql from data (rw)
      /opt/bitnami/mysql/conf/my.cnf from config (rw,path="my.cnf")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4pgm7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-mysql-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mysql
    Optional:  false
  kube-api-access-4pgm7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Normal   Created    58m (x23 over 148m)     kubelet  Created container mysql
  Warning  Unhealthy  8m44s (x326 over 148m)  kubelet  Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!
  Warning  BackOff  3m52s (x332 over 132m)  kubelet  Back-off restarting failed container mysql in pod mysql-0_mxgo(6c9e2952-a328-4310-a55f-af2ab8b4099c)

I don't have any clue, why MySQL pod prints "Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock'". I followed the instructions with all of the helm commands.

Does anyone have the same problem?

Best Julian

Hi Julian

We have not seen this issue, but we also haven't done much work with k3s running in a virtual machine outside of AWS ec2 instances. What virtualization solution are you using? I'm not seeing too much with some quick googling, some posts indicate this could be related to the underlying pvc implementation. I'll keep looking for thoughts on debugging & fixing.

Jay

I haven't had luck reproducing the mysql issue. You could try an alternate database container such as mysql 5.7 which is what we have used in the past in development at times. This should be a drop in replacement if you want to try it.

First remove the current mysql container: `helm uninstall mysql -n mxgo`. Then delete the persistent storage with `kubctl delete pvc data-mysql-0 -n mxgo`.

Save the attachment and copy it to your vm. Then deploy it like this: `kubectl apply -f /tmp/mysql-5.7.txt -n mxgo`.

This should result in a new mysql pod ready to use with Foundry.

jboyd@LP3-US-51811616:~/work/kubernetes$ kubectl get pods -w -n mxgo
NAME READY STATUS RESTARTS AGE
mysql-deployment-9c69d5f6c-cbkdt 0/1 ContainerCreating 0 8s
mysql-deployment-9c69d5f6c-cbkdt 1/1 Running 0 18s

Jay

mysql-5.7.txt

Hi @Jay Boyd,

thanks for your help. I followed your instruction and MySQL is now up and running now and eventually Foundry was able to connect to the database.

My virtual machine is based on RHEL 9.1 + k3s. It is running in our private and self hosted cloud. We are using Apache Cloudstack and KVM as a hypervisor. I also tried to install Volt MX Go in one of our k8s clusters (Debian hosts) that is running in the same environment. There I got exactly the same error for MySQL.

Julian