For us, our Nomad cluster clients (the instances that runs the workloads), are a bit more ephemeral, the workloads are moved around servers, and the servers go up and down as we scale the infrastructure.

A database, that should be highly available, that is on a node that's going down, might disrupt database clients and end-users.

Not to say that you can't host databases inside the cluster, you can place databases on more stable servers (tag them with "db", be careful when you make changes) and don't move workloads, or manage expectations with the database clients.

Most of my k8s experience is from using it 4 years ago, so it may be better with storage now. But we required bare metal for all our dbs. We required full performance.

We did have redis in k8s, but the redis cluster our team needed was over 50 nodes and was experiencing all kinds of odd problems and our ops team insisted we move to vms. Performance was better. We didn't use disc, all memory. Wish I could recall the issues better.

from my own experience, running stateful services in k8s is not easy and prone to errors.

so unless your devops/sysops/whateveryouwannacallitops has a lot of experience, it's just not worth it when things like RDS/ElastiCache exists.

i worked at a place that was 100% k8s, but all stateful things (redis, mysql, rabbitmq) were either hosted or in VMs.