Are you not using docker swarm behind a reverse proxy?
ie nginx has the public facing ipv4 & ipv6 address(es), with docker swarm behind that communicating to the nginx proxies over ipv4
I have traefik as part of the docker-compose file. Installing nginx on the host seems less reproducible, though it could fix my problem. I guess I would choose something as caddy (I'm not that happy with traefik)
I've not used Caddy before personally, but if it does reverse proxying then it'd probably work fine rather than Nginx. :)